-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
utf8_toUtf8(const ENCODING *UNUSED_P(enc),
const char **fromP, const char *fromLim,
char **toP, const char *toLim)
{
char *to;
const char *from;
const char *fromLimInitial = fromLim; /* Avoid copying partial characters. */ for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++) if (fromLim < fromLimInitial)
return XML_CONVERT_INPUT_INCOMPLETE;
else if ((to == toLim) && (from < fromLim))
// <===== Bug is here. In case (to == toLim), it's possible that
// from is still pointing to partial character. For example,
// a character with 3 bytes (A, B, C) and form is pointing to C.
// It means only A and B is copied to output buffer. Next
// scanning will start with C which could be considered as invalid
// byte and got dropped. After this, only "AB" is kept in memory
// and thus it will lead to invalid continuation byte.
return XML_CONVERT_OUTPUT_EXHAUSTED;
else
return XML_CONVERT_COMPLETED;
} |
This is not our code. Please use corresponding bug tracker [1] if you have found a bug in Expat. But I think there is not a bug here. |
Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here) |
For the record, the upstream fix is commit libexpat/libexpat@74a7090 and it will be part of Expat 2.2.4. |
Expat 2.2.4 with a fix has been released now: https://github.com/libexpat/libexpat/releases |
I confirm the regression (see bpo-31303 for reproducer). Victor, do you mind to update expat to 2.2.4? This issue can be classified as a security issue, since a regression was added in security update. |
Corresponding Expat issue: libexpat/libexpat#115. |
We can put expat 2.2.4 in 2.7.14 final. |
I produced attached PR 3315 using attached cpython_rebuild_expat_dir.sh + revert Modules/expat/expat_external.h change to keep #include "pyexpatns.h". |
I wrote an non-regression test for the Python master branch using the test case attached to libexpat/libexpat#115: PR 3570. |
I believe all the branches except 3.5 and 3.4 have been updated so adjusting the Versions field accordingly. All yours, Larry! |
libexpat has been upgraded to version 2.2.4 in Python 2.7, 3.3, 3.4, 3.5, 3.6 and master. I added an unit test to Python 2.7, 3.6 and master. I'm not sure about backporting the new unit test to Python 3.3, 3.4 and 3.5. I close the issue. |
Oh, it seems like Steve Dower found an issue on Windows: PR 3751. I reopen the issue. |
Can this issue be closed now? |
Ah yes, it can be closed. I was waiting 3.4 and 3.5 fixes to be fixed, which is now the case. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: