-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-38589: Fix repeated reads with stream handle #49
Commits on Apr 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 40679a3 - Browse repository at this point
Copy the full SHA 40679a3View commit details -
Update current position with bytes read not bytes requested
The current position should be reported as the number of bytes that were successfully read.
Configuration menu - View commit details
-
Copy full SHA for f2766d1 - Browse repository at this point
Copy the full SHA f2766d1View commit details -
This passes on file but fails on S3 and HTTP.
Configuration menu - View commit details
-
Copy full SHA for bf4b4a8 - Browse repository at this point
Copy the full SHA bf4b4a8View commit details -
Fix running off the end of files over http and s3
Both s3 and http file handles use byte ranges to request subsequent byte reads. Fix an issue where the requested range is past the end of the file by catching response codes and requesting the rest of the file where appropriate. The handles will return empty byte string when there is nothing remaining in the file.
Configuration menu - View commit details
-
Copy full SHA for 1e8ae03 - Browse repository at this point
Copy the full SHA 1e8ae03View commit details -
Revert the http resource handle change
We have decided to change tack and use the content-range header to determine EOF rather than forcing an additional read from the server to trigger a 416 status code.
Configuration menu - View commit details
-
Copy full SHA for 22b96b0 - Browse repository at this point
Copy the full SHA 22b96b0View commit details
Commits on Apr 11, 2023
-
Support flush method on http handle in read-only mode
TextIOWrapper can call flush as part of seek, and so we have to support the call in read-only mode.
Configuration menu - View commit details
-
Copy full SHA for 8c7ff88 - Browse repository at this point
Copy the full SHA 8c7ff88View commit details -
Disable gzip encoding from server when using byte range
If the server supports gzip encoding (which is the default for urllib) the byte ranges refer to bytes in the compressed version of the content. This can break the client because there is no guarantee that those bytes can be uncompressed. Instead disable the Accept-Encoding header so that the server is forced to use the original byte range.
Configuration menu - View commit details
-
Copy full SHA for 1d2c02f - Browse repository at this point
Copy the full SHA 1d2c02fView commit details -
Be a bit more careful with detecting EOF
Rather than always contacting the server one more time and using 416 status code to indicate EOF, instead look at the Content-Range header to determine EOF or else use the knowledge that the number of bytes received is less than the number of bytes requested to indicate EOF.
Configuration menu - View commit details
-
Copy full SHA for 21dcb52 - Browse repository at this point
Copy the full SHA 21dcb52View commit details -
Configuration menu - View commit details
-
Copy full SHA for 34b8662 - Browse repository at this point
Copy the full SHA 34b8662View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe863d2 - Browse repository at this point
Copy the full SHA fe863d2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 38c3d19 - Browse repository at this point
Copy the full SHA 38c3d19View commit details -
Configuration menu - View commit details
-
Copy full SHA for e13cbb6 - Browse repository at this point
Copy the full SHA e13cbb6View commit details -
Be a bit more defensive in determining EOF
Just in case the reported end position is off the end of the file.
Configuration menu - View commit details
-
Copy full SHA for b58e1ad - Browse repository at this point
Copy the full SHA b58e1adView commit details -
Configuration menu - View commit details
-
Copy full SHA for ed3c4be - Browse repository at this point
Copy the full SHA ed3c4beView commit details -
Significantly expand handle read tests
This found a bug in S3 handle. For now we can not pass byte range to wsgidav test server that is wholly off the end of the file. Will add a test when upstream is fixed.
Configuration menu - View commit details
-
Copy full SHA for 45af22f - Browse repository at this point
Copy the full SHA 45af22fView commit details