Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-38589: Fix repeated reads with stream handle #49

Merged
merged 15 commits into from
Apr 11, 2023
Merged

Commits on Apr 10, 2023

  1. Configuration menu
    Copy the full SHA
    40679a3 View commit details
    Browse the repository at this point in the history
  2. Update current position with bytes read not bytes requested

    The current position should be reported as the number of
    bytes that were successfully read.
    timj committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    f2766d1 View commit details
    Browse the repository at this point in the history
  3. Add test for repeated reads

    This passes on file but fails on S3 and HTTP.
    timj committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    bf4b4a8 View commit details
    Browse the repository at this point in the history
  4. Fix running off the end of files over http and s3

    Both s3 and http file handles use byte ranges to request subsequent
    byte reads. Fix an issue where the requested range is past the end
    of the file by catching response codes and requesting the rest of
    the file where appropriate. The handles will return empty byte
    string when there is nothing remaining in the file.
    natelust authored and timj committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    1e8ae03 View commit details
    Browse the repository at this point in the history
  5. Revert the http resource handle change

    We have decided to change tack and use the content-range header
    to determine EOF rather than forcing an additional read from
    the server to trigger a 416 status code.
    timj committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    22b96b0 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2023

  1. Support flush method on http handle in read-only mode

    TextIOWrapper can call flush as part of seek, and so we have
    to support the call in read-only mode.
    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    8c7ff88 View commit details
    Browse the repository at this point in the history
  2. Disable gzip encoding from server when using byte range

    If the server supports gzip encoding (which is the default for
    urllib) the byte ranges refer to bytes in the compressed version
    of the content. This can break the client because there is no
    guarantee that those bytes can be uncompressed. Instead disable
    the Accept-Encoding header so that the server is forced to
    use the original byte range.
    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    1d2c02f View commit details
    Browse the repository at this point in the history
  3. Be a bit more careful with detecting EOF

    Rather than always contacting the server one more time and
    using 416 status code to indicate EOF, instead look at the
    Content-Range header to determine EOF or else use the knowledge
    that the number of bytes received is less than the number
    of bytes requested to indicate EOF.
    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    21dcb52 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    34b8662 View commit details
    Browse the repository at this point in the history
  5. Use named HTTP status codes

    Easier to understand than integers.
    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    fe863d2 View commit details
    Browse the repository at this point in the history
  6. Add news fragment

    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    38c3d19 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e13cbb6 View commit details
    Browse the repository at this point in the history
  8. Be a bit more defensive in determining EOF

    Just in case the reported end position is off the end of
    the file.
    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b58e1ad View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ed3c4be View commit details
    Browse the repository at this point in the history
  10. Significantly expand handle read tests

    This found a bug in S3 handle.
    
    For now we can not pass byte range to wsgidav test server that
    is wholly off the end of the file. Will add a test when upstream
    is fixed.
    timj committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    45af22f View commit details
    Browse the repository at this point in the history