New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Large (~13.8kb) HTTP gzip responses aren't decompressed #11252
Comments
I reported a similar issue to @straight-shoota yesterday where HTTP:Client response's body is binary. I have yet to determine whether it's gibberish or if it didn't deflate the gzip encoding. It's very intermittent, and happens for small data (1kb) for a specific HTTP domain (api.airtable.com). Other domains used are fine. |
The issue here is that you write the content to the HTTP response before setting the It works with smaller content sizes because the data is buffered. You have to move the configuration lines setting content encoding and content type above the IO copy. @ysbaddaden This appears to be unrelated to your problem. |
This is not a bug, but I believe we can improve the developer experience by trying to raise an error on incorrect usage.
This is not easy to implement with setting arbitrary headers (such as in Maybe we could consider adding |
Oh thanks! But, the server code above was actually an attempt to replicate the issue I was getting locally, and it looks like I messed up on that. Oops 😅. From your explanation, the actual problem I'm getting might be more related to what @ysbaddaden is experiencing. But the gist of it is the same; require "http"
require "json"
require "uri"
client = HTTP::Client.new(URI.parse("https://www.youtube.com"))
client.compress = true
post_data = {"context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "WEB",
"clientVersion" => "2.20210721.00.00", "clientScreen" => "WATCH_FULL_SCREEN"}},
"continuation" => "4qmFsgJIEhhVQ1h1cVNCbEhBRTZYdy15ZUpBMFR1bncaLEVnWjJhV1JsYjNNd0FqZ0JZQUZxQUxnQkFDQUE2Z01JUTJkU1JGRlZSVGs9",
}.to_json
yt_headers = HTTP::Headers{
"Content-Type" => "application/json",
"Accept-Encoding" => "gzip",
}
results = client.post("/youtubei/v1/browse?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8", body: post_data,
headers: yt_headers)
# Should be string but instead it's some random binary data
results.body This returns gibberish binary data, but replicating the same query on curl does result in the correct (and decompressed) JSON response. |
You're actively opting out of the automatic decompression by setting the You don't need to assign |
Though, it is not explained that passing the |
|
I am hitting this or #11354, In the body the first bytes in the returned string are reported as "\u001F\x8B". I'm a lttle confused about whether this is reporting an extra zero byte or not. The Conttent-Encoding header says "gzip". It's intermittent - does not always occurr on the same server response. |
Bug Report
If I have this server:
And a client requesting it like so:
I'd expect Crystal to decompress the response automatically, and give me the resulting data. While this is the behavior for smaller responses, it seems like anything larger than ~13.8kb causes this step to be skipped. The above code would just return random gibberish binary data, instead of the long hex string.
The text was updated successfully, but these errors were encountered: