Skip to content

Compressor and decompressor window_bits set to 15 leads to periodically truncated data #32221

Closed as not planned

Description

@Stono

I'm trying to transparently compress response data between services on my (Istio) mesh.
To do this, I have a compressor filter on destination services and a decompressor on the source services.

I discovered that setting window_bits to 15 on both of them results in periodically truncated data. This script loops through the same call and you can see how the data was truncated for one of them:

Screenshot 2024-02-06 at 11 56 58

This does not happen if i set the compressor window bits to 14 or lower.

The documentation on https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/compression/gzip/decompressor/v3/gzip.proto.html#extensions-compression-gzip-decompressor-v3-gzip says

The decompression window size needs to be equal or larger than the compression window size

So "larger" seems to be true, but "equal to" - not so much.

Repro steps:
Here are two gists for the two filters:

Activity

  1. alyssawilk commented on Feb 6, 2024

    @alyssawilk
    Contributor
  2. KBaichoo commented on Feb 7, 2024

    @KBaichoo
    Contributor

    Hmm, from the https://www.zlib.net/manual.html it seems like those parameters should be ok.

    Do you have any Envoy stats, or logs that occur on decompression showing any errors with your reproduction?

  3. Stono commented on Feb 11, 2024

    @Stono
    Author

    Hey sorry for the delay, as i've been coming back to this in bits and bobs.
    No errors in the envoy logs, everything looks "normal" at debug level.
    I've tried all sorts of different configurations in case it was some other weird interaction with things we're doing and haven't found anything at all. The only thing that "fixes" it is a window_size of <= 14.

    Not sure where else to go with it to help you debug!

    Attached some debug logs from our non prod in case you can spot anything, first request worked, second was truncated.
    debug.txt

  4. Stono commented on Feb 11, 2024

    @Stono
    Author

    Just realised i was logging from the compressor side there, and it's the decompressor side that would likely show any errors. Will have another stab tomorrow.

  5. github-actions commented on Mar 12, 2024

    @github-actions

    This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

  6. added
    stalestalebot believes this issue/PR has not been touched recently
    on Mar 12, 2024
  7. github-actions commented on Mar 20, 2024

    @github-actions

    This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

  8. jessebye commented on Oct 30, 2024

    @jessebye

    @Stono we just ran into a problem that is similar to this. Most gzipped requests are fine, but then some are truncated. We have our window_bits set to 15 as well. Did you manage to grab those decompressor logs?

  9. Stono commented on Oct 30, 2024

    @Stono
    Author

    Nope, and I haven't been able to recreate it since!

  10. jessebye commented on Oct 30, 2024

    @jessebye

    Strange... I'm also confused how this would happen. We applied the compressor globally using an EnvoyFilter and had no issue for hundreds of our endpoints, but one endpoint gave back truncated responses. Removing the filter solved the issue, but now we're back to uncompressed responses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugstalestalebot believes this issue/PR has not been touched recently

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions