Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
compress/flate: deflatefast does not deliver consistent results after reset #34121
Omitting here the reasons why stableflate does what it does, and focusing just on the one bad effect:
deflatefast.go maintains a counter "cur" in the object. When cur advances past 1<<30, resetAll is called, which discards the current match table. Discarding the match table changes the behavior of further compression at that point, of course.
Normal reset of the object calls reset, which advances cur by 1<<15 (and then is one of the places that checks if it's now past 1<<30, but this is not important here).
If the same object is repeatedly used for compression runs, with a reset in between, then two different objects, with different histories, but both reset, will have different values for cur. This in turn means there is a different run distance before the match table is discarded, and therefore, potentially different compression behavior even with identical input.
O(32K) (1<<30 / 1<<15) resets is sufficient to produce the problem, which is not absurd for a high-volume server.
changed the title
deflatefast does not deliver consistent results after reset [compress/flate]
Sep 5, 2019
My concern with shifting down the table is that it makes compression slower. If we had Reset clear the table entirely, then compression remains as fast as it does now. Reset doesn't need to be super fast IMO.
However, I have no objection to this style of fix if it's what you prefer.