Initial implementation of InflaterSource and DeflaterSink

[swankjesse]

No description provided.

[swankjesse]

There’s a decent amount of complexity in the two new loops, writeBytesFromSource() and readBytesToTarget().

They’re both almost the same, except:

• writeBytesFromSource() counts about how many bytes are consumed. It doesn’t care how many are produced!
• readBytesToTarget() counts how many bytes are produced. It doesn’t care how many are consumed!

Critically, both inflator and deflator are tested on ratios from 0.001% through 700%. That makes me confident that we’re handling cases where a small input produces a big output, and vice versa.

I originally tried to do this as one big function and that sucked. Doing it as two functions is better, though the helper functions are themselves a bit awkward.

I’m tempted to follow-up by replacing the DataProcessor’s ByteArray property with a Segment. I’d need to rework some other stuff, but it might be a net reduction in code which I’d like. The only real gotcha there is the segment’s pos+limit are potentially different from next_out+avail_out in cases where we don’t want to fill up the entire segment.

Next steps - I don’t think this is particularly useful as-is and I promised a release for Feb 9. I’d like to cut that release beforehand and keep working on this into next week.

[JakeWharton]

Better to do this inside the loop or outside? I would have thought outside since this is called once per write on the source.

[swankjesse]

Good question! I think inside is correct specifically for inflate + deflate.

Delivering data one-segment-at-a-time is pretty balanced:

• limit how much memory it holds. If you have a bunch of these on a web server, you don‘t have to worry about giant allocations when data decompresses tremendously!
• reduce latency. We can deliver data to the user (or network, or whatever) as soon as it’s ready
• still get some benefits from batch processing. We do expensive operations like syscalls once every 8 KiB to amortize the cost

We’d get maximum efficiency (fewer syscalls) if we delivered more segments at a time. But I think the real optimization there is to do a once-per-decade review of Segment.SIZE and see if 8 KiB makes as much sense in 2024 as it did in 2014.

[swankjesse]

(Last time I considered dynamic segment sizes I concluded it was a bad fit. Pooling gets weirder, for one.)

[JakeWharton]

Yeah I guess I was more thinking as to whether it made sense for a single Sink.write call to result in 500 Sink.writes beneath this, or whether it should be a single call with a multi-segment buffer. I suppose in practice the compression level isn't going to be astronomical so it doesn't really matter.