[Java] ArrowFileWriter/ArrowStreamWriter lack compression support

[lidavidm]

Describe the enhancement requested

Their reader counterparts support compression, but the writers don't give you any way to enable it. Furthermore, the design of the compression API means that enabling this will require modifying several classes: the compression API closes the input buffer after compression, but this runs contrary to the expectation of VectorUnloader, which isn't supposed to mutate the source VectorSchemaRoot. (Arguably, this is a flaw in the VectorUnloader API that prevents efficient memory usage, but fixing that would mean invalidating essentially all current Arrow Java code.)

Also there are various other flaws (e.g. VectorUnloader says it is OK to pass a null compression codec but will crash if you do so because it assumes the codec is never null.)

See also: #15102

It is possible to manually generate compressed data yourself in Java, but this requires taping together several low-level APIs yourself.

Component(s)

Java

[lidavidm]

CC @liyafan82 did I miss anything about the Java compression API here?

[liyafan82]

CC @liyafan82 did I miss anything about the Java compression API here?

I guess not. Thanks for bringing up the problems.