Add support for compression: Gzip and deflate #1244

ghostbuster91 · 2020-08-24T09:22:54Z

This PR adds operators to compress/decompress stream of bytes using gzip and deflate algorithms.

Most of the heavy work was done by my colleague @LGLO while implementing this feature in ZIO (zio/zio#3825)

I've just adapted his work to monix. I also did some improvements based on the fs2 code (added option to provide fileName, comment and modificationTimestamp when compressing a stream).

Not sure if the packages are ok, looking forward for your review.

ghostbuster91 · 2020-08-24T09:24:58Z

monix-reactive/jvm/src/main/scala/monix/reactive/compress/package.scala

+      bufferSize: Int = 32 * 1024,
+      chunkSize: Int = 32 * 1024,
+      compressionParameters: CompressionParameters = CompressionParameters.Default,
+      noWrap: Boolean = false


Should we keep it as Boolean or model it as in fs2: https://github.com/typelevel/fs2/blob/main/core/jvm/src/main/scala/fs2/compression.scala#L15 ?

I'm personally fine with API without DeflateParams overload

Avasil

Thank you for the hard work, it looks very good and the test coverage is fantastic. :)

I've left some comments, mainly nitpicks.
I didn't fully review compression logic and I only glanced at the tests, I'll have another pass later, PR is quite big 😅

monix-reactive/jvm/src/main/scala/monix/reactive/compress/CompressionException.scala

monix-reactive/jvm/src/main/scala/monix/reactive/compress/CompressionParameters.scala

...reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/DeflateOperator.scala

Avasil · 2020-09-06T16:13:50Z

monix-reactive/jvm/src/main/scala/monix/reactive/compress/package.scala

+    val DEFLATE: Byte = Deflater.DEFLATED.toByte
+  }
+
+  implicit class RichByteObservable(source: Observable[Byte]) {


Alternatively, we could use Transformer that was brought recently but it would be a precedence. :D

I feel like it could be a bit nicer and more discoverable (source.transform(gzip)) than syntax import.
I would also move the methods to the top of the file so users don't have to scroll through private fields if they explore the file

WDYT?

If the Transformer is a new way of doing such things then we should go with it.

I don't know if it is nicer or more discoverable but certainly it would be easier to use it from java.

I will take a look at Transformer and try to use it.

Also transform might be better because this operation doesn't change the shape of the stream as most of the operators do but only affects its payload.

I don't use it too frequently myself, but many people like it, it's the equivalent of via / transduce / through etc

Avasil · 2020-09-06T16:16:56Z

monix-reactive/jvm/src/main/scala/monix/reactive/compress/package.scala

+      fileName: Option[String] = None,
+      comment: Option[String] = None,
+      modificationTime: Option[Instant] = None
+    ): Observable[Byte] = {


Is it desirable to return Observable[Byte] ?

I feel like it would be better to take and return Observable[Array[Byte]] and if the users need Observable[Byte] then they could do buffering and concatenation themselves.

fs2 and ZIO return Stream[Byte] but in reality, it's a stream of Chunk[Byte] which has to be explicit in Observable

When you read data in a streaming fashion in binary format (e.g. from file) you mostly get Observable[Byte], so that would require users to always do buffering before using any compression transformer. Also they would need to know what chunk size to choose. Choosing too small or too big value can cause performance degradation. This explains my intention behind using Observable[Byte] as the input.

To not introduce any confusion and be transparent for the end users I decided to do not change the type and return Observable[Byte]

I was wrong, you might get Stream[Byte] but that's rather popular among libraries that chunk data under the hood (fs2/zio) which effectively means that you always deal with Stream[Array[Byte]] either explicitly or not. That's good as transforming bytes one by one might not be the optimal solution in terms of performance.

Also monix-nio returns Observable[Array[Byte]] and if we should be compatible with anything, that should be monix-nio.

As another reason for Array, Akka Streams (which doesn't do chunking) uses ByteString which extends IndexedSeq[Byte]

When you read data in a streaming fashion in binary format (e.g. from file) you mostly get Observable[Byte], so that would require users to always do buffering before using any compression transformer.

Related operators, like Observable.fromInputStream return Observable[Array[Byte]] and if we add more, I feel like we should emit Array[Byte] for performance reasons, as you mentioned.

Also they would need to know what chunk size to choose.

That's a fair point, maybe we could add a comment with a suggestion/example how to use it with Observable[Byte]?

Ideally we would write a small benchmark on both to compare their performance :)

About that:

That's a fair point, maybe we could add a comment with a suggestion/example how to use it with Observable[Byte]?

I thought that maybe such helpful extension method could be declared for those who will wonder?

final def chunked[B >: A](chunkSize :Int = 8 * 1024)(implicit ev: B =:= Byte) : Observable[Array[Byte]] = { bufferTumbling(chunkSize) .map(seq => ev.liftCo[Seq](seq).toArray) }

To make it more discover-able I would define it on Observable itself although that's a precedence.

I think this operator might make sense for performance scenarios, not only for Byte.

bufferTumbling uses Array as a buffer but it converts it to Seq before sending downstream.
We could copy-paste implementation and send the copy of the array without the conversion.

Although it would be better to do with benchmarks (there are some that use bufferTumbling and it's a bottleneck) so I feel like it would be preferable to tackle in another PR

Avasil · 2020-09-06T16:23:13Z

monix-reactive/jvm/src/main/scala/monix/reactive/compress/package.scala

+      bufferSize: Int = 32 * 1024,
+      chunkSize: Int = 32 * 1024,
+      compressionParameters: CompressionParameters = CompressionParameters.Default,
+      noWrap: Boolean = false


I'm personally fine with API without DeflateParams overload

monix-reactive/jvm/src/test/scala/monix/reactive/compress/DeflateIntegrationTest.scala

monix-reactive/jvm/src/test/scala/monix/reactive/compress/InflateTest.scala

…han operators

ghostbuster91 · 2020-09-07T14:04:29Z

Last but not least fs2 returns GunzipResult when using gunzip transformer which contains some additional metadata along with the decompressed stream. I decided not to go this way since it seems unnatural for observables but I would like to know your opinion on that.

… than byte

monix-reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/GzipOperator.scala

Avasil · 2020-09-26T21:09:46Z

monix-reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/GzipOperator.scala

+          isDone = true
+          if (ack == null) ack = Continue
+          ack.syncOnContinue {
+            out.onNext(gzipper.finish())


gzipper.finish() never throws, right?

It seems that it can, should be fixed now.

...-reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/GunzipOperator.scala

...reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/InflateOperator.scala

Avasil

Sorry for the delay, I've made another pass - I've resolved cosmetic comments and elaborated on those that should be fixed. If you don't have time then let me know and I can finish it - it already looks fantastic!

Avasil · 2020-09-26T21:20:18Z

Last but not least fs2 returns GunzipResult when using gunzip transformer which contains some additional metadata along with the decompressed stream. I decided not to go this way since it seems unnatural for observables but I would like to know your opinion on that.

Yeah, I think that's fine. If there's ever a demand, we could create an extra method but working with stream of streams could be awkward for the general use case and I assume you're implementing it because you need it in sttp/tapir

…nt with fs2/zio

ghostbuster91 · 2020-09-26T23:16:22Z

Sorry for the delay, I've made another pass - I've resolved cosmetic comments and elaborated on those that should be fixed. If you don't have time then let me know and I can finish it - it already looks fantastic!

No problem, I know how it is ;) Thanks for the careful code-review. I think that now the code should comply with monix-reactive protocol.

One think that bothers me is that this inconsistency with existing protocol wasn't caught by tests. Correct me if I am wrong, but I think that you mentioned (during your last presentation) something about having a generic test for checking protocol compliance. Maybe I've just forgot to implement some trait in tests?

Avasil · 2020-09-27T10:43:06Z

One think that bothers me is that this inconsistency with existing protocol wasn't caught by tests. Correct me if I am wrong, but I think that you mentioned (during your last presentation) something about having a generic test for checking protocol compliance. Maybe I've just forgot to implement some trait in tests?

There is BaseOperatorSuite - I've forgotten to mention it on this PR. Now I'm actually curious how many violations would it catch :D

ghostbuster91 · 2020-09-28T20:54:25Z

We can check it, but I am not sure how to combine this BaseOperatorSuite with existing decompression tests. Ideally I would mix it into GunzipTest, GzipTest, InflateTest and DeflateTest but this is an abstract class (any reason for that?) so I can't. Even if I could I would have to use this testScheduler which requires manual ticking which seems irrelevant from the perspective of these tests. Also I think that I need to rename these classes to suits to follow convention. On the other hand the BaseOperatorSuite is about something different than these tests are so maybe it should go into separate files?

Avasil · 2020-09-28T21:21:13Z

Even if I could I would have to use this testScheduler which requires manual ticking which seems irrelevant from the perspective of these tests.

Many other operators are synchronous too so that shouldn't be a problem.
For custom test cases, you can just supply normal Scheduler if you prefer to use it

Also I think that I need to rename these classes to suits to follow convention. On the other hand the BaseOperatorSuite is about something different than these tests are so maybe it should go into separate files?

Usually, we just add test cases below implementation of BaseOperatorSuite

I don't know how well will the test suite fit for these operators so feel free to skip it if there are too many problems

…ming them to follow convention

…ression tests

Avasil

It looks great to me - thanks once again :)

Although few tests seem to fail

Add support for compression: Gzip and deflate

a07fc77

ghostbuster91 commented Aug 24, 2020

View reviewed changes

Implement manually inputstream.readAllBytes as it wasn't present in jdk8

d9d4c24

Avasil reviewed Sep 6, 2020

View reviewed changes

ghostbuster91 added 7 commits September 7, 2020 11:20

[Refactor] CompressionLevel - change jValue to value

ff2df3c

[Refactor] Mark CompressionParameters as final

ba60b11

[Refactor] DeflateOperator - remove redundant flag isDone

0946f95

[Refactor] DeflateOperator - use continue as a default value for ack

fda8964

[Refactor] Expose compression utils as stream transformation rather t…

0a15274

…han operators

[Refactor] Use <-> to indicate function equality

7306052

[Refactor] Extract common part of GunzipTest and InflateTest

5372531

Avasil added this to the 3.3.0 milestone Sep 9, 2020

[Refactor] Compression operators should operate on array[Byte] rather…

7e78c69

… than byte

Avasil reviewed Sep 26, 2020

View reviewed changes

monix-reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/GzipOperator.scala Outdated Show resolved Hide resolved

Avasil reviewed Sep 26, 2020

View reviewed changes

...-reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/GunzipOperator.scala Outdated Show resolved Hide resolved

Avasil reviewed Sep 26, 2020

View reviewed changes

...reactive/jvm/src/main/scala/monix/reactive/compress/internal/operators/InflateOperator.scala Outdated Show resolved Hide resolved

Avasil reviewed Sep 26, 2020

View reviewed changes

ghostbuster91 added 6 commits September 27, 2020 00:39

[Compression] Fix GunzipOperator monix.Observable protocol violations

921fefd

[Compression] Remove redundant flag from GzipOperator

c86da71

[Compression] Fix GzipOperator monix.Observable protocol violations

0de5fc7

[Compression] Fix DeflateOperator monix.Observable protocol violations

b06ade3

[Compression] Fix InflateOperator monix.Observable protocol violations

cf4955c

[Compression] Rename package (compress -> compression) to be consiste…

54919bb

…nt with fs2/zio

[Compression] Fix compilation on older versions of scala

c1a8cc8

ghostbuster91 added 3 commits September 29, 2020 20:38

[Compression] Refine tests by implementing BaseOperatorSuite and rena…

ca22152

…ming them to follow convention

[Compression] Implement brokenUserCodeObservable test case for decomp…

7c11b9b

…ression tests

[Compression] Use provided error rather than hardcoded one

a66185d

Avasil approved these changes Oct 2, 2020

View reviewed changes

[Compression] Fix gunzipOperatorSuite and remove resolved todo

87c4346

Avasil merged commit 46d91c2 into monix:series/3.x Oct 2, 2020

Add support for compression: Gzip and deflate #1244

Add support for compression: Gzip and deflate #1244

ghostbuster91 commented Aug 24, 2020

ghostbuster91 Aug 24, 2020 •

edited

Avasil Sep 6, 2020

Avasil left a comment

Avasil Sep 6, 2020

ghostbuster91 Sep 7, 2020 •

edited

Avasil Sep 26, 2020

Avasil Sep 6, 2020

ghostbuster91 Sep 7, 2020

ghostbuster91 Sep 13, 2020

ghostbuster91 Sep 13, 2020

Avasil Sep 21, 2020

paualarco Sep 26, 2020 •

edited

ghostbuster91 Sep 26, 2020

Avasil Sep 27, 2020

Avasil Sep 6, 2020

ghostbuster91 commented Sep 7, 2020 •

edited

Avasil Sep 26, 2020

ghostbuster91 Sep 26, 2020

Avasil left a comment

Avasil commented Sep 26, 2020

ghostbuster91 commented Sep 26, 2020

Avasil commented Sep 27, 2020

ghostbuster91 commented Sep 28, 2020 •

edited

Avasil commented Sep 28, 2020

Avasil left a comment

Add support for compression: Gzip and deflate #1244

Add support for compression: Gzip and deflate #1244

Conversation

ghostbuster91 commented Aug 24, 2020

ghostbuster91 Aug 24, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Avasil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghostbuster91 Sep 7, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paualarco Sep 26, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghostbuster91 commented Sep 7, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Avasil left a comment

Choose a reason for hiding this comment

Avasil commented Sep 26, 2020

ghostbuster91 commented Sep 26, 2020

Avasil commented Sep 27, 2020

ghostbuster91 commented Sep 28, 2020 • edited

Avasil commented Sep 28, 2020

Avasil left a comment

Choose a reason for hiding this comment

ghostbuster91 Aug 24, 2020 •

edited

ghostbuster91 Sep 7, 2020 •

edited

paualarco Sep 26, 2020 •

edited

ghostbuster91 commented Sep 7, 2020 •

edited

ghostbuster91 commented Sep 28, 2020 •

edited