Problematic third-party dependencies when moving from ByteBuffers to Memory

[leventov]

Main issue is #3892

• Roaring depends on ByteBuffers. Should either be merged into Druid codebase (see https://groups.google.com/d/msg/druid-development/Q039gtYVSQM/TgFzzezJCwAJ), or Roaring upstream library should adopt Memory (unlikely). cc @lemire
• https://github.com/jpountz/lz4-java supports only byte[] and ByteBuffer. Should likely be merged into Druid to add Memory support. Or merged into Memory library itself? or another separate library, lz4-memory, should be created?

More problematic dependencies, when found, to be mentioned and discussed in comments.

cc @leerho

[lemire]

@leventov

Should either be merged into Druid codebase (see https://groups.google.com/d/msg/druid-development/Q039gtYVSQM/TgFzzezJCwAJ)

I think my answer would be the same, still. I don't see anything that would prevent it.

Roaring depends on ByteBuffers.

I don't think that's true. Maybe you mean the other way around?

$ mvn dependency:resolve
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building RoaringBitmap 0.6.40-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:resolve (default-cli) @ RoaringBitmap ---
[INFO]
[INFO] The following files have been resolved:
[INFO]    org.apache.commons:commons-lang3:jar:3.4:test
[INFO]    org.hamcrest:hamcrest-core:jar:1.3:test
[INFO]    junit:junit:jar:4.12:test
[INFO]    com.google.guava:guava:jar:18.0:test
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.552 s
[INFO] Finished at: 2017-03-08T15:37:50-05:00
[INFO] Final Memory: 16M/160M
[INFO] ------------------------------------------------------------------------

Roaring upstream library should adopt Memory (unlikely)

Why is that unlikely?

[leventov]

@lemire I mean that there is a way to create Roaring Bitmap, backed by ByteBuffers. ByteBuffers is a part of JDK so it's not in maven dependencies :)

Unlikely that you want to start depending third-party Memory library and integrate it into upstream Roaring. But I could be wrong. If you are not conceptually against it, we can submit a PR with Memory support.

[lemire]

@leventov

Unlikely that you want to start depending third-party Memory library and integrate it into upstream Roaring. But I could be wrong. If you are not conceptually against it, we can submit a PR with Memory support.

Roaring is community-driven. I don't think we have a rule against third-party libraries.

[niketh]

@leventov I had a chat with @cheddar a couple of weeks ago while figuring a way around this. Both of us felt it was better to submit a PR with Memory against Roaring repo

[leventov]

@leerho
It seems to me that compression (LZF, LZ4) and hashing should be implemented on the Memory library side and be part of the Memory API. Otherwise they couldn't be implemented efficiently without excessive data copies.

[lemire]

@leventov BTW if you have arrays of integers, they can be more efficiently compressed than by Lempel Ziv. I do not necessarily recommend using JavaFastPFOR as a dependency, but you should definitively be aware of what it does if you are dealing with arrays of integers at all.

[leventov]

@lemire thanks for pointing this out. Sometimes we compress integers and sometimes floating point data.

[lemire]

@leventov Both of these are subject to data-aware optimized codecs; there has been a ton of work done on compressing integers in particular. Parquet, LinkedIn Pinot and others use such techniques to good effect. I'm available to chat about it.

[leerho]

@leventov @lemire

It seems to me that compression (LZF, LZ4) and hashing should be implemented on the Memory library side and be part of the Memory API. Otherwise they couldn't be implemented efficiently without excessive data copies.

Sorry, I am coming in at the tail end of this thread and not sure I fully understand the use-case. But I am skeptical of adding data-aware transformations directly into Memory. Please explain where the efficiency gain would be.

[lemire]

@leerho Oh. I don't know whether it would be beneficial, but you can compress and uncompress arrays of integers an order of magnitude faster using dedicated algorithms (as opposed to Lempel Ziv) while improving compression ratios. This was just a passing remark, not a specific recommendation.

[leerho]

@lemire I agree. But it is up to the user, who understands their data, to choose the appropriate transformation algorithm. Trying to generalize that down to a low-level Memory API would be fraught with difficulties.

[leventov]

@leerho

Sorry, I am coming in at the tail end of this thread and not sure I fully understand the use-case. But I am skeptical of adding data-aware transformations directly into Memory. Please explain where the efficiency gain would be.

The only was how you can compress Memory using e. g. LZ4 JNI is to copy the data to byte array/byte buffer and feed to JNI. Because Memory abstraction doesn't guarantee the data resides continuous chunk of actual virtual memory of the current process. ByteBuffer does guarantee this.

The other way around is to add something like isContinuous(offset, len) to Memory API, getObject(offset), getCumulativeOffset(offset).

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.