Repository navigation
Replies: 3 comments 1 reply
|
Here are the current family IDs, including the set operations and filters in the Java registry. The hex column shows the family byte at offset 2 in the DataSketches preamble. Java family registry
Both formats using 19 ( 23 ( |
|
I saw @leerho's comment on #datasketches-dev Slack channel and I think we are generally fine to change DensitySketch's Family ID to 23 now. |
Uh oh!
There was an error while loading. Please reload this page.
While implementing
DensitySketchindatasketches-rust(apache/datasketches-rust#62), we noticed that Family ID 19 is currently used by two different sketches:ser_verfamily_iddensity_sketchpreamble_intsin{3, 6}119density_sketch.hpp(added in Apr 2023)ebpps_sketch/EbppsItemsSketchpreamble_longsin{1, 5}119Family.java,ebpps_sketch.hpp(added in Jan 2024)Because byte 0 happens to be disjoint (
{3, 6}vs.{1, 5}), cross-deserializing one as the other fails on the preamble check rather than silently corrupting data. However,FamilyIDs are meant to be globally unique: Java'sFamily.idToFamily(int)enforces a 1-to-1 ID map (XORFILTERcurrently occupies22), and any header inspection or multi-sketch support in Java/Rust/Go runs into the collision.Questions
19forEBPPSand assign Family ID23(DENSITY) inFamily.javaand across implementations before more languages shipDensitySketch?density_sketchblobs, should readers accept legacyfamily_id == 19alongside23(while all new writers emit23), or only support19in C++ as a legacy path?All reactions