[WIP] HLL Rollup field mapper plugin implementation and Cardinality Agg hacking

[amontalenti]

• Write up basic spec of this project in Notion
• Explore HyperLogLogPlusPlus class; read associated paper
• Write new unit tests for serialization and deserialization of HLL
• Record empirical results of those tests in Notion
• Add a skeletal mapper-hll plugin based on the mapper-murmur3 plugin
• Hack around with making the serialized HLL go into binary doc values storage for the custom field type
• Get remote debugging of ES and gradle integration tests with intellij working
• Write a very basic end-to-end integration test for using the hll field and cardinality agg together
• Hack around in CardinalityAggregator to make it pick up serialized HLL bytes
• Make the initial integration test pass based on HLLFieldMapper and CardinalityAggregator changes!
• Get answer on why initialBucket and otherBucket are always 0 right now
• Work on making the HLLFieldMapper support arrays of values and a specified precision field; use the GeoShape field mapper for inspiration.
• Explore how XContentParser works so that we can do streaming parsing of the items array.
• Look into this comment in SortedDocValuesField: "This value can be at most 32766 bytes long."
• Figure out if BinaryDocValuesField is what HLLFieldMapper should really be using, due to the above limit, but can't, perhaps due to the current ES ValuesSource implementation. (Is this what Zach was talking about?)
• Investigate whether the ByteArrayOutputStream, ByteArrayInputStream are the right options for HLL serialization/deserialization.
• See whether the stored field in the HLLFieldMapper is actually storing the byte[], and also what option we could possibly have for displaying it (in searches) and using it for re-indexing operations.
• Explore whether there's a way to change what gets stored in _source for a given field, so that, e.g. we could store the serialized blob of the HLL in _source (note: we'd also need to support serialized HLLs in the HLLFieldMapper class) rather than being forced to store the raw items
• ⭐ BytesRef stores whole bytes array, and it stores its own offset and length. Which means passing the underlying bytes array is not the right move. Might just be lucky in current impl.
• ⭐ Figure out how big an HLL Rollup can be to fit into 32766 bytes, just in case. (Somewhere near 10K.)
• ⭐ Make HLLFieldMapper support arrays of values via the streaming JSON parsing technique (tokens) rather than all-at-once parsing.
• ⭐ Rename hll field type to hll-rollup
• ⭐ Write up spec for abstracting the CardinalityAggregator changes into a new plugin with a new agg name, such as hll-uniq
• ⭐ Split hacked CardinalityAggregator code changes into that hll-uniq plugin
• ⭐ Build a real-world data set of URLs, timestamps, and UUIDs using BigQuery
• ⭐ Do a real world performance benchmark

[amontalenti]

Re: this just-completed task:

• Hack around in CardinalityAggregator to make it pick up serialized HLL bytes

Happy to say I just got an integration test passing with this test input:

# Integration tests for Mapper HLL components
#

---
"Mapper HLL":

  - do:
      indices.create:
        index: test
        body:
          mappings:
            type1: { "properties": { "foo": { "type": "hll" } } }

  - do:
      index:
        index: test
        type: type1
        id: 0
        body: { "foo": "bar" }

  - do:
      indices.refresh: {}

  - do:
      search:
        body: { "aggs": { "foo_count": { "cardinality": { "field": "foo" } } } }

  - match: { aggregations.foo_count.value: 1 }

  - do:
      index:
        index: test
        type: type1
        id: 1
        body: { "foo": "bar" }

  - do:
      index:
        index: test
        type: type1
        id: 2
        body: { "foo": "baz" }

  - do:
      index:
        index: test
        type: type1
        id: 3
        body: { "foo": "bam" }

  - do:
      index:
        index: test
        type: type1
        id: 4
        body: { "foo": "bar" }

  - do:
      indices.refresh: {}

  - do:
      search:
        body: { "aggs": { "foo_count": { "cardinality": { "field": "foo" } } } }

  - match: { aggregations.foo_count.value: 3 }

... which means, the mapper-hll plugin prototype is serializing HLLs as binary blobs into the binary doc values area of the index, and my hacked CardinalityAggregator is successfully de-serializing those blobs and calculating HLL merges from them, rather than building HLLs "on the fly" from the raw values.

My debug screen in intellij captured here showcases the code in action, being caught during a live ES cardinality agg query.

There's still more to do (as described in the checklist above), but my, my, how very promising!

[amontalenti]

This is for my own benefit for local debugging; won't be in the final PR. That said, might be nice to make this customizable.

[amontalenti]

Likewise, this is for my own debugging.

[amontalenti]

The above long values are just for debugging and will come out once we think on this section a bit more.

[amontalenti]

Due to the above, this test takes about 1.5s to run, whereas other tests in this suite only take 500-700ms. May want to think about a way to make this test faster by using fewer unique values.

This code also exercises the deserialization & serialization points.

[amontalenti]

query-side deserialization point

[amontalenti]

index-side serialization point

[amontalenti]

This introduces a fielding test case until I can get HLLFieldMapper to fully support objects/arrays, but I'm already partway there as of d557530

[amontalenti]

Should be aware that this probably allocates the whole Map into memory, and, in practice, we're expecting this to be a pretty large map. It seems like the ParseContext object is flexible enough to support streaming parsing of JSON, so, the right way to handle that would be to advance, token at a time, via calls to .parser().currentToken() and parser().nextToken(), and actually offer the items to the in-memory HLL one-string-at-a-time as you come across the VALUE_STRING objects inside the array.

[amontalenti]

In a couple of commits, I thought the right way to do this was to pass bytes.bytes, bytes.offset, bytes.length to the ByteArrayInputStream constructor, since via Lucene talks I discovered that the buffer underneath the BytesRef is not just something you can pass around freely. But, when I tried to make use of a copy or do that trick, I hit other issues, so I tabled it for now.