[mongodb] Improve mongodb quantization

[pawelchcki]

This PR backports Hash Quantization from 0.13-dev with some necessary changes to make it a tiny bit more compatible with previous MongoDb quantization.

It does so by adding "truncate_arrays" option ensuring the quantization will only include first item from array with nested arrays or objects. Its an important facet to effective MongoDb quantization.

in addition it normalizes hash keys used in resource string. Allowing expected merging of to occur and avoiding situation when one field was included twice.

{.., :collection: "name", "collection": "name",...}

dd-trace-rb/test/contrib/mongodb/client_test.rb

Line 126 in a9087ad

assert_equal('{"operation"=>:insert, "database"=>"test", "collection"=>"people", "documents"=>[{:name=>"?", :hobbies=>["?"]}, "?"], "ordered"=>"?"}', span.resource)

TODO:

• docs for quantization in mongo

[delner]

Can you explain what truncate_arrays does and why it's necessary? And the other features like indifferent equals and so on?

I'm concerned this is making the quantization more complicated than it should be, and thus reducing performance.

[pawelchcki]

Expanding on the PR description:

Array truncation is something that was already used in MongoDB quantization. I reimplemented that in Hash quantization.

The case for it is that mongodb can have quite a lot of embeded documents in its query. And that number can be variable.

Imagine a query that depending on conditions updates 1 to 10000 different documents.
That would mean that there will up to 10000 different resource names created for essentially the same operation. And that would convey little information to the enduser So instead of creating a resource:

{"u"=>[{a: ?}, {a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?}

Array truncation will cut it down to just

{"u"=>[{a:?}, "?"]}

This is something that already exists in current implementation of mongodb quantization.
However it results in following resource name:

{"u"=>{a:?}}

Which I think hides too much information. So I tweaked it a tiny bit, however I'm not married to that idea.

As for performance of the changes, adding one if and a method call overhead that is performed only on arrays should have minimal penalty.

When arrays are truncated it should however give quite measurable speedup due to not having to traverse all objects in array.

[delner]

I think your case where {"u"=>[{a: ?}, {a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?},{a:?}]} repeats, it's pretty clear how it benefits. But what if there are cases where that {a:?} isn't a simple hash but a meaningful object? Such as a document bulk insert? Then that wouldn't be so good, I'd think.

I think we should adopt a simple rule for each data type that makes general sense and live with some of the consequences. Otherwise I fear quantization strategy will constantly thrash whenever the general strategy is sub-optimal for a particular schema. Hash::Quantization should be that general strategy, and if it's too sub-optimal for a specific schema (say ElasticSearch) then, that integration should re-implement its own strategy. In this case, I think the generic strategy is fine for ES.

But getting back to the specifics, and applying what I said in the previous paragraph, I think your point that Arrays are repetitive content, and should be truncated is probably the best general fit. In which case I would suggest Arrays always become ["?"] in Hash::Quantization.

However, I do like your idea of {"u"=>[{a:?}, "?"]} since it gives a sample of what complex object would have been truncated entirely. Maybe that's a decent compromise, and should be integrated in somehow, or at least be an option. And the more I think about this, the more I think we need to refactor the Hash::Quantization to use a strategy pattern, where an integration or some other part of code can define specifics of what strategy it wants to implement (like your truncation strategy.)

But before we do that, let's do the minimal changes here to fix the bug, then pursue a better quantization strategy.

[pawelchcki]

@delner
I've implemented your initial suggestion of using Hash quantization module, to filter MongoDb unwanted object keys. I think it works well, to remove that I'd have to reimplement quite a bit of code thats already in Hash quantization module.

As for {'a'=>?} it works as before, i.e. if ? is a object and not a strings it will be recursed into to perform quantization and reflect the structure of the object. So [{a:{b:{c:1}}}, {a:{b:...}] will be quantised to [{a:{b:{c:"?"}}}, "?"]

I've rebased this branch on 0.13-dev - once #465 is merged, I'll refresh this PR to reflect only relevant changes.

I'll make the array truncation a default per your suggestions.

dd-trace-rb/spec/ddtrace/quantization/hash_spec.rb

Lines 78 to 82 in 47bc33e

	context 'given a Array with nested hashes' do
	let(:hash) { [{ foo: { bar: 1 } }, { foo: { bar: 2 } }] }
	it { is_expected.to eq([{ foo: { bar: '?' } }, '?']) }
	end

[pawelchcki]

Added this PR #467 to test the changes on ES

[delner]

👍