[Traceql metrics] New (unsafe) query hints

[mdisibio]

What this PR does:
Adds several new query hints which expose traditionally server-side config options. These are useful for operators (like us) to try out and quickly optimize tuneables like job size, sharding interval, and concurrency etc. But because these are unsafe to expose to end-users they are protected behind a per-tenant override flag. For example a job size of 1 byte would quickly take out the read path.

Safe hint:
The only safe hint is the existing sample=<float>.

Unsafe hints:

• dedupe=<bool>
  • Use dedupe=false to disable deduping spans. I am doing this mainly for profiling analysis, but a use case is to check compactor performance (how well is your data deduped?).
• job_size=<int>
  • Used in the query-frontend, override for target_bytes_per_request. Default = configured in yaml
• job_interval=<duration>
  • How to divide the time range (5m, 1h, etc), override for interval. Default = configured in yaml
• concurrent_blocks=<int>
  • How many blocks to inspect concurrently. Works for both querier backend and local blocks processor. Default=configured in yaml. This is a new yaml setting with defaults: querier = 2, local blocks processor = 10.
• time_overlap_cutoff=<float>
  • The new tunable added in Update traceql metrics to use the trace-level timestamp columns #3353. Now exposed to the yaml also Default = 0.2

Requires #3388

Which issue(s) this PR fixes:
Fixes n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mapno]

LGTM.

About dedup, it might be safe but do we want users passing this hint to Tempo? Is there a use case other than for operators to tune compaction?

[mdisibio]

LGTM.

About dedup, it might be safe but do we want users passing this hint to Tempo? Is there a use case other than for operators to tune compaction?

Right, there isn't really a use-case for it besides tuning compaction. Currently it's faster to not dedupe, so you could trade accuracy for speed, but that should go away after we change dedupe to loser-tree.

1. Do you want to put it in the unsafe bucket? We can always make it "safe" again later if the use case is found.
2. Also what do you think about the location of all the hints? Naturally we want to centralize them together in hints.go, but it also makes sense to keep the hints close to the caller. For example target_bytes_per_request is only relevant to the query-frontend.

[mdisibio]

@mapno This is ready for re-review. Centralized all of the hint definitions and safety to the traceql package, exposed config options for concurrent_blocks and time_overlap_cutoff, and moved dedupe to be unsafe.

[mapno]

LGTM