Remove WAL contention between ingest and searches

[mdisibio]

What this PR does:
This PR contains 2 search-related changes:

1. Remove the most of the WAL contention between the write path and searches, by removing the block-level mutex from the ingest path. It is safe for search to read from the WAL while ingest is appending to it, as search gets a copy of the index and will never attempt to read beyond the flushed portion of the file. The single block-level mutex is replaced with shorter-lived mutexes around the index and block header (tags).
2. SearchDataMap is the underlying structure for several operations: deduping and flushing trace segments, and block headers. This PR changes its internal structure from slice to map-based to better handle tags with many values. Benchmarks show that the slice-based approach is competitive up to ~20 values (i.e. it is as fast to iterate the slice than do a single map-lookup). Beyond that the map-based approach is significantly better, and testing it on realistic workloads shows a reduction in ingester CPU of 15+%, and 15+% working set.

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Two quick questions, but lgtm.

Would like input from one of the other maintainers more closely aligned with search.

[kvrhdn]

LGTM 🚀