Search: Enforce Max Trace Size Bytes

[joe-elliott]

What this PR does:
Uses the per tenant max trace size bytes to enforce limits on search. This has primarily been put in place to reduce serverless OOMs.

Once concern I have about this PR is that it adds a skippedTraces field to the returned search metrics. This field is named similarly to skippedBlocks but they have very different meanings.

• Block skipping occurs in the ingester and indicates that an entire block didn't need to evaluated for the search conditions. It is a performance improvement.
• Trace skipping occurs in the queriers/serverless functions and indicates a trace not evaluated for the conditions provided. It indicates that there is potentially a trace missing from your results.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[zalegrala]

This looks reasonable to me, though I think perhaps some documentation about the expected behavior would be good. I'm not sure who would expect to be searching for large traces, but if they were, knowing what to expect up front seems prudent.

[joe-elliott]

This looks reasonable to me, though I think perhaps some documentation about the expected behavior would be good. I'm not sure who would expect to be searching for large traces, but if they were, knowing what to expect up front seems prudent.

Yeah, I agree. I need to generally improve the search/serverless docs.

[mdisibio]

LGTM - could we update the comments in limits.go for MaxBytesPerTrace?