Improved Ingester out-of-order error for faster troubleshooting

[wardbekker]

What this PR does / why we need it:
See #963

Which issue(s) this PR fixes:
Fixes #963, although not returning a JSON struct yet, that could be a follow up improvement

Special notes for your reviewer:
First pull request to Loki, love to get your feedback!

Checklist

• Documentation added
• Tests updated

[CLAassistant]

All committers have signed the CLA.

[rfratto]

LGTM! I do have concerns about how it changes the behavior of the code when we received errors that aren't out-of-order errors.

[slim-bean]

My only big concern with merging this, is that I think we need to figure out the return first. That is I don't want to merge this now then convert it to JSON output later.

[slim-bean]

This kind of then opens a bigger can of worms around what our API error handling looks like/should look like

[wardbekker]

My only big concern with merging this, is that I think we need to figure out the return first. That is I don't want to merge this now then convert it to JSON output later.

no worries. will work on a JSON return within this pull request

[slim-bean]

Hold off just a bit @wardbekker we were discussing a little in slack, but I think we need a strategy for error return types in general. Don't want you to head down a road until we are sure what direction we want to go with this

[rfratto]

This LGTM, we're going to tackle changes to error return types in a future PR, and this is a good improvement to have in the interrim.