Improve error message when object storage "sanity checks" fail

[56quarters]

Is your feature request related to a problem? Please describe.

A user ran into the following error message when trying to run Mimir:

level=warn ts=2022-08-10T14:00:30.459580751Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="2 errors: blocks storage: unable to successfully send a request to object storage: 302 Found; ruler storage: unable to successfully send a request to object storage: 302 Found"

Describe the solution you'd like

Turns out this was because insecure: true was set and the object storage backend was attempting to redirect to HTTPS. 302 is a redirect so including the requested URL and Location header would have helped make it obvious what the problem was here.

Describe alternatives you've considered

N/A

[hi-rustin]

@56quarters Where should we add it? I can't find a place to add it.

[pracucci]

The "302 Found" error message comes from the specific object client implementation, but which was the type of object storage used (e.g. S3? GCS? ...). I would first look at the client, and I would also investigate if the client could be configured to follow redirects instead.

[jhalterman]

It looks like we won't be able to improve the error message since our client abstracts away the Http response. We could try to add support for following redirects to objstore, but that project may want a good use case for doing so.

Along those lines, the S3 folks specifically mentioned that they don't do redirects, but the GCS docs indicate that 302s are possible. And in any case, it's possible that a user could be running a reverse proxy that does redirects in front of their object storage.

I opened thanos-io/objstore#49 to see if we can get this added upstream.

[56quarters]

Hey, if this turns into an ordeal don't worry about spending too much time on it. It was a "nice to have" improvement that came out of helping someone troubleshoot Mimir setup.