More descriptive AWS error messages.

[nevumx]

What is the underlying problem you're trying to solve?

Currently, AWS error messages only reveal the error message associated with the status code that AWS returns, which can be quite vague, for example, running opa run -s -c opa-bad-config.yml with the following config

services:
  - name: s3
    url: http://s3.amazonaws.com/fsssssssshhhhhhhhhhh/
    credentials:
      bearer:
        token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
bundles:
  policies:
    service: s3
    resource: policies.tar.gz
    polling:
      min_delay_seconds: 10
      max_delay_seconds: 20
    persist: true
persistence_directory: /opt/gusto/opa/var/cache/

Will produce an error like {"level":"error","msg":"Bundle load failed: server replied with Bad Request","name":"policies","plugin":"bundle","time":"2024-02-26T08:28:12-08:00"} which basically says that AWS returned a 400.

Describe the ideal solution

The error message has all of the information received from AWS.

Describe a "Good Enough" solution

The error message has more of the information received from AWS.

Additional Context

First discussed with @kroekle.

[ashutosh-narkar]

This error is returned by the OPA downloader and is then included in the status message. It's possible that the errors returned by AWS in this instance could contain sensitive information. This information would get logged and also included in the status API if enabled. We probably want to keep the downloader service agnostic. What if we logged this at debug level?

[nevumx]

Yeah that would definitely be a "good enough" solution; the data would be available to us if we needed it. The ideal solution might be a config value, but I don't want to clutter it unless absolutely necessary.

[ashutosh-narkar]

PR: #6647