[aws-http4s] Credential provider chain swallows errors, probably works incorrectly on ECS/EC2

[keynmol]

(follow up from DM with @Baccata)

This one is difficult to reproduce, but the issue can be summarised in this screenshot:

In short, using the default credentials provider seems to be brittle - even though the ECS endpoint env variable is present, for some reason getting credentials through that fails, and the error that bubbles up is for the credentials file.

My hunch is that actual source of the error is timeouts (e.g. during task start up, 1 second might not be enough) and lack of retries. My current workaround is to manually create a provider with desired logic:

  def awsEnvironment(
      httpClient: org.http4s.client.Client[IO]
  ): Resource[IO, AwsEnvironment[IO]] =
    std
      .Env[IO]
      .get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
      .toResource
      .flatMap:
        case None =>
          AwsEnvironment.default[IO](httpClient, AWS_REGION)

        // Below is a hack to work around a bug in credentials loading
        // chain in smithy4s - we manually prioritise ECS credentials
        case Some(_) =>
          val provider = new AwsCredentialsProvider[IO]
          provider
            .refreshing(
              provider
                .fromECS(httpClient, 10.second)
                .onError(exc =>
                  scribe.cats.io
                    .error("Failed to get credentials from ECS endpoint", exc)
                )
            )
            .map: cred =>
              AwsEnvironment.make(
                httpClient,
                IO.pure(AWS_REGION),
                cred,
                IO.realTime.map(_.toSeconds).map(Timestamp(_, 0))
              )

But I think this whole credentials chain thing should be redesigned to support

1. Enabling/disabling providers
2. Retries for network-based providers
3. Error accumulation
4. Per-provider error summary when everything fails