Deadlock with concurrent credentials fetch

[kennyjwilli]

#130 attempted to fix a deadlock issue while fetching credentials for aws-api calls. This fix does not work in the general case.

Dependencies

com.cognitect.aws/api       {:mvn/version "0.8.445"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.607"}
com.cognitect.aws/sts       {:mvn/version "741.2.504.0"}

Description with failing test case

We have a SQS message processing service that runs 10 processing threads. All 10 of those threads have gotten deadlocked on calls to aws-api. A message processing thread makes calls to our customers' AWS accounts using IAM AssumeRole. Each customer has a different IAM Role ARN and ExternalId which results in the aws-api calls using different CredentialProvider. As stated before, these calls occur in parallel.

If you have 4 or more CredentialsProviders running fetch at the same time, you will hit a deadlock.

I created a repro here.

I think the problem is the same as #130. Its fix only made it less likely to occur. In my repro, this is what is happening in each thread.

| invoke DescribeInstances
  | fetch-creds (AssumeRole)
    | invoke STS AssumeRole
      | fetch-creds (implicit)

The fetch-creds calls go through the async-fetch-pool which is size 4. If you launch N threads to run the above flow, where N >= async-fetch-pool size, you will get a deadlock. This occurs because the inner fetch-creds call is sitting in the async-fetch-pool's queue. The queue will never get processed since the executor is waiting on all the outer AssumeRole fetch-creds calls to finish.

[dchelimsky]

Thank you @kennyjwilli for the thorough report.

[dchelimsky]

Fixed in 0.8.456