cURL manifest command lines expire after only one hour

[hannes-ucsc]

• Store OAuth2 client secret in AWS Secrets Manager #7950 (aka SM)
• Implement authorization code flow with Azul and Data Browser #7954 (aka DB, backend work)
• Implement authorization code flow with Azul and Data Browser #7954 data-browser#4793 (frontend work)
• Tighten verification of access tokens #7993, once Data Browser participates in authorization code flow
• Store access and refresh token in DynamoDB #7991, using the iss and sub claims from ID token as key
• When manifest is first requested with regular access token:
  • Use access token against tokeninfo endpoint to obtain iss and sub
  • Retrieve access and refresh tokens from DynamoDB using iss and sub as key
  • If missing, fall back to regular manifest generation, otherwise:
    • Mint a new JWT access token, henceforth Azul Access Token or AZAT …
      • with sub set to iss + sub, i.e., the key of the DynamoDB row holding the access and refresh token
      • iss set to URL of Azul instance
      • exp set to now plus 7 days
      • signed using KMS.sign() and an asymmetric key (RS256 or similar)
    • Use AZAT for the Authorization header in the 2nd curl invocation of the manifest command line
• When /repository/files requests come in with an AZAT
  • Validate the AZAT using KMS.verify(), return error if AZAT is invalid or stale
  • Attempt to retrieve access and refresh tokens from DynamoDB using AZAT's sub as key, error if missing
    • If access token is fresh, use it to download the file from the mirror or the underlying repository, otherwise:
      • Use the refresh token to obtain a new access token from Google
      • Store new access token in DynamoDB and retry file download with that new access token

Optimizations (included in MVP):

• Outsource the public key of the KMS key used for AZAT signing
• Use the outsourced public key to validate the AZAT offline, instead of invoking KMS.verify()

Enhancements (beyond MVP)

• Add flow for refreshing access tokens with Azul and DB, given old but still valid access token and ID token
• Implement authorization code flow in Azul's Swagger UI
• Also use AZAT to authenticate the /manifest/files/{manifest_key} request in the curl command line, i.e. the left-hand side of the pipe construct in the command line
• When storing the client secret in SM, verify that the secret matches the ID
  • There are some vague references on the web that indicate this can be achieved by hitting the /token endpoint with a fake authorization code and distinguishing between the invalid_grant error (indicating success) and all other errors (indicating an invalid client or some other problem)
  • after successful verification, store the client ID along with the secret in SM
  • before using the secret, verify that the configured client ID matches the stored one
  • for extra points: add a TF null resource (or the modern equivalent, I forgot what its called) that performs the same verification at deploy time