destination-s3 use instanceprofile if credentials are not provided

[Álvaro Queiroz (alvaroqueiroz)]

What

Describe what the change is solving
Destination-s3 connector does not support using instanceprofile authentication. This is a huge drawback for some organizations.

This solves the issues 5942 and 8227

How

Allow the fields accessKeyId and secretAccessKey to be optional on the destination-s3 configuration page.
If they are not provided, the getS3Client method will use InstanceProfileCredentialsProvider.

effect:

build:
cmd used - ./gradlew :airbyte-integrations:connectors:destination-s3:build

result of sync, Airbyte is running on ec2 instance, and no aws credentials were provided

my modified connector image can be found here

Recommended reading order

doc for InstanceProfileCredentialsProvider

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Users will now be able to use instance profile authentication for the connector destination-s3

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

[CLAassistant]

All committers have signed the CLA.

[M. Marx (marcosmarxm)]

thanks Álvaro Queiroz (@alvaroqueiroz)

[M. Marx (marcosmarxm)]

Sherif A. Nada (@sherifnada) can you check this PR adding the option to use S3 destination with InstanceProfile. No additional change in UI, but how users can experience errors from the front-end. What do you think? Should add an option to select the connection method => (credentials or instance profile)?

[Álvaro Queiroz (alvaroqueiroz)]

I think that the user-experience will not suffer,

if access_key_id AND secret_access_key are not provided -> instanceprofile auth
if access_key_id OR secret_access_key are not provided -> standard authentication error
the rest will stay the same.

[Sherif A. Nada (sherifnada)]

reassigning to Edward Gao (@edgao) for review

[Edward Gao (edgao)]

added a few minor comments!

[Edward Gao (edgao)]

one small wording change, otherwise LGTM!

[Álvaro Queiroz (alvaroqueiroz)]

exception was added for the case when only id OR secret is inputed

[Álvaro Queiroz (alvaroqueiroz)]

M. Marx (@marcosmarxm) and Sherif A. Nada (@sherifnada) is anything else needed to proceed with the merge?

[M. Marx (marcosmarxm)]

thanks Álvaro Queiroz (@alvaroqueiroz)