🎉 File source: Add support for Azure blob storage

[Phlair]

What

Adding to Files source connector, ability to grab from Azure Blob Storage

How

smart_open already supports Azure Blob so this was just a case of following existing structure of the Files source connector to create relevant AzBlob functions and integration tests (these required creating a storage account in Airbyte's Azure manually with user integration-test@airbyte.io to use for blob upload/read)

Pre-merge Checklist

• Run integration tests
• Publish Docker images

Recommended reading order

1. test.java
2. component.ts
3. the rest

[Phlair]

I've successfully run the new integration tests locally, however I'm assuming I need to put the new Azure secrets somewhere so that the build can access them for the tests?
(@sherifnada this PR aims to close #1391)

[sherifnada]

@Phlair thank you! review coming shortly. in the meantime for this to run in CI can you add the azblob.json config you're using to Lastpass as a secure note named Source file azure creds?

@davinchia will take care of adding it to github secrets for you where it will be made available in Ci

[Phlair]

^ those creds are now in Lastpass :)

[davinchia]

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/885079739
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/885079739

[davinchia]

Looks great! Thanks George!

One minor comment. Other than that, I want to wait for the integration test to succeed before merging this in.

[sherifnada]

@Phlair please follow the instructions here #3733 to use the secrets in CI. I've given you permissions on the repo. Then run the test command Davin commented with above to run tests on the branch!

[davinchia]

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/888404928
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/888404928

[davinchia]

This is passing even though I haven't yet added the secret because the workflow is set up to pull from our repo instead of the forked repo.

[davinchia]

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/888503849
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/888503849

[davinchia]

/test connector=source-file

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/890356896
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/890356896

[davinchia]

/test connector=source-file repo=Phlair/airbyte

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/890357454
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/890357454

[davinchia]

@Phlair sorry for the merge delay. I'm also using this PR as a way to refine our tooling around contribution from forked repos. I think I've gotten it mostly working - the current error is due to a bad dependency I fixed in the course of looking at this PR. Do you mind adding this to your changes so I can successfully build this? Happy to do so myself if you give me write permission to your fork!

[Phlair]

@davinchia No problem, happy to be part of progress 👍
I've merged master of the main repo into my fork, which included your changes. Just started to follow @sherifnada's advice above but looks like the secrets are already added to Github, following commit adds references to these in relevant files.

[Phlair]

/test connector=source-file repo=Phlair/airbyte

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/893048880
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/893048880

[Phlair]

/test connector=source-file repo=Phlair/airbyte

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/893100706
❌ source-file https://github.com/airbytehq/airbyte/actions/runs/893100706

[davinchia]

/test connector=source-file repo=Phlair/airbyte

🕑 source-file https://github.com/airbytehq/airbyte/actions/runs/894557199
✅ source-file https://github.com/airbytehq/airbyte/actions/runs/894557199