Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3, Google Drive, Azure Blob Storage: Update cdk #33411

Merged
merged 16 commits into from Dec 15, 2023

Conversation

flash1293
Copy link
Contributor

@flash1293 flash1293 commented Dec 13, 2023

This PR rolls out #33300 and #32781 to all relevant source connectors.

As remote processing is currently experiencing the issue described in https://github.com/airbytehq/airbyte-internal-issues/issues/6199, this option is temporarily hidden from the spec. Once the platform supports managing secrets in arrays correctly, this limitation will be lifted.

Copy link

vercel bot commented Dec 13, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 15, 2023 11:22am

@octavia-squidington-iii octavia-squidington-iii added the area/connectors Connector related issues label Dec 13, 2023
Copy link
Contributor

github-actions bot commented Dec 13, 2023

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@flash1293 flash1293 marked this pull request as ready for review December 13, 2023 13:28
@octavia-squidington-iv octavia-squidington-iv requested a review from a team December 13, 2023 13:29
@flash1293 flash1293 changed the title S3, Google Drive, Azure Blob Storage: Update cdk for remote document file type processing S3, Google Drive, Azure Blob Storage: Update cdk Dec 14, 2023
@flash1293
Copy link
Contributor Author

@clnoll @aaronsteers pinging you on this - as we can't roll out the remote processing feature right now without weird behavior, I hid it temporarily by adjusting the spec in the connector. Once the linked issue is resolved, it can be shown "properly"

Copy link
Contributor

@clnoll clnoll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me @flash1293, just one small question.

@@ -67,22 +67,19 @@ class Config:
def documentation_url(cls) -> str:
return "https://docs.airbyte.com/integrations/sources/google-drive"

@staticmethod
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this have to be reverted when the platform issue is fixed? Couldn't quite see why google drive had this method but the other sources don't.

Copy link
Contributor Author

@flash1293 flash1293 Dec 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been moved into the abstract file based spec and doesn't have to be moved back. It's only on google drive because the other connectors didn't have oneOfs in their spec which made it necessary (Google drive has two auth modes)

@flash1293 flash1293 merged commit 5324da8 into master Dec 15, 2023
27 of 28 checks passed
@flash1293 flash1293 deleted the flash1293/cdk-rollout-pr branch December 15, 2023 13:52
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants