Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set availabilitystrategy for GA connectors + gitlab with httpstreams explicitly to None #21888

Closed
wants to merge 43 commits into from

Conversation

erohmensing
Copy link
Contributor

@erohmensing erohmensing commented Jan 25, 2023

What

Now that the HttpAvailabilityStrategy is published in the CDK, we want to turn it on as default for HttpStreams. In order to prevent issues in GA connectors if there is unexpected incompatibility or issues, we want to turn on the HttpAvailability (or a substitute custom AvailabilityStrategy)

How

I found a list of GA python connectors by scripting the following:

  • get a list of connectors from the airbyte-integrations/connectors folder
  • filter for sources only
  • filter for sources which have setup.py files somewhere in their folder
  • filter for python sources which are marked as generally-available

For each of those, if the GA python source was using HttpStreams and was an AbstractSource, I set the AvailabilityStrategy explicitly to None by doing the following:

  • added the availability_strategy to the base stream class for that source's streams, set to None
  • bumped the version in the Dockerfile
  • added a changelog entry

There are 31 of these, + gitlab = 32, so there should be 96 changed files.

Note that this doesn't change how the connector will behave - the default for all AvailabilityStrategies at the moment is None. This is being done in preparation for #21924, where the default will be set to HttpAvailabilityStrategy, which will switch it on for all alpha/beta connectors minus gitlab.

Pre-merge

  • Test all connectors
  • Update changelog date for all connectors
  • Merge in master and handle merge conflicts (connectors that have been merged in the meantime)
  • Publish all connectors

Follow up

  • Create an issue for each GA connector to implement an availabilitystrategy (default or custom)
  • Create an issue for Gitlab to implement the httpavailabilitystrategy for all streams possible (those which don't have per-record 403s), or implement a better custom availabilitystrategy

@octavia-squidington-iv octavia-squidington-iv added the CDK Connector Development Kit label Jan 25, 2023
@octavia-squidington-iv octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/gitlab labels Jan 25, 2023
This was referenced Jan 27, 2023
@erohmensing
Copy link
Contributor Author

Closing in favor of the separate PRs now that all follow up issues have been created

@erohmensing erohmensing closed this Feb 2, 2023
@erohmensing erohmensing deleted the enable-availability branch April 25, 2023 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment