Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fixes discover taking too long #144

Merged
merged 1 commit into from
Aug 23, 2023

Conversation

a-rampalli
Copy link

What

Airbyte made a change in logic of getting resources for discover call recently. They 1st try to fetch some data from each resource and if this step is successful they are saying this stream is supported by source. This is to avoid issues where some api’s are not available due to pricing plans
Because of this logic this step (discover) is taking more time and causing some timeouts

How

https://linear.app/rudderstack/issue/ETL-82#comment-4830b5f9

Testing

Tested locally and observed discover calls are much faster now.

@linear
Copy link

linear bot commented Aug 23, 2023

ETL-82 Mixpanel discover call taking too long

Airbyte made a change in logic of getting resources for discover call recently. They 1st try to fetch some data from each resource and if this step is successful they are saying this stream is supported by source. This is to avoid issues where some api’s are not available due to pricing plans
Because of this logic this step (discover) is taking more time and causing some timeouts

@github-actions
Copy link

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • The connector tests are passing in CI
  • You've updated the connector's metadata.yaml file (new!)
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@a-rampalli a-rampalli merged commit c6ea047 into main Aug 23, 2023
4 of 7 checks passed
@a-rampalli a-rampalli deleted the fix/etl-82-mixpanel-discover-taking-too-long branch August 23, 2023 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants