Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Source-mongo] : Adding a new iterator in preparation of adding chunking subqueries #37525

Merged
merged 4 commits into from
Apr 24, 2024

Conversation

akashkulk
Copy link
Contributor

Similar to the MySQL iterator : https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mssql/src/main/java/io/airbyte/integrations/source/mssql/initialsync/MssqlInitialLoadRecordIterator.java

This is a pure refactor - Currently, an extra query will be issued (which shouldn't actually retrieve any data). An edge case where new records where added to the collection after the first query ran will, whoever return additional data

Next Steps are :

  1. Querying for table estimate
  2. Using that to define chunk size

Copy link

vercel bot commented Apr 23, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 24, 2024 4:26pm

@akashkulk akashkulk marked this pull request as ready for review April 23, 2024 23:29
@akashkulk akashkulk requested a review from a team as a code owner April 23, 2024 23:29
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 24, 2024
@akashkulk akashkulk merged commit d9ee103 into master Apr 24, 2024
34 checks passed
@akashkulk akashkulk deleted the akash/mongo-chunking branch April 24, 2024 17:26
FVidalCarneiro pushed a commit to AgiData/airbyte that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/mongodb-v2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants