chore: remove cursor.is_greater_than_or_equal #674
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Testing This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@maxi297/clean-declarative-stream#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch maxi297/clean-declarative-streamHelpful ResourcesPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
📝 WalkthroughWalkthroughThis change removes the Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Suggested reviewers
Would you like to have a second reviewer with deep familiarity in the incremental sync logic, wdyt? Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (12)
💤 Files with no reviewable changes (10)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
airbyte_cdk/sources/declarative/retrievers/simple_retriever.py (1)
536-537: Consider prioritizing the cleanup referenced in the FIXME comment.The FIXME comment suggests that the removed
_get_most_recent_recordlogic can be fully eliminated as part of addressing the linked internal issue. Since this migration is removing legacy declarative cursor functionality, would it make sense to tackle that cleanup sooner rather than later to avoid accumulating technical debt, wdyt?Would you like me to help track down any remaining references to the removed record tracking logic or generate a script to verify the cleanup is complete across the codebase?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (12)
airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py(0 hunks)airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py(0 hunks)airbyte_cdk/sources/declarative/incremental/per_partition_cursor.py(0 hunks)airbyte_cdk/sources/declarative/incremental/per_partition_with_global.py(0 hunks)airbyte_cdk/sources/declarative/incremental/resumable_full_refresh_cursor.py(0 hunks)airbyte_cdk/sources/declarative/retrievers/simple_retriever.py(1 hunks)airbyte_cdk/sources/streams/checkpoint/cursor.py(0 hunks)airbyte_cdk/sources/streams/checkpoint/resumable_full_refresh_cursor.py(0 hunks)airbyte_cdk/sources/streams/checkpoint/substream_resumable_full_refresh_cursor.py(0 hunks)unit_tests/sources/declarative/incremental/test_datetime_based_cursor.py(0 hunks)unit_tests/sources/declarative/incremental/test_per_partition_cursor.py(0 hunks)unit_tests/sources/declarative/retrievers/test_simple_retriever.py(0 hunks)
💤 Files with no reviewable changes (11)
- unit_tests/sources/declarative/incremental/test_per_partition_cursor.py
- airbyte_cdk/sources/declarative/incremental/global_substream_cursor.py
- airbyte_cdk/sources/declarative/incremental/per_partition_with_global.py
- airbyte_cdk/sources/declarative/incremental/per_partition_cursor.py
- unit_tests/sources/declarative/incremental/test_datetime_based_cursor.py
- airbyte_cdk/sources/streams/checkpoint/resumable_full_refresh_cursor.py
- airbyte_cdk/sources/declarative/incremental/resumable_full_refresh_cursor.py
- airbyte_cdk/sources/streams/checkpoint/substream_resumable_full_refresh_cursor.py
- airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py
- unit_tests/sources/declarative/retrievers/test_simple_retriever.py
- airbyte_cdk/sources/streams/checkpoint/cursor.py
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: ChristoGrab
PR: airbytehq/airbyte-python-cdk#58
File: airbyte_cdk/sources/declarative/yaml_declarative_source.py:0-0
Timestamp: 2024-11-18T23:40:06.391Z
Learning: When modifying the `YamlDeclarativeSource` class in `airbyte_cdk/sources/declarative/yaml_declarative_source.py`, avoid introducing breaking changes like altering method signatures within the scope of unrelated PRs. Such changes should be addressed separately to minimize impact on existing implementations.
airbyte_cdk/sources/declarative/retrievers/simple_retriever.py (1)
Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/_run.py:62-65
Timestamp: 2024-11-15T01:04:21.272Z
Learning: The files in airbyte_cdk/cli/source_declarative_manifest/, including _run.py, are imported from another repository, and changes to these files should be minimized or avoided when possible to maintain consistency.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: Check: destination-motherduck
- GitHub Check: Check: source-shopify
- GitHub Check: Check: source-intercom
- GitHub Check: Check: source-hardcoded-records
- GitHub Check: SDM Docker Image Build
- GitHub Check: Pytest (Fast)
- GitHub Check: Pytest (All, Python 3.10, Ubuntu)
- GitHub Check: Pytest (All, Python 3.11, Ubuntu)
🔇 Additional comments (1)
airbyte_cdk/sources/declarative/retrievers/simple_retriever.py (1)
533-533: LGTM - Simplified cursor.close_slice call aligns with the migration goals.The removal of the most recent record parameter from
cursor.close_slice(_slice)is consistent with eliminating theis_greater_than_or_equalcomparison logic from cursor implementations. This simplification is a good step toward the concurrent cursor framework migration, wdyt?
PyTest Results (Fast)3 689 tests - 11 3 678 ✅ - 11 6m 28s ⏱️ -16s Results for commit 0f17729. ± Comparison against base commit 51cfea5. This pull request removes 11 tests.♻️ This comment has been updated with latest results. |
maxi297
force-pushed
the
maxi297/clean-declarative-stream
branch
from
0b3e5bd to
0f17729
Compare
PyTest Results (Full)3 692 tests - 11 3 681 ✅ - 11 11m 46s ⏱️ - 6m 25s Results for commit 0f17729. ± Comparison against base commit 51cfea5. This pull request removes 11 tests. |
brianjlai
reviewed
Jul 29, 2025
maxi297
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
We still have a couple of usage of the declarative cursors. We would rather use the concurrent one because the declarative ones are maintained to a minimum. This will allow us to migrate to the DefaultStream and remove the DeclarativeStream down the line.
How
is_greater_than_or_equal: based on the comment above in SimpleRetriever.read_records, it seems like we can tackle https://github.com/airbytehq/airbyte-internal-issues/issues/6955 and remove thisSummary by CodeRabbit
Refactor
Tests