fix: require stream_count >= 1 on DynamicStreamCheckConfig#992
Conversation
…kConfig Previously, DynamicStreamCheckConfig.stream_count defaulted to 0, which caused _check_generated_streams_availability to slice generated_streams[:0], resulting in zero streams being checked. This meant connectors using DynamicStreamCheckConfig without explicitly setting stream_count would silently skip all dynamic stream availability checks. Now, stream_count <= 0 means 'check all generated streams', which is the expected behavior when no limit is specified. Co-Authored-By: bot_apk <apk@cognition.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1776453297-fix-stream-count-zero-check-all#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1776453297-fix-stream-count-zero-check-allPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/prerelease
|
|
Devin is currently unreachable - the session may have died. |
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
stream_count semantics on DynamicStreamCheckConfig
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
stream_count semantics on DynamicStreamCheckConfigstream_count >= 1 on DynamicStreamCheckConfig
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/prerelease
|
Summary
Clarifies and tightens the semantics of
stream_countonDynamicStreamCheckConfigused byCheckStreamfor dynamic streams:stream_countunset (default) — all generated streams are checked.stream_count>= 1 — the first N generated streams are checked (capped at the number of available streams).stream_count< 1 — rejected at validation time viaminimum: 1/ge=1.Previously
stream_countdefaulted to0and the runtime treated<= 0as "check all", which conflated "unset" with "zero" and accepted nonsensical negative values. Nowstream_countis optional; when set, it must be a positive integer.Changes
airbyte_cdk/sources/declarative/declarative_component_schema.yaml: dropdefault: 0, addminimum: 1, update description.airbyte_cdk/sources/declarative/models/declarative_component_schema.py:stream_count: Optional[int] = None,ge=1.airbyte_cdk/sources/declarative/checks/check_stream.py:DynamicStreamCheckConfig.stream_countis nowOptional[int] = None._check_generated_streams_availabilitychecks all streams whenmax_count is None, otherwise slices tomax_count.airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py: passstream_countthrough as-is soNoneis preserved.Review & Testing Checklist for Human
None= check all,N >= 1= check first N,< 1rejected) match the intended behavior forDynamicStreamCheckConfig.stream_count.stream_count: 0or a negative value will now fail validation. Those values were accepted before even though they had no sensible meaning.check.Test plan
Run the declarative checks unit tests locally:
All tests should pass, including the new
test_stream_count_unset_checks_all_streams,test_stream_count_unset_failed, andtest_check_stream_non_positive_stream_count[zero|negative]cases.Notes
DynamicStreamCheckConfig.stream_countbecomingOptional[int]with defaultNoneis technically a type-level breaking change for any external Python caller constructing the dataclass with a positional int. In practice the component is built via the factory from the declarative manifest, so this should be safe.test_concurrent_declarative_source.py(test_read_with_concurrent_and_synchronous_streams_with_concurrent_state,test_read_with_concurrent_and_synchronous_streams_with_sequential_state,test_read_concurrent_skip_streams_not_in_catalog) also fail onmainwithout these changes and are unrelated tostream_count.Link to Devin session: https://app.devin.ai/sessions/ec09e8ac822145cdb110cd25dce60013