355 - add computed exhausted property #380
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs canceled.
|
rudolfix
requested changes
Jun 6, 2023
dlt/extract/source.py
Outdated
| @@ -614,6 +614,15 @@ def root_key(self) -> bool: | |||
| config = RelationalNormalizer.get_normalizer_config(self._schema).get("propagation") | |||
| return config is not None and "root" in config and "_dlt_id" in config["root"] and config["root"]["_dlt_id"] == "_dlt_root_id" | |||
|
|
|||
| @property | |||
| def exhausted(self) -> bool: | |||
| # check all selected pipes wether one of them is exhausted | |||
dlt/extract/source.py
Outdated
| def exhausted(self) -> bool: | ||
| # check all selected pipes wether one of them is exhausted | ||
| for resource in self._resources.extracted.values(): | ||
| for item in resource._pipe: |
There was a problem hiding this comment.
you do not need to enumerate items. Pipe has gen property which is the main generator of the pipe. all others are transforms. so check only gen
dlt/extract/source.py
Outdated
| # check all selected pipes wether one of them is exhausted | ||
| for resource in self._resources.extracted.values(): | ||
| for item in resource._pipe: | ||
| if isinstance(item, types.GeneratorType): |
There was a problem hiding this comment.
interesting! I'm using inspect.isgenerator and it's probably better to stick to that (or you double check if the above catches all cases of generators)
rudolfix
requested changes
Jun 7, 2023
tests/extract/test_sources.py
Outdated
| def some_data(): | ||
| yield from [1, 2, 3, 4] | ||
|
|
||
| s = DltSource("source", "module", Schema("source"), [dlt.resource(some_data())]) |
There was a problem hiding this comment.
all good, but please add more resource types to source
- a regular list (iterable)
- an iterator (ie. on a list)
- join gen resource to transformer
I just want to make sure all cases are passing, you can do it on one test
sh-rp
force-pushed
the
d#/355-proper-exhausted-property
branch
from
946bb52
to
788994f
Compare
rudolfix
force-pushed
the
d#/355-proper-exhausted-property
branch
from
c69968e
to
6c7dd39
Compare
…duration of __next__ call
rudolfix
approved these changes
Jun 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.