This repository has been archived by the owner on Jul 3, 2023. It is now read-only.
subdag modifier + tag_outputs + refactor #237
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elijahbenizzy
force-pushed
the
subdag-modifier-rebase
branch
from
26ef74d
to
718b790
Compare
|
This PR has turned into a massive one due to a rebase. It's replacing: And will likely replace: |
skrawcz
reviewed
Nov 27, 2022
| load_from=[unique_users, interactions_filtered], | ||
| ) | ||
| def quarterly_user_data_US() -> reuse.MultiOutput({"unique_users_daily_US": pd.Series}): | ||
| pass |
There was a problem hiding this comment.
add documentation to explain what this variant is.
| load_from=[unique_users, interactions_filtered], | ||
| ) | ||
| def daily_user_data_CA() -> reuse.MultiOutput({"unique_users_daily_CA": pd.Series}): | ||
| pass |
There was a problem hiding this comment.
add docs to explain what this variant is.
|
Some thoughts we should sync up on:
|
|
TODO prior to general release:
|
This makes it backwards compatible -- we previously had two files: 1. function_modifiers.py 2. function_modfiers_base.py We now create a module `function_modifiers.py` which enables us to separate out decorators, as the file is getting unwieldy and ugly. That said, we need to maintain backwards compatibility on import. To do this, we: 1. Create a function_modifiers subpackage 2. Put everything that was in function_modifiers.py in __init__.py in that package (so imports work) 3. Move function_modifiers_base.py over there 4. Create a "dummy" function_modifiers_base.py that just imports everything from the actual one. Will need to test this out on sample workflows prior to publishing.
1. configuration -- if/else equivalents 2. dependencies -- utilities for specifying dependencies, generally used for parameterized, but this will likely be expanded. Note decorators. 3. expanders -- decorators that turn one node into many 4. function_modifiers_base -- the base classes/common utilities 5. macros -- decorators that replace a function's implementation with something else (think @does/@dynamic_transform) 6. metadata -- decorators that add metadata to a node (tag right now) 7. validation -- decorators that do some validation on a node Note that this was all done to preserve pure backwards compatibility. Care was taken that every public component of the API was placed within the __init__.py of function_modifiers.
Previously this was in the node's name. This makes it part of the node, allowing for: 1. No string parsing in querying the namespace 2. Other constructs to use namespace effectively 3. Potentially layered namespaces Note we still concatenate strings to reference upstream nodes -- TBD on exactly how we should do this, but i think the convention of '.' joining names is fairly reasonable.
elijahbenizzy
force-pushed
the
subdag-modifier-rebase
branch
from
718b790
to
7f0ed06
Compare
elijahbenizzy
force-pushed
the
subdag-modifier-rebase
branch
from
b9ef6c8
to
f36ce50
Compare
elijahbenizzy
force-pushed
the
subdag-modifier-rebase
branch
4 times, most recently
from
d8b38c6
to
2035e92
Compare
Solves #254 `tag_outputs` enables you to attach metadata to a function that outputs multiple nodes, and give different tag values to different items. This can be applied multiple times, and precedence order of the `tag` family of nodes is in reverse application order (see docs). Note that if we have `tag` before `tag_outputs` this is undefined behavior.
elijahbenizzy
force-pushed
the
subdag-modifier-rebase
branch
2 times, most recently
from
f09f1c5
to
f93785b
Compare
elijahbenizzy
force-pushed
the
subdag-modifier-rebase
branch
2 times, most recently
from
f13dff4
to
a78ff0c
Compare
elijahbenizzy
changed the title
elijahbenizzy
changed the title
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Short description explaining the high-level reason for the pull request]
Changes
How I tested this
Notes
Checklist