Enable dlt asset metadata via translator #22047
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rexledesma
requested changes
May 23, 2024
| @@ -11,6 +11,8 @@ | |||
|
|
|||
| @dataclass | |||
| class DagsterDltTranslator: | |||
| metadata_by_resource_name: Optional[Mapping[str, Any]] = None | |||
There was a problem hiding this comment.
Feels weird to have this in the base DagsterDltTranslator class. If a user wants to have this, you can subclass DagsterDltTranslator and add it yourself as a property. [1]
python_modules/libraries/dagster-embedded-elt/dagster_embedded_elt/dlt/translator.py
Outdated
Show resolved
Hide resolved
rexledesma
approved these changes
May 23, 2024
python_modules/libraries/dagster-embedded-elt/dagster_embedded_elt/dlt/asset_decorator.py
Outdated
Show resolved
Hide resolved
danielgafni
pushed a commit
to danielgafni/dagster
that referenced
this pull request
Jun 18, 2024
## Summary & Motivation Asset metadata is an essential component of assets definition in real projects. This PR adds a parameter in the DagsterDltTranslator to define resource specific metadata. This is an idea to circumvent the fact that dlt does nor uses a configuration file such as Sling's replication.yaml, where would be natural to define metadata to be read by the translator. ## How I Tested These Changes make pyright make ruff python -m pytest python_modules/libraries/dagster-embedded-elt/dagster_embedded_elt_tests/dlt_tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Asset metadata is an essential component of assets definition in real projects. This PR adds a parameter in the DagsterDltTranslator to define resource specific metadata.
This is an idea to circumvent the fact that dlt does nor uses a configuration file such as Sling's replication.yaml, where would be natural to define metadata to be read by the translator.
How I Tested These Changes
make pyright
make ruff
python -m pytest python_modules/libraries/dagster-embedded-elt/dagster_embedded_elt_tests/dlt_tests