dbt select job blueprints example #22908
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sryza
added a commit
that referenced
this pull request
Jul 19, 2024
## Summary & Motivation This PR proposes turning Definitions into a "dumb" data class for passing around diverse sets of definitions. This means that users won't encounter any validation errors for unsatisfied resources, duplicate asset keys, etc., until either: - They run a code server to load the definitions - They invoke `Definitions.validate_loadable` Why do this? - Less work will happen at module import time. This gives users more flexibility to determine where they want to incur the cost of this validation. It also helps enable a future world where run/step workers only need to load/validate the subset of definitions that they need to execute. - Less repetition of resources. This was sited as an issue by the developers of our internal platform: https://dagsterlabs.slack.com/archives/C03AF3MMEEN/p1720546166484669?thread_ts=1720545875.842949&cid=C03AF3MMEEN. - Makes it possible to define asset jobs / schedules in separate `Definitions` objects from the assets they target. I encountered this when trying to write an [example that that allows using blueprints to define dbt selection jobs](#22908). The original Definitions.merge PR contains some discussion on this topic: #21746. ## How I Tested These Changes
sryza
force-pushed
the
blueprints-example-dbt-jobs
branch
from
29f130a
to
253a267
Compare
sryza
force-pushed
the
blueprints-example-dbt-jobs
branch
from
253a267
to
0093049
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
See the README for details.
How I Tested These Changes