Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in auto-materialize, handle asset observations with data times but no data versions #19866

Open
sryza opened this issue Feb 16, 2024 · 0 comments
Labels
area: auto-materialize Related to Auto Materialization

Comments

@sryza
Copy link
Contributor

sryza commented Feb 16, 2024

Imagine asset graph A->B, where A is an observable source asset and B is an asset with an eager auto-materialize policy.

Currently, if A is observed, the materialize_on_parent_updated rule will consider A updated iff its latest observation has a different data version than its previous observation.

#19830 introduces the ability for observations to have empty data versions, but instead specify a data time.

In these cases, we'd probably want the materialize_on_parent_updated rule to consider A updated iff its latest observation has a different data time than its previous observation.

One way to implement this would be to expand the logic inside _asset_partition_versions_updated_after_cursor.

@sryza sryza added the area: auto-materialize Related to Auto Materialization label Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: auto-materialize Related to Auto Materialization
Projects
None yet
Development

No branches or pull requests

1 participant