Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Technical walk through how feed data updates are detected and loaded in warehouse #296

Closed
machow opened this issue Aug 23, 2021 · 1 comment
Assignees

Comments

@machow
Copy link
Contributor

machow commented Aug 23, 2021

TODO: go through with @ccjarvus, to ensure the process is sound. Determine path for #235. Currently feeds that are removed from agencies.yml are not marked as deleted in our type2 slowly changing dimension data.

@machow
Copy link
Contributor Author

machow commented Sep 7, 2021

Copying in from #277

helpful notes from Chris--there are 4 jobs being done here (should also have 4 dags):

  1. put into cloud storage
  2. understand files (and when they change)
  3. load feeds with changed files into external tables
  4. understand what has changed in each table (e.g. build SCD tables)

@machow machow closed this as completed Sep 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants