Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify Daily pipeline so fills in missing data for a given time window #44

Open
Tracked by #62
franTarkenton opened this issue Mar 16, 2023 · 1 comment
Open
Tracked by #62
Assignees
Labels
bug Something isn't working enhancement New feature or request

Comments

@franTarkenton
Copy link
Member

currently the script is setup to run on a daily basis where it runs using the current date.

Issues with this approach is the data is not always available for the current date. Sometimes there can be a 3-4 day lag before data becomes available.

Changes:

  • instead of just running a specific date, the script will check what data exists in the object store bucket vs the data that is currently available. It will then run the pipeline for consecutive days, untill al the data that is currently available has been processed.
@franTarkenton
Copy link
Member Author

Code was re-worked for modis flow, however ended up being a lot of work so didn't re-work for the viirs flow.

Added action that identifies what data is missing and then passes that onto the downstream jobs. Downstream jobs are setup as matrix builds, that allows them to be processed in parrallel.

@franTarkenton franTarkenton added zen_reorg temp label to help with migration to ghp and removed zen_reorg temp label to help with migration to ghp labels Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
Status: Review / QA
Development

No branches or pull requests

1 participant