Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modis / Download and process step #63

Open
Tracked by #62
franTarkenton opened this issue May 3, 2023 · 2 comments
Open
Tracked by #62

Modis / Download and process step #63

franTarkenton opened this issue May 3, 2023 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@franTarkenton
Copy link
Member

franTarkenton commented May 3, 2023

Sub ticket of 31...

Rework the download of modis data.

  • downloads and then uploads to object store
  • check ostore first for data then source
  • configure persistence to ostore after downloaded from source.

relates to: #44

@franTarkenton franTarkenton added the enhancement New feature or request label May 3, 2023
@franTarkenton
Copy link
Member Author

To get this working needed to re-work how the downloads are being performed.

The refactor of the granule downloads is complete, the next step that has been started is to include the object storage sync.

@franTarkenton franTarkenton self-assigned this May 9, 2023
@franTarkenton
Copy link
Member Author

Completed... Download process has been reworked to:

  1. check for the file locally. If the file exists locally then it also checks to make sure its in object storage. If not then it gets uploaded there.
  2. if the file doesn't exist locally it checks to see if its in object storage, and if so pulls from there.
  3. if the file does not exist locally AND does not exist in object storage THEN it will go to the National Snow and Ice Data Centre and pull from there, and then immediately after it will upload the file to object storage.

@franTarkenton franTarkenton added zen_reorg temp label to help with migration to ghp and removed zen_reorg temp label to help with migration to ghp labels Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Review / QA
Development

No branches or pull requests

1 participant