Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First try for v1 pipeline #180

Closed
wants to merge 4 commits into from
Closed

First try for v1 pipeline #180

wants to merge 4 commits into from

Conversation

yellowcap
Copy link
Member

@yellowcap yellowcap commented Mar 14, 2024

This is a proof of concept for #178. This is kind of a generic approach that is solely based on STAC, so we could spin this out into a separate pipeline repository and make it standallone for it to be used also in other projects.

Its functional but a bit clunky, but it does show how this could work.

To test it locally:

Create a working directory with an /items subdirectory.

  1. In there, run create_stac.sh, this should create local static STAC items
  2. Then run edit_stac.py this will change default source paths for Landsat to S3 locations.
  3. Then run indexing.py this should create an index json file that represents the items and the chips in them
  4. Then finally the chipping.py script should print chips that are dynamically created from the index.

@yellowcap yellowcap changed the title Fist try for v1 pipeline First try for v1 pipeline Mar 14, 2024
@yellowcap
Copy link
Member Author

I have been working more on the v1 pipeline, realizing that we will need cloud and nodata tracking at the chip level. I played around with creating classes where the platform specific cloud and nodata filters can be added through class method overrides.

the classes are at

https://github.com/Clay-foundation/model/blob/clay-v1-pipeline/scripts/pipeline_v1/chip_indexer.py

script to try them out with the data attached below is

https://github.com/Clay-foundation/model/blob/clay-v1-pipeline/scripts/pipeline_v1/run_chip_indexer.py

test_stac_items.zip

@yellowcap
Copy link
Member Author

This has migrated to
https://github.com/Clay-foundation/stacchip

@yellowcap yellowcap closed this Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant