Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: Onboard Cloud Storage Geo Index #367

Merged
merged 10 commits into from
Jun 2, 2022
Merged

Feat: Onboard Cloud Storage Geo Index #367

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5e17ce5
fix: Added partitioning and clustering. Not production ready.
nlarge-google May 25, 2022
d4cb285
fix: Works for landsat_index
nlarge-google May 25, 2022
f881750
fix: Added sentinel-2 pipeline as well as clustering.
nlarge-google May 25, 2022
fe6fe07
fix: Clean up.
nlarge-google May 26, 2022
d4525ba
fix: attempt to resolve pod not started.
nlarge-google May 27, 2022
ca05fd0
fix: various fixes to pipeline.yaml which caused 'pod not started' is…
nlarge-google May 27, 2022
7142fd2
fix: Resolve resource issues in pipeline.yaml.
nlarge-google May 27, 2022
75c146e
fix: Moved schema JSON files as per code review request.
nlarge-google May 27, 2022
ec2e26a
fix: Exposed variable values into pipeline.yaml as per code review.
nlarge-google May 27, 2022
f25e268
Update pipeline.yaml
adlersantos Jun 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
104 changes: 104 additions & 0 deletions ...pipelines/_images/run_csv_transform_kub/cloud_storage_geo_index_landsat_index_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
[
{
"name": "scene_id",
"type": "STRING",
"mode": "REQUIRED"
},
{
"name": "product_id",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "spacecraft_id",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "sensor_id",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "date_acquired",
"type": "DATE",
"mode": "NULLABLE"
},
{
"name": "sensing_time",
"type": "TIMESTAMP",
"mode": "NULLABLE"
},
{
"name": "collection_number",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "collection_category",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "data_type",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "wrs_path",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "wrs_row",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "cloud_cover",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "north_lat",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "south_lat",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "west_lon",
"type": "FLOAt",
"mode": "NULLABLE"
},
{
"name": "east_lon",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "total_size",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "base_url",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "source_url",
"type": "STRING",
"description": "Source ",
"mode": "NULLABLE"
},
{
"name": "etl_timestamp",
"type": "TIMESTAMP",
"description": "Load time for this data row",
"mode": "NULLABLE"
}
]
84 changes: 84 additions & 0 deletions ...ex/pipelines/_images/run_csv_transform_kub/cloud_storage_geo_index_sentinel_2_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
[
{
"name": "granule_id",
"type": "STRING",
"mode": "REQUIRED"
},
{
"name": "product_id",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "datatake_identifier",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "mgrs_tile",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "sensing_time",
"type": "TIMESTAMP",
"mode": "NULLABLE"
},
{
"name": "geometric_quality_flag",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "generation_time",
"type": "TIMESTAMP",
"mode": "NULLABLE"
},
{
"name": "north_lat",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "south_lat",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "west_lon",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "east_lon",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "base_url",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "total_size",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "cloud_cover",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "source_url",
"type": "STRING",
"description": "Source ",
"mode": "NULLABLE"
},
{
"name": "etl_timestamp",
"type": "TIMESTAMP",
"description": "Load time for this data row",
"mode": "NULLABLE"
}
]