Fix: Resolve CRON value in dataset Cloud Storage Geo Index (#413)

* fix: Added partitioning and clustering. Not production ready.

* fix: Works for landsat_index

* fix: Added sentinel-2 pipeline as well as clustering.

* fix: Clean up.

* fix: attempt to resolve pod not started.

* fix: various fixes to pipeline.yaml which caused 'pod not started' issues.  Added bigquery to requirements.txt

* fix: Resolve resource issues in pipeline.yaml.

* fix: Moved schema JSON files as per code review request.

* fix: Exposed variable values into pipeline.yaml as per code review.

* Update pipeline.yaml

* fix: Resolved older reference in cron value.

Co-authored-by: Adler Santos <adlersantos@google.com>

datasets/cloud_storage_geo_index/pipelines/cloud_storage_geo_index/cloud_storage_geo_index_dag.py

@@ -27,7 +27,7 @@
     dag_id="cloud_storage_geo_index.cloud_storage_geo_index",
     default_args=default_args,
     max_active_runs=1,
-    schedule_interval="0 6 * * 1",
+    schedule_interval="* 1 * * 6",
     catchup=False,
     default_view="graph",
 ) as dag:

datasets/cloud_storage_geo_index/pipelines/cloud_storage_geo_index/pipeline.yaml

@@ -30,7 +30,7 @@ dag:
       depends_on_past: False
       start_date: '2021-03-01'
     max_active_runs: 1
-    schedule_interval: "0 6 * * 1"  # 06:00 on Monday
+    schedule_interval: "* 1 * * 6" # 06:00 on Monday
     catchup: False
     default_view: graph