Import workflow fixes

[vish-cs]

• Use PENDING state to track imports with failed ingestion for retry: update spanner-ingestion-workflow to mark imports as PENDING after a failure.
• Update spanner client to advance the stale read timestamp only if there are no PENDING imports. Refactor update_ingestion_history into multiple methods updating each table.
• Update get_import_info to support importList as override: this enables running ingestion for specific imports
• Use cloud scheduler job to determine next_refresh instead of passing schedule string in the import config.
• Handle duplicate events in CDA feed handler. Feed handler can have duplicate messages delivered so use GCS to detect duplicates
• Reactored aggregation-helper/main.py to move helper functions to import_herlper.py
• Update import automation workflow to add default resources and job name to avoid passing them in import config.
• Add handling for provenance mcf generation: Copy provenance MCF files from GCS. Generate a default metadata file if none found
• Add spanner schema for import status metadata tables
• Add a field to track failed ingestion in IngestionHistory table for mixer to read stale read timestamp
• Cloud build config fixes
• Other clean up: Add python doc strings for various methods

[gemini-code-assist]

Code Review

This pull request aims to fix and enhance the import workflow by adding retry logic and improving import list fetching. However, it introduces significant security vulnerabilities, including path traversal in StorageClient methods, which could allow writing blobs to arbitrary GCS locations, and an MCF injection vulnerability due to unsanitized user input in metadata generation. Furthermore, a critical bug in ingestion-helper/main.py prevents import status updates, and there's a medium-severity suggestion for code clarity. It is imperative to address these security flaws and functional issues to ensure the workflow's integrity and secure operation.