Fix Airflow API request to patch or update a DAG

[nikki-t]

Purpose

Update OGC API replace method to copy over DAG to Airflow and send an update request for the DAG to the Airflow API

The developer workflow is as follows:

1. Update DAG and push to GitHub
   a) Repo: https://github.com/unity-sds/unity-sps.git
   b) Branch: main
2. Make an OGC API request to "replace" or update the DAG: https://docs.ogc.org/DRAFTS/20-044.html#_scope
   a) PUT request to /processes/{processID} endpoint
   b) 204 No Content is returned
3. DAG code is updated and enabled/disabled toggle state is respected

Proposed Changes

• [CHANGE] Modify replace function (PUT request) to send a PATCH request to the Airflow API to update the DAG

Issues

• #408 - [New Feature]: Improve the OGC registration process

Testing

Ran locally to debug and confirm response from PUT method.

Ran on nikki-3 deployment in unity-venue-dev:

1. Built OGC API container: "ghcr.io/unity-sds/unity-sps-ogc-processes-api/unity-sps-ogc-processes-api:408-ogc-registration-replace"
2. Updated git-sync repo to point to 408-ogc-registration-replace branch and pushed a DAG with a small change
3. Deployed to nikki-3
4. Ran PUT request to test replace (or update) of the DAG and confirmed the DAG code had changed and the enabled/disabled toggle was respected

Sample query:

curl --location --request PUT 'https://v3oj1n6s3g.execute-api.us-west-2.amazonaws.com/dev/ogc/api/processes/cwl_dag' \
--header 'Expect;' \
--header 'Content-Type: application/json; charset=utf-8' \
--header 'Authorization: Bearer ************' \
--data '{
  "executionUnit": {
    "image": "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.6.0",
    "type": "docker"
  },
  "processDescription": {
    "description": "This process executes any CWL workflow.",
    "id": "cwl_dag",
    "inputs": {
      "cwl_args": {
        "description": "The URL of the CWL workflow'\''s YAML parameters file",
        "maxOccurs": 1,
        "minOccurs": 1,
        "schema": {
          "format": "uri",
          "type": "string"
        },
        "title": "CWL Workflow Parameters URL"
      },
      "cwl_workflow": {
        "description": "The URL of the CWL workflow",
        "maxOccurs": 1,
        "minOccurs": 1,
        "schema": {
          "format": "uri",
          "type": "string"
        },
        "title": "CWL Workflow URL"
      },
      "log_level": {
        "description": "Parameter to set the log verbosity (default: INFO)",
        "maxOccurs": 1,
        "minOccurs": 1,
        "schema": {
          "type": "string"
        },
        "title": "Log Level"
      },
      "request_instance_type": {
        "description": "The specific EC2 instance type requested for the job",
        "maxOccurs": 1,
        "minOccurs": 1,
        "schema": {
          "type": "string"
        },
        "title": "Requested EC2 Type"
      },
      "request_storage": {
        "description": "The amount of storage requested for the job",
        "maxOccurs": 1,
        "minOccurs": 1,
        "schema": {
          "type": "string"
        },
        "title": "Requested Storage"
      }
    },
    "jobControlOptions": [
      "async-execute"
    ],
    "outputs": {
      "result": {
        "description": "The result of the CWL Workflow execution",
        "schema": {
          "$ref": "some-ref"
        },
        "title": "Process Result"
      }
    },
    "title": "Generic CWL Process",
    "version": "1.0.0"
  }
}
'

[LucaCinquini]

I was not able to explicitly test the new feature because of proxy issues, but the code looks correct. Could you please:
o Create a new tag 2.1.0 for the OGC container, and set it as default in the SPS deployment
o Update the instructions on GitBook
Thanks!