RD-WATCH/Smartflow Integration

[mvandenburgh]

This PR adds some configuration needed to get RD-WATCH to communicate with SmartFlow. The bulk of the needed changes are done in the infrastructure repo (https://smartgitlab.com/kitware/rgd_aws_infrastructure/-/merge_requests/11), while this PR will just add some initial configuration to demonstrate a proof-of-concept DAG kicked off from RD-WATCH.

[BryonLewis]

You can feel free to ignore most of the comments, the only things I think may be necessary before merging are:

• The cd.yaml update to make sure it creates the image
• The healthcheck port change
• Not Sure as to why the :latest tag didn't work for me but possibly pinning it to prevent future breaking changes.

[BryonLewis]

this is hardcoded so in the .env file should we make a note of this? It isn't like other containers where the username/password is parameterized. Or maybe we should use the airflow environment variables?

https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#environment-variables-supported-by-docker-compose

[mvandenburgh]

I saw this, but I couldn't get it to work so I just gave up and did that command in the docker file. Does it work for you?

[BryonLewis]

took a little searching but I think you can remove the local airflow DockerFile. These _ based commands are only meant for local development, but I assume your infrastructure code handles all this.
I wouldn't worry about implementation now, just logging this comment for the future.

  airflow:
    image: apache/airflow:slim-latest-python3.12
    volumes:
      - airflow-data:/opt/airflow
      - type: bind
        source: ${PWD}/dev/airflow_sample_dag.py
        target: /opt/airflow/dags/airflow_sample_dag.py
    environment:
      AIRFLOW_HOME: "/opt/airflow/"
      _AIRFLOW_DB_MIGRATE: True
      _AIRFLOW_WWW_USER_CREATE: True
      _AIRFLOW_WWW_USER_USERNAME: "rdwatch"
      _AIRFLOW_WWW_USER_PASSWORD: "rdwatch"
    command: ["airflow", "standalone"]
    ports:
      - 8002:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s

[BryonLewis]

I figured out the default password/username for airflow when running locally which could allow us to remove the airflow.Docker but I wouldn't worry about it for now. I just added a comment for future reference.