[New Feature]: Create DAG to execute the Application Package generator

[LucaCinquini]

PM Ticket Reference: unity-sds/unity-app-build-trigger#2

Description:

Convert App Builds system into a docker container runnable in Airflow - Allows user to provide tokens, credentials. See Status of builds. Etc. Initial version should be what we're doing today. enhancements (future) should include pointing to an eCR repository, non-docker repo, etc. Needs Documentation. Should this be its own DAG that can run 'faster' than the cwl operators (e.g. SPOT instances).

Acceptance Criteria:

Docker Container Conversion: The App Builds system must be successfully converted into a Docker container.
Airflow Integration: Ensure the Docker container runs smoothly within the Airflow environment.
Validation and Testing: The Docker container and its integration with Airflow must be thoroughly validated and tested to ensure it operates without errors.
Documentation: Provide detailed documentation on the app-pack process, including steps taken, and configurations applied

Dependencies:
From U-ADS: Docker container to execute the Application Package Generation, together with complete documentation.

[LucaCinquini]

Goal (restated): follow James’s manual instructions to create a Docker container and DAG that will replace the automatic build system on GitLab (invoked via the Unity-py API)

As discussed in meeting with @mcduffie , the following steps will be required:

• Preliminary: publish the repos: “unity-app-generator” and “unity-pack-generator” to PyPI
• Build a Docker container that executes the following steps, run manually on laptop
  • build_ogc_app init
  • build_ogc_app build_docker --no_owner
  • build_ogc_app push_docker to GitHub
• Build a DAG to execute that Docker container via UI
• Update the current Lambda to invoke the DAG. The Lambda is invoked via the existing Unity API.
• Future optional: add a Github Action to push automatically push any changes to the “unity-app-generator” and “unity-pack-generator” repositories to PyPI

References:

• How to build the app-package generator manually: https://unity-sds.gitbook.io/docs/mdps-overview/tutorials/algorithms-and-algorithm-development/packaging-an-algorithm
• How to invoke the current automatic build system: https://github.com/unity-sds/mdps-tutorial/blob/main/jupyter-notebooks/tutorials/5_building_application_packages.ipynb
• Example slide: https://docs.google.com/presentation/d/1CrUsnoKpCkQTECEjHZz4ysduBfrkDH4APcx4CmtYJXM/edit#slide=id.g3092c7e18ae_0_42

[grallewellyn]

I completed this ticket. Here are some notes about it:

DockerHub and Dockstore credentials are being stored in AWS SSM parameter store. I uploaded for Unity-venue-dev, Unity-venue-test and Unity-venue-ops

For Unity-venue-ops, I added the access tokens as a secure string because this seemed to be following the convention that ops was already doing. I don’t access the parameters any differently, but costs might apply: AWS Systems Manager Parameter Store - AWS Systems Manager If we don’t want to store tokens as secure strings, we should discuss that

These DockerHub and Dockstore accounts were created with the email mdps.dockerhub at jpl.nasa.gov which is our new shared dev team email

I published the Docker image that we are using in the DAG to our jplmdps account: https://hub.docker.com/r/jplmdps/unity-app-gen

I tested the generated process.cwl file with a CWL DAG Modular

My code for the Dockerfile is here: unity-sds/unity-app-generator#31 and my code for the DAG is here: #390

Note that the Dockerfile will need to change when we want to deploy and start using v1.1.0 of unity app generator and app pack generator (needs to be deployed on pip when changes to these packages are finished)

The logs are not displaying any sensitive data such as DockerHub and Dockerstore tokens

We can add "Application Package Generator DAG" as an option for users and close this ticket once my PRs are merged:
unity-sds/unity-app-generator#31
#390

[mcduffie]

I don't see the DAG file at the URL in the comment above. Also shouldn't the DAG file live in the SPS repo?

[grallewellyn]

Sorry, I originally had the DAG there then realized it makes more sense in the SPS repo so made the PR for it there: #390
I went back and modified all the comments on issues related to this ticket which were referencing just the Dockerfile