Skip to content

Latest commit

 

History

History
63 lines (50 loc) · 3.33 KB

README.md

File metadata and controls

63 lines (50 loc) · 3.33 KB
Raw

Coverage Status

libsys-airflow

Airflow DAGS for libsys processes and migrating ILS data into FOLIO

Dependency Management and Packaging

Run pip install -r requirements.txt to install the dependencies.

Running Locally with Docker

Based on the documentation, Running Airflow in Docker.

NOTE Make sure there is enough RAM available locally for the docker daemon, we recommend at least 5GB.

  1. Clone repository git clone https://github.com/sul-dlss/libsys-airflow.git
  2. If it's commented out, uncomment the line - ./dags:/opt/airflow/dags in docker-compose.yaml (under volumes, under x-airflow-common).
  3. Start up docker locally.
  4. Build the docker image with Docker build .
  5. Create a .env file with the AIRFLOW_UID and AIRFLOW_GROUP values.
  6. Run docker-compose build to build the customized airflow image. (Note: the usermod command may take a while to complete when running the build.)
  7. Run docker compose up airflow-init to initialize the Airflow
  8. Bring up airflow, docker compose up to run the containers in the foreground, use docker compose up -d to run as a daemon.
  9. Access Airflow locally at http://localhost:8080
  10. Log into the worker container using docker exec -it libsys-airflow_airflow-worker-1 /bin/bash to view the raw work files.

Deploying

Prerequisites

  1. Install pip3 with apt install python3-pip
  2. Install python virtual enviroments: apt install python3.8-venv
  3. Create the virtual envirnment in the home directory: python3 -m venv virtual-env
  4. Install docker-compose in the virtual environment: source virtual-env/bin/activate && pip3 install docker-compose

Tasks

  1. List all the airflow tasks using cap -AT airflow
  2. cap {stage} airflow:build
  3. cap {stage} airflow:init
  4. cap {stage} airflow:start

For FOLIO migration loads

  1. In the Airflow UI under Admin > Connections, add bib_path with connection type File (Path).
  2. In the Airflow UI under Admin > Variables, import the folio-dev-variables.json file from shared_configs.

For Aeon and Lobbytrack API calls

  1. In the Airflow UI under Admin > Variables, import the aeon-variables.json and the lobbytrack-variables.json files from shared_configs.

FOLIO Plugin

All FOLIO related code should be in the folio plugin. When developing code in the plugin, you'll need to restart the airflow-webserver container by running docker-compose restart airflow-webserver to see changes in the running Airflow environment.

Testing

To run the test suite, use pytest passing in the location of where you have local clone repository of the folio_migration_tools/with the PYTHONPATH i.e.

PYTHONPATH='{path-to-folio_migration_tools}' pytest

To run with code coverage: PYTHONPATH=../folio_migration_tools/ coverage run -m pytest

Symphony Mount

MARC data to be converted will be mounted on the sul-libsys-airflow server under /sirsi_prod which is a mount of /s/SUL/Dataload/Folio on the Symphony server.