DAGS

There are, at the time of writing, 6 DAGS:

3 single finding aid DAGS

Index Finding Aid DAG

1. Make s3 key for temporary working directory s3_key = f"indexing/{finding_aid_id}/{datetime.now().isoformat()}"
2. Prepare finding aid in CincoCtrl container: python manage.py prepare_finding_aid --finding_aid_id {finding_aid_id} --s3_key {s3_key}
3. Index finding aid in ArcLight container bin/index-from-s3 {finding_aid_id} {s3_key} {repository_code} {finding_aid_ark} {eadid} {preview}
4. Clean Up S3 working directory delete_results = bucket.objects.filter(Prefix=prefix).delete()
5. Request static finding aid rebuild in ArcLight container bin/generate-static-findaid {finding_aid_ark}
6. Wait 1 minute with a TimeDeltaSensor
7. Clear Cloudfront cache:

invalidation_paths = [
    f"/findaid/{finding_aid_ark}*", 
    f"/findaid/static/{finding_aid_ark}*"
]

Delete Finding Aid DAG

1. Remove from index in ArcLight container: bin/remove-from-solr {finding_aid_ark} {repository_code}
2. Remove from database in CincoCtrl container: python manage.py remove_finding_aid --ark {finding_aid_ark}
3. Remove static finding aids from S3:

paths = [
    f"static_findaids/static_findaids/{ark}",
    f"static_findaids/oac4/{ark}",
    f"static_findaids/oac5/{ark}",
]

Unpublish Finding Aid DAG

1. Remove from index in ArcLight container: bin/remove-from-solr {finding_aid_ark} {repository_code}
2. Mark unpublished in CincoCtrl container: python manage.py mark_unpublished --ark {finding_aid_ark}

2 Polling DAGS

Poll Airflow Api

1. Poll Airflow API from Stage CincoCtrl Container: python manage.py poll_airflow
2. Poll Airflow API from Prod CincoCtrl Container: python manage.py poll_airflow

Poll Message Queue

1. Poll SQS Message Queue from Stage CincoCtrl Container: python manage.py poll_sqs
2. Poll SQS Message Queue from Prod CincoCtrl Container: python manage.py poll_sqs

2 Bulk DAGS

Bulk Index Finding Aids DAG

TODO: Have not added static finding aid rebuild to this DAG

1. Make s3 key for temporary working directory s3_key = f"indexing/bulk/{s3_prefix}"
2. Bulk prepare finding aids in CincoCtrl Container: python manage.py bulk_prepare_finding_aids --filters {queryset_filters} --s3_key {s3_key} --max_num_records {max_num_records} --max_file_size_in_MB {max_file_size_in_MB}
3. Get list of finding aid batches from s3 [batch.get("Prefix").replace("media/", "").strip("/") for batch in s3.list_objects_v2(Bucket=bucket_name, Prefix=f"media/{s3_key}/", Delimiter="/").get('CommonPrefixes', [])]
4. Run bulk indexing and s3 Cleanup as a task group in parallel for each batch of finding aids a. Bulk index batch of finding aids in Arclight container: bin/bulk-index-from-s3 {batch_s3_key} b. Cleanup s3 delete_results = bucket.objects.filter(Prefix=prefix).delete()

Bulk Delete Finding Aids DAG

1. Bulk remove finding aids from solr in ArcLight container: bin/bulk-remove-from-solr {s3_key}
2. Bulk remove finding aids from database in CincoCtrl container: python manage.py bulk_remove_finding_aids --s3_key {s3_key}
3. Bulk remove static finding aids from s3:

removals = s3.get_object(Bucket=bucket_name, Key=s3_key)["Body"].read().decode("utf-8").splitlines()
for line in removals:
    ark, _ = line.split(",", 1)
    prefixes = [
        f"static_findaids/static_findaids/{ark}",
        f"static_findaids/oac4/{ark}",
        f"static_findaids/oac5/{ark}",
    ]
    for prefix in prefixes:
        delete_results = bucket.objects.filter(Prefix=prefix).delete()