Reingestion workflows report misleading date range to Slack

[stacimc]

Description

At the end of all provider workflows we report the number of records ingested to Slack. This slack message also includes information about the date range covered by the run: https://github.com/WordPress/openverse/blob/main/catalog/dags/providers/provider_dag_factory.py#L337

For reingestion workflows this range does not make sense. Most are run on a weekly basis, so the Slack message reports the week as the data interval; in actuality the DAG is performing ingestion for a series of many disparate reingestion days.

The most sensible thing to do is probably either:

• change the text of "date range" to something less suggestive. "data interval" is misleading in the same way, unfortunately.
• remove this part of the message entirely for reingestion workflows

Screenshots

[BaileyMcKelway]

Hi all, I was hoping to start contributing and I am looking for a first issue. Would updating the create_report_load_completion function to something like this be a proper solution?

It checks for if the DAG has "reingestion" in the id and then conditional adds the date_range_start and date_range_end.

@stacimc @AetherUnbound

def create_report_load_completion(
    dag_id,
    media_types,
    ingestion_metrics,
    dated,
):
    is_reingestion_workflow = "reingestion" in dag_id

    op_kwargs = {
        "dag_id": dag_id,
        "media_types": media_types,
        "duration": ingestion_metrics["duration"],
        "record_counts_by_media_type": ingestion_metrics["record_counts_by_media_type"],
        "dated": dated,
        "is_reingestion_workflow": is_reingestion_workflow,
    }

    if not is_reingestion_workflow:
        op_kwargs["date_range_start"] = "{{ data_interval_start | ds }}"
        op_kwargs["date_range_end"] = "{{ data_interval_end | ds }}"

    return PythonOperator(
        task_id="report_load_completion",
        python_callable=reporting.report_completion,
        op_kwargs=op_kwargs,
        trigger_rule=TriggerRule.ALL_DONE,
    )

[AetherUnbound]

@BaileyMcKelway welcome! That definitely looks like an appropriate start! I have a few suggestions around the structure of the Python itself, but those can be made directly on the PR. Do you mind opening a pull request for this particular change?

[BaileyMcKelway]

@AetherUnbound Thanks!

I have a PR up here #4093.

It is currently in draft because I have a few questions on how to properly test my change. Currently all tests are passing for the DAGs.

I guess it would be better to ask these questions on the Slack? I tried to register an account last week ago and I am still waiting approval.

[AetherUnbound]

That's...really odd 😮 Slack would indeed be the best way, but it's unfortunate you can't get connected 😕 Do feel free to ask your questions here, or if you'd prefer to have a 1:1 conversation about it you can message me on LinkedIn if that's easier!

[BaileyMcKelway]

@AetherUnbound Everything is now sorted! I signed up with a different email and was approved instantly. I now have access to Slack. Not sure why my main gmail didn't work.