Skip to content

feat: Update Episcanner-Downloader DAG with Improved PSQL Connection and Schedule#25

Merged
esloch merged 4 commits intoAlertaDengue:mainfrom
esloch:fix-episcanner-dag
May 18, 2023
Merged

feat: Update Episcanner-Downloader DAG with Improved PSQL Connection and Schedule#25
esloch merged 4 commits intoAlertaDengue:mainfrom
esloch:fix-episcanner-dag

Conversation

@esloch
Copy link
Copy Markdown
Collaborator

@esloch esloch commented May 15, 2023

This pull request updates the Episcanner-Downloader DAG. The following changes have been made:

  • Improved type hints and docstrings for better code readability and maintainability.
  • Set variables for establishing a connection between Episcanner and PostgreSQL database, allowing seamless data retrieval and storage.
  • Updated the schedule to run the DAG every Sunday at 3 AM, ensuring timely and consistent data processing.
  • Refactored the cloning of the repository and installation of Poetry, streamlining the process and enhancing efficiency.
  • Renamed the task "set_variables_task" to "set_connection_variables" to reflect its purpose accurately.
  • Updated the Dockerfile to include necessary dependencies and configurations for the Episcanner-Downloader APP.
  • Set catchup to False to prevent backfilling of missed task instances.
  • Modify the data source parameter to export data for all available sources.
  • Add a task to remove the episcanner-downloader repository after data export.

@esloch esloch requested a review from luabida May 15, 2023 17:57
@esloch esloch force-pushed the fix-episcanner-dag branch from d1ecf39 to 8d7e6ad Compare May 16, 2023 15:05
@esloch esloch changed the title feat: Correct issues in episcanner-downloader DAG feat: Update Episcanner-Downloader DAG with Improved PSQL Connection and Schedule May 16, 2023
@esloch esloch marked this pull request as ready for review May 16, 2023 15:13
bash_command='git clone --branch main --single-branch --depth 1 '
'https://github.com/AlertaDengue/episcanner-downloader.git '
'/opt/airflow/episcanner-downloader',
dag=dag,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using the DAG context manager, it is not necessary to set the dag variable to the tasks

bash_command='git clone --branch main --single-branch --depth 1 '
'https://github.com/AlertaDengue/episcanner-downloader.git '
'/opt/airflow/episcanner-downloader',
dag=dag,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using the DAG context manager, it is not necessary to set the dag variable to the tasks

@esloch esloch requested a review from fccoelho May 17, 2023 18:38
@esloch esloch merged commit e9cbbf4 into AlertaDengue:main May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants