Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make a DAG for pointing ES aliases #3493

Closed
Tracked by #392
stacimc opened this issue Dec 7, 2023 · 0 comments · Fixed by #3890
Closed
Tracked by #392

Make a DAG for pointing ES aliases #3493

stacimc opened this issue Dec 7, 2023 · 0 comments · Fixed by #3890
Assignees
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs

Comments

@stacimc
Copy link
Contributor

stacimc commented Dec 7, 2023

Problem

#2372 adds a DAG for creating ES indices, but does not have any way for promoting the index/applying an alias.

Description

For this issue, we should create a DAG that takes the following conf options:

  • source_index: the name of the index that we want to apply an alias to
  • target_alias: the name of the alias we want to apply
  • delete_old_index: a bool, default to False. Whether to delete the index previously pointed to by the target_alias, if applicable

It should do the following:

  • Attempt to get the current index with the given target_alias (it should skip if no such index exists)
  • If the target alias is already being used, remove it from the index it currently points to.
  • Apply the target_alias to the source_index
  • If delete_old_index is True, and we found an index in the first step, delete it
  • Report to slack

We should package this logic in a reusable TaskGroup, and also add it as an additional step to the create_new_es_index DAG from #2372.

Alternatives

Alternatively, we could trigger this DAG from the create_new_es_index DAG. If possible it would be better to have those steps actually part of the index creation DAG, but simply reuse the code. This makes it much easier to follow when reading through a DagRun.

@stacimc stacimc added 🟨 priority: medium Not blocking but should be addressed soon 🌟 goal: addition Addition of new feature 💻 aspect: code Concerns the software code in the repository 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Dec 7, 2023
@stacimc stacimc mentioned this issue Dec 7, 2023
16 tasks
@stacimc stacimc self-assigned this Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant