Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Time-boxed] Create operators to clean temporary tables #436

Closed
kaxil opened this issue Jun 7, 2022 · 1 comment
Closed

[Time-boxed] Create operators to clean temporary tables #436

kaxil opened this issue Jun 7, 2022 · 1 comment
Assignees
Labels
documentation Improvements or additions to documentation feature New feature or request
Milestone

Comments

@kaxil
Copy link
Collaborator

kaxil commented Jun 7, 2022

Time-boxed: 2-3 days

Although Astro SDK records data into temporary tables, it does not offer any tooling for users to clean temporary tables.
We should provide users with a separate Airflow Operators/tasks/decorators that:

  • Identifies temp tables created by tasks in the same Dag Run
  • Deletes temp tables for that dagrun
  • Allows users to selectively specific tables to clean up instead of defaulting all temp tables created in the same dag run.

Open Question:

  • Wait for the delete operation to be complete vs submitting deletion via async and marking task as complete?

Acceptance criteria

  • Have full coverage with tests
  • Have at least one example DAG
  • Document this feature, both using doc strings and in a higher-level in our guides (both REFERENCE.md and TUTORIAL.md including limitations

Related issues:

@kaxil kaxil added documentation Improvements or additions to documentation feature New feature or request labels Jun 7, 2022
@kaxil kaxil added this to the 1.0.0 milestone Jun 7, 2022
@kaxil
Copy link
Collaborator Author

kaxil commented Jun 7, 2022

@dimberman Feel free to add more description and sketch out the task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants