Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean up tables on snowflake for temporary table #1663

Closed
sunank200 opened this issue Jan 23, 2023 · 11 comments
Closed

Clean up tables on snowflake for temporary table #1663

sunank200 opened this issue Jan 23, 2023 · 11 comments
Assignees
@rajaths010494
Milestone
1.5.0

Comments

@sunank200
Copy link
Contributor

Describe the bug
Seems like the clean-up is not working properly for snowflake as aspected. There are many tables which are there in https://app.snowflake.com/us-east-1/gp21411/#/data/databases/SANDBOX/schemas/ASTROFLOW_CI/table

@rajaths010494 WDYT?
@phanikumv phanikumv added this to the 1.5.0 milestone Jan 23, 2023
@sunank200
Copy link
Contributor Author

@rajaths010494 could you update your findings and approach you had discussed last week on this ticket please

@rajaths010494
Copy link
Contributor

Cleaned up temp tables in snowflake.
To add some identifiers for temp tables for test cases and see which is causing it.
But when a PR is raised and tests are running, if the user makes a commit again, then the tests are stopped, in which case the temp tables aren't deleted.

@sunank200
Copy link
Contributor Author

@rajaths010494 lets add an issue to clean up the snowflake tables for failed runners as discussed.

@sunank200
Copy link
Contributor Author

Also, could you post the screenshot of cleaned-up tables from snowflake UI, please?

@rajaths010494
Copy link
Contributor

clean up the snowflake tables for failed runners

#1682.

@rajaths010494
Copy link
Contributor

When tmp tables exist
Screenshot 2023-01-30 at 6 37 54 PM

After deletion
Screenshot 2023-01-30 at 6 47 05 PM

@sunank200 sunank200 mentioned this issue Jan 31, 2023
Open
@sunank200
Copy link
Contributor Author

@rajaths010494 as discussed we concluded to use implementation similar to astronomer-providers for the clean-up

@rajaths010494
Copy link
Contributor

Have added an dag to clean up resources as part of astro cloud running all of our example dags

1 similar comment
@rajaths010494
Copy link
Contributor

Have added an dag to clean up resources as part of astro cloud running all of our example dags

@sunank200
Copy link
Contributor Author

#913

@sunank200 sunank200 mentioned this issue Feb 6, 2023
Closed
@sunank200
Copy link
Contributor Author

This PR: #1499 adds the clean DAG as part of CI. Follow up ticket added to create named tables as part of tests

@rajaths010494 rajaths010494 mentioned this issue Feb 13, 2023
Merged
rajaths010494 added a commit that referenced this issue Feb 20, 2023
@rajaths010494
Cleanup non deleted Snowflake temp tables on CI (#1758)
c92849a 
**Describe the bug**
Create a named temp table on the test with the identifier for each test
for cleanup. More context on
#1663


closes #1712
utkarsharma2 pushed a commit that referenced this issue Feb 21, 2023
@rajaths010494 @utkarsharma2
Cleanup non deleted Snowflake temp tables on CI (#1758)
bdf8ff1 
**Describe the bug**
Create a named temp table on the test with the identifier for each test
for cleanup. More context on
#1663


closes #1712
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rajaths010494 rajaths010494

Labels
None yet
Projects
None yet
Milestone
1.5.0
Development

No branches or pull requests
3 participants
@sunank200 @rajaths010494 @phanikumv