Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Trino delta lake setup with glue metastore unable to delete delete underlying data from storage #4738

Closed
sumanshusamarora opened this issue Mar 12, 2024 · 0 comments · Fixed by #4739
Closed

[BUG] Trino delta lake setup with glue metastore unable to delete delete underlying data from storage #4738

sumanshusamarora opened this issue Mar 12, 2024 · 0 comments · Fixed by #4739
Labels
bug Something isn't working

Comments

@sumanshusamarora
Copy link
Contributor

Mage version

0.9.65

Describe the bug

Trino delta lake setup with glue metastore unable to delete underlying data from storage when temp tables are dropped. It happens because when property 'with location' is used to create tables, they are considered as external tables. So when these tables are dropped, it drops the meta information but does not delete underlying data. Which means when next time the pipeline runs, it already finds the data sitting in the location specified and raises error about using system.register_table() function.

To reproduce

  1. Setup trino delta lake connector with glue metastore
  2. Provide a location parameter in the trino configuration
  3. Run a integration pipeline once (It should succeed since this is the first time and temp table does not exist)
  4. Run the integration pipeline again

Expected behavior

The pipeline should successfully run

Screenshots

before-update.zip

Operating system

Any

Additional context

No response
@sumanshusamarora sumanshusamarora added the bug Something isn't working label Mar 12, 2024
@wangxiaoyou1993 wangxiaoyou1993 linked a pull request Mar 13, 2024 that will close this issue
Trino temp table location fix - Fixes issue https://github.com/mage-ai/mage-ai/issues/4738 #4739
Merged
2 tasks
wangxiaoyou1993 pushed a commit that referenced this issue Mar 13, 2024
@sumanshusamarora
Trino temp table location fix - Fixes issue #4738 (#4739)
c594367 
* Guide update to setup mage_inetgrations locally

* temp table update

* Documentation update for ignore_location_for_temp_tables

* removed breakpoint

* minor fix to documentation

* fixing flake E501

* hooks changes
@wangxiaoyou1993 wangxiaoyou1993 mentioned this issue Mar 13, 2024
Merged
15 tasks
oonyoontong pushed a commit to bunker-tech/mage-ai that referenced this issue May 2, 2024
@sumanshusamarora @oonyoontong
Trino temp table location fix - Fixes issue mage-ai#4738 (mage-ai#4739)
026d70c 
* Guide update to setup mage_inetgrations locally

* temp table update

* Documentation update for ignore_location_for_temp_tables

* removed breakpoint

* minor fix to documentation

* fixing flake E501

* hooks changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@sumanshusamarora