Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Athena iceberg tables have trailing slash in s3 path #1226

Closed
romanperesypkin opened this issue Apr 16, 2024 · 1 comment · Fixed by #1230
Closed

Athena iceberg tables have trailing slash in s3 path #1226

romanperesypkin opened this issue Apr 16, 2024 · 1 comment · Fixed by #1230

Comments

@romanperesypkin
Copy link
Contributor

romanperesypkin commented Apr 16, 2024

dlt version

0.4.8

Describe the problem

Athena code to create table_prefix for iceberg table has trailing slash which leads to a path like s3://*/_dlt_version//metadata.

Here is the query DLT generates to create _dlt_version.

CREATE TABLE `bronze_facebook`.`_dlt_version`
(
    `version` bigint,
    `engine_version` bigint,
    `inserted_at` timestamp,
    `schema_name` string,
    `version_hash` string,
    `schema` string
)
LOCATION 's3://data-bronze-bucket/facebook/bronze_facebook/_dlt_version/'
TBLPROPERTIES ('table_type'='ICEBERG', 'format'='parquet');

That tiny slash at the end of _dlt_version causes all the problems when DLT tries to write any data to the table

Expected behavior

DLT removes trailing slash for Athena iceberg tables

Steps to reproduce

Just create simple source with resource which writes to Athena

Operating system

Linux

Runtime environment

Airflow

Python version

3.11

dlt data source

No response

dlt destination

AWS Athena / Glue Catalog

Other deployment details

No response

Additional information

No response

@romanperesypkin
Copy link
Contributor Author

romanperesypkin commented Apr 16, 2024

Seems like we have either to add a 'simple' fix to remove trailing slash directly to create statement or update path_utils.get_table_prefix_layout function to be smart enough

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant