Prune path before using a CTA

[nicor88]

What

to avoid: HIVE_PATH_ALREADY_EXISTS, ensure that the path is pruned. Check implementation here, in our case we can build something similar.

Iceberg table should avoid to use that, as the table drop take care of pruning the location.

[nicor88]

@antruo this fix should also cover tmp tables right??

[0m15:55:16    HIVE_PATH_ALREADY_EXISTS: Target directory for table 'default.parquet_example_3__dbt_tmp' already exists: s3://silver-datalake-dev-xxxxx-eu-central-1/parquet_example_3__dbt_tmp. You may need to manually clean the data at location 's3://athena-query-results-dev-xxxxxx-eu-central-1/dbt/tables/2c1d4480-15a0-4b80-bdf4-ba43c0c028af' before retrying. Athena will not delete data in your account.
[0m15:55:16 HIVE_PATH_ALREADY_EXISTS: Target directory for table 'default.parquet_example_3__dbt_tmp' already exists: s3://silver-datalake-dev-901621332207-eu-central-1/parquet_example_3__dbt_tmp. You may need to manually clean the data at location 's3://athena-query-results-dev-xxxxxxx-eu-central-1/dbt/tables/2c1d4480-15a0-4b80-bdf4-ba43c0c028af' before retrying. Athena will not delete data in your account.

I got this with 1.3.3 version,

[antruo]

Yes! I guess you are referring to the temp tables that are being created when running an incremental model? That one just uses the create_table_as() macro, which runs the prune_s3_table_location() prior to the CTA statement.

[andrewcstewart]

Are there other known causes for the HIVE_PATH_ALREADY_EXISTS that the path pruning isn't addressing?