Fix: saving compressed load files with .gz extension

[anuunchin]

Description

This PR adds the .gz extension to compressed files based on a new config disable_extension that is set to True by default.

In a nutshell:

• is_compression_disabled() is removed and destination load jobs have access to BufferedDataWriterConfiguration.
• Since the task involves getting rid of tricks like FileStorage.is_gzipped(), solely relying on BufferedDataWriterConfiguration is not sufficient when it comes to imported files, since they must not be compressed in any situation. For this reason, a new context class FileImportContext was introduced that tracks whether a file is an imported file and, thus, does not require compression. The is_compressed_file attribute of FileImportContext is then accessed by the duckdb and clickhouse load jobs to correctly set the compression type.

Related Issues

• Resolves Save compressed load files with .gz extension #925

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	1f7da2e
🔍 Latest deploy log	https://app.netlify.com/projects/dlt-hub-docs/deploys/6877b40ad10c770008ceb5c6

[anuunchin]

This might be redundant. We can just check is self.open is gzip_open

[anuunchin]

I'm not 100% sure if this makes sense because it runs in every case except one corner-case with empty schema and empty dataset_name 🤔

[anuunchin]

This test is not exhaustive, since some writers are missing, but i don't think it's a problem for the first round of reviews 👀

[rudolfix]

the direction is good and the only complication is backward compat for filesystem destination. I think @sh-rp wants to chip-in here.

with this PR we can remove some of weird things we do to discover if files are zipped.

is_compression_disabled() - this function should go away and whenever it is used there's something wrong. we are better if we use extension to guess if file is zipped.

another so so pattern used for example in clikckhouse.py

if ext == "jsonl":
                compression = "gz" if FileStorage.is_gzipped(file_path) else "none"

we actually probe the file to see if it is zipped

we remove all those tricks and just use extensiosn. the only place is filesystem destination where we must decide how/if we do backward compatibility

[sh-rp]

I did not read the PR, but just from a user / compatibility perspective, I think dlt should behave the exact same way with regards to adding these extensions after updating if you do not change any settings. This means:

• adding the gzip extension must be configurable
• This setting must be set to off by default
• It would be nicer to have it switched on by default, but then we would need to make this change in dlt 2.0 and mark it as breaking

[sh-rp]

@anuunchin what I would do is the following: Use the .gz extension by default everywhere because it makes stuff simpler and then remove the .gz extensions for the cases where the filesystem is the main destination and the user has selected to not keep the extension there (which should be the default). So as we discussed:

• Always add or keep .gz extension in the DataWriter if compression is enabled, so after extracing and normalizing we have .gz on files that are compressed
• When importing files, probe the file and add .gz if compressed
• Add a new setting to the filesystem destination that controls wether the files should have a .gz compression if compressed, it be set to remove this extension by default to ensure backward compatibility. This setting should probably be ignored if the filesystem is used as the staging destination.
• If this setting is set, rename files that are copied to buckets or the local filesystem to not have this extension while actually running the copy job.

Things to check:

• This approach assumes that the users are ok with having the .gz extension on files in their staging buckets. Please confirm with @rudolfix that we can do this.
• Check if this has any influence on delta and iceberg. I think we output parquet files in the normalizer for both of these destinations, so we should be good there
• Athena (without iceberg) uses the filesystem destination as staging, but uses those exact files for backing the tables with data. In this case, the above setting should be respected.