Skip to content

Redshift - remove extra cleanup on unload_to_files #935

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 5, 2021
Merged

Redshift - remove extra cleanup on unload_to_files #935

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2032395
Redshift - avoid extra cleanup on UNLOAD
kukushking Sep 29, 2021
eefe3d2
Merge branch 'main' into feat-unload-no-cleanup
kukushking Sep 29, 2021
e6f9c2c
Merge branch 'main' into feat-unload-no-cleanup
kukushking Sep 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions awswrangler/redshift.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -864,7 +864,6 @@ def unload_to_files(
max_file_size: Optional[float] = None,
kms_key_id: Optional[str] = None,
manifest: bool = False,
use_threads: bool = True,
partition_cols: Optional[List[str]] = None,
boto3_session: Optional[boto3.Session] = None,
) -> None:
Expand Down Expand Up @@ -910,9 +909,6 @@ def unload_to_files(
kms_key_id : str, optional
Specifies the key ID for an AWS Key Management Service (AWS KMS) key to be
used to encrypt data files on Amazon S3.
use_threads : bool
True to enable concurrent requests, False to disable multiple threads.
If enabled os.cpu_count() will be used as the max number of threads.
manifest : bool
Unload a manifest file on S3.
partition_cols: List[str], optional
Expand Down Expand Up @@ -941,7 +937,6 @@ def unload_to_files(
if unload_format not in [None, "CSV", "PARQUET"]:
raise exceptions.InvalidArgumentValue("<unload_format> argument must be 'CSV' or 'PARQUET'")
session: boto3.Session = _utils.ensure_session(session=boto3_session)
s3.delete_objects(path=path, use_threads=use_threads, boto3_session=session)
with con.cursor() as cursor:
format_str: str = unload_format or "PARQUET"
partition_str: str = f"\nPARTITION BY ({','.join(partition_cols)})" if partition_cols else ""
Expand All @@ -955,7 +950,7 @@ def unload_to_files(
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
boto3_session=boto3_session,
boto3_session=session,
)

sql = (
Expand Down Expand Up @@ -1106,7 +1101,6 @@ def unload(
max_file_size=max_file_size,
kms_key_id=kms_key_id,
manifest=False,
use_threads=use_threads,
boto3_session=session,
)
if chunked is False:
Expand Down