Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-721174: Support COPY INTO <location> with storage integration and gcs bucket #368

Open
dahlbaek opened this issue Jan 4, 2023 · 1 comment · May be fixed by #487
Open

SNOW-721174: Support COPY INTO <location> with storage integration and gcs bucket #368

dahlbaek opened this issue Jan 4, 2023 · 1 comment · May be fixed by #487
Labels
feature status-triage_done Initial triage done, will be further handled by the driver team

Comments

@dahlbaek
Copy link

dahlbaek commented Jan 4, 2023 •
edited
Loading

What is the current behavior?

The COPY INTO <location> statement support gcs buckets, but there is no GCSBucket ClauseElement in https://github.com/snowflakedb/snowflake-sqlalchemy/blob/a84fa77ae4387ed60db036db61dc7d8f82849a98/src/snowflake/sqlalchemy/custom_commands.py. Similarly, the statement supports using a STORAGE_INTEGRATION option, but that option is not supported by CopyInto

snowflake-sqlalchemy/src/snowflake/sqlalchemy/custom_commands.py

Lines 110 to 157 in a84fa77

class CopyInto(UpdateBase):
"""Copy Into Command base class, for documentation see:
https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html"""
__visit_name__ = "copy_into"
_bind = None
def __init__(self, from_, into, formatter=None):
self.from_ = from_
self.into = into
self.formatter = formatter
self.copy_options = {}
def __repr__(self):
"""
repr for debugging / logging purposes only. For compilation logic, see
the corresponding visitor in base.py
"""
return f"COPY INTO {self.into} FROM {repr(self.from_)} {repr(self.formatter)} ({self.copy_options})"
def bind(self):
return None
def force(self, force):
if not isinstance(force, bool):
raise TypeError("Parameter force should be a boolean value")
self.copy_options.update({"FORCE": translate_bool(force)})
return self
def single(self, single_file):
if not isinstance(single_file, bool):
raise TypeError("Parameter single_file should be a boolean value")
self.copy_options.update({"SINGLE": translate_bool(single_file)})
return self
def maxfilesize(self, max_size):
if not isinstance(max_size, int):
raise TypeError("Parameter max_size should be an integer value")
self.copy_options.update({"MAX_FILE_SIZE": max_size})
return self
def files(self, file_names):
self.copy_options.update({"FILES": FilesOption(file_names)})
return self
def pattern(self, pattern):
self.copy_options.update({"PATTERN": pattern})
return self

What is the desired behavior?

The CopyInto class supports Copy Into <location> statements of the type

COPY INTO '<gcs_bucket>'
FROM (<select>)
FILE_FORMAT = ( TYPE = PARQUET )
STORAGE_INTEGRATION = <gcs_storage_integration>
HEADER = TRUE;

How would this improve snowflake-connector-python?

This would make it possible to use the CopyInto class on order to execute such variants of the COPY INTO <location> statement.

References, Other Background

We're a small company that make use of the snowflake-sqlalchemy library. If someone will give us pointers as to what we should add in terms of classes and tests, we can allocate time to work on a pr.
@github-actions github-actions bot changed the title Support COPY INTO <location> with storage integration and gcs bucket SNOW-721174: Support COPY INTO <location> with storage integration and gcs bucket Jan 4, 2023
@sfc-gh-dszmolka sfc-gh-dszmolka added the status-triage_done Initial triage done, will be further handled by the driver team label Mar 15, 2024
@sfc-gh-dszmolka
Copy link

thank you for submitting this enhancement idea, we'll consider it for future planning. Of course a PR is more than welcome, if there's such possibility from your end.

@sjhewitt sjhewitt linked a pull request Apr 15, 2024 that will close this issue
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature status-triage_done Initial triage done, will be further handled by the driver team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add STORAGE_INTEGRATION support to AWSBucket and AzureContainer sjhewitt/snowflake-sqlalchemy
2 participants
@dahlbaek @sfc-gh-dszmolka