Skip to content

Add s3.copy_objects() #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 17, 2020
Merged

Add s3.copy_objects() #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
73f3b88
Add s3.copy_objects() #186
igorborgest Apr 16, 2020
f480e74
Add merge tutorial
igorborgest Apr 16, 2020
e6d808d
Add s3.copy_objects() on the docs #186
igorborgest Apr 16, 2020
537afe5
Splitting up and creating two functions, copy_objects() and merge_dat…
igorborgest Apr 17, 2020
b6b8bfc
Fixing Flake8 compatibility
igorborgest Apr 17, 2020
30f3ea0
Updating tutorial 13
igorborgest Apr 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")
- [10 - Parquet Crawler](https://github.com/awslabs/aws-data-wrangler/blob/master/tutorials/10%20-%20Parquet%20Crawler.ipynb)
- [11 - CSV Datasets](https://github.com/awslabs/aws-data-wrangler/blob/master/tutorials/11%20-%20CSV%20Datasets.ipynb)
- [12 - CSV Crawler](https://github.com/awslabs/aws-data-wrangler/blob/master/tutorials/12%20-%20CSV%20Crawler.ipynb)
- [13 - Merging Datasets on S3](https://github.com/awslabs/aws-data-wrangler/blob/master/tutorials/13%20-%20Merging%20Datasets%20on%20S3.ipynb)
- [**API Reference**](https://aws-data-wrangler.readthedocs.io/en/latest/api.html)
- [Amazon S3](https://aws-data-wrangler.readthedocs.io/en/latest/api.html#amazon-s3)
- [AWS Glue Catalog](https://aws-data-wrangler.readthedocs.io/en/latest/api.html#aws-glue-catalog)
Expand Down
9 changes: 8 additions & 1 deletion awswrangler/_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@ def client(service_name: str, session: Optional[boto3.Session] = None) -> boto3.
)


def resource(service_name: str, session: Optional[boto3.Session] = None) -> boto3.resource:
"""Create a valid boto3.resource."""
return ensure_session(session=session).resource(
service_name=service_name, use_ssl=True, config=botocore.config.Config(retries={"max_attempts": 15})
)


def parse_path(path: str) -> Tuple[str, str]:
"""Split a full S3 path in bucket and key strings.

Expand Down Expand Up @@ -62,7 +69,7 @@ def ensure_cpu_count(use_threads: bool = True) -> int:

Note
----
In case of `use_threads=True` the number of process that could be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that could be spawned will be get from os.cpu_count().

Parameters
----------
Expand Down
4 changes: 2 additions & 2 deletions awswrangler/athena.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,7 @@ def read_sql_query( # pylint: disable=too-many-branches,too-many-locals
Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().
Parameters
----------
Expand Down Expand Up @@ -605,7 +605,7 @@ def read_sql_table(
Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().
Parameters
----------
Expand Down
10 changes: 5 additions & 5 deletions awswrangler/db.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ def copy_to_redshift( # pylint: disable=too-many-arguments

Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().

Parameters
----------
Expand Down Expand Up @@ -576,7 +576,7 @@ def copy_files_to_redshift( # pylint: disable=too-many-locals,too-many-argument

Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().

Parameters
----------
Expand Down Expand Up @@ -798,7 +798,7 @@ def write_redshift_copy_manifest(

Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().

Parameters
----------
Expand Down Expand Up @@ -908,7 +908,7 @@ def unload_redshift(

Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().

Parameters
----------
Expand Down Expand Up @@ -1024,7 +1024,7 @@ def unload_redshift_to_files(

Note
----
In case of `use_threads=True` the number of process that will be spawned will be get from os.cpu_count().
In case of `use_threads=True` the number of threads that will be spawned will be get from os.cpu_count().

Parameters
----------
Expand Down
Loading