Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GS progress for push & pull #2809

Merged
merged 11 commits into from
Nov 21, 2019
Merged

GS progress for push & pull #2809

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6fdba1d
initial upload idea
casperdcl Nov 17, 2019
6b388ed
py2/3 fix
casperdcl Nov 17, 2019
1f442a4
gs download idea
casperdcl Nov 17, 2019
321a46b
add gs stats
casperdcl Nov 18, 2019
32c7b60
reduce gs upload chunk size
casperdcl Nov 18, 2019
934850f
neaten default chunks
casperdcl Nov 18, 2019
eb5a0a9
fix gs chunks
casperdcl Nov 19, 2019
5eefacd
back to gs multiple
casperdcl Nov 19, 2019
28b99dc
expose and use name, no_progress_bar
casperdcl Nov 19, 2019
b811a77
update dynamic chunk tests
casperdcl Nov 19, 2019
b2f7c84
fix deepsource issues
casperdcl Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions dvc/remote/gs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@
from dvc.utils.compat import FileNotFoundError # skipcq: PYL-W0622

logger = logging.getLogger(__name__)
MIN_CHUNKSIZE = 256 * 1024
casperdcl marked this conversation as resolved.
Show resolved Hide resolved


def dynamic_chunk_size(func):
@wraps(func)
def wrapper(*args, **kwargs):
import requests
from google.cloud.storage.blob import _DEFAULT_CHUNKSIZE

# `ConnectionError` may be due to too large `chunk_size`
# (see [#2572]) so try halving on error.
Expand All @@ -37,7 +37,7 @@ def wrapper(*args, **kwargs):
while True:
try:
# skipcq: PYL-W0212
chunk_size = _DEFAULT_CHUNKSIZE * multiplier
chunk_size = MIN_CHUNKSIZE * multiplier
return func(*args, chunk_size=chunk_size, **kwargs)
except requests.exceptions.ConnectionError:
multiplier //= 2
Expand Down