Skip to content

Commit

Permalink
reftest: add progress bar for uploads
Browse files Browse the repository at this point in the history
This script performs both downloads and uploads for some large-ish
files. We previously had a progress bar for downloads, but not
uploads.

For me at least the uploads can be slow enough to give the impression
that the script is stuck, so let's add a progress bar for uploads too.
  • Loading branch information
rohanpm committed Feb 4, 2022
1 parent dd56844 commit 4b1f4bf
Showing 1 changed file with 18 additions and 4 deletions.
22 changes: 18 additions & 4 deletions support/reftest/reftest
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,24 @@ class S3Handler:
self.bucket = bucket

def upload_from_localfile(self, path, checksum):
with open(path, "rb") as data:
self.s3_client.upload_fileobj(
Fileobj=data, Bucket=self.bucket, Key=checksum
)
st = os.stat(path)
bar = tqdm(
desc="%s [upload]" % checksum,
total=st.st_size,
unit="iB",
unit_scale=True,
)

try:
with open(path, "rb") as data:
self.s3_client.upload_fileobj(
Fileobj=data,
Bucket=self.bucket,
Key=checksum,
Callback=bar.update,
)
finally:
bar.close()


def parse_aws_session(parser):
Expand Down

0 comments on commit 4b1f4bf

Please sign in to comment.