Skip to content

Commit

Permalink
Download product with multiple workers
Browse files Browse the repository at this point in the history
This patch introduces workers for downloading files using the executable.
For programs using the library the parallelization should be done in the calling code itself.

There could be a lot of md5sums to process when working on long download periods.
Therefore `multiprocessing` is used to be able to calculate those hashes on multiple cores.
  • Loading branch information
shaardie authored and lkiesow committed Jan 2, 2020
1 parent 2a01ec5 commit 8df231d
Showing 1 changed file with 13 additions and 2 deletions.
15 changes: 13 additions & 2 deletions sentinel5dl/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import dateutil.parser
import certifi
import logging
import multiprocessing
import textwrap
import sentinel5dl
from sentinel5dl import search, download
Expand Down Expand Up @@ -148,6 +149,13 @@ def main():
providing a cabundle.'''
)

parser.add_argument(
'--worker',
type=int,
default=1,
help='Number of parallel downloads',
)

parser.add_argument(
'download_dir',
metavar='download-dir',
Expand All @@ -170,8 +178,11 @@ def main():
processing_mode=args.mode
)

# Download found products to the local folder
download(result.get('products'), args.download_dir)
# Download found products to the download directory with number of workers
with multiprocessing.Pool(args.worker) as p:
p.starmap(download, map(
lambda product: ((product,), args.download_dir),
result.get('products')))


if __name__ == '__main__':
Expand Down

0 comments on commit 8df231d

Please sign in to comment.