Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TransferManager.shutdown() is not functional #143

Open
JasonDeArte opened this issue Oct 20, 2019 · 2 comments
Open

TransferManager.shutdown() is not functional #143

JasonDeArte opened this issue Oct 20, 2019 · 2 comments

Comments

@JasonDeArte
Copy link

JasonDeArte commented Oct 20, 2019

In TransferManager.shutdown's call to TransferManager._shutdown(), it passes the same boolean parameter twice, thus setting the string param to a bool and exception_type to a string instead of a class

https://github.com/boto/s3transfer/blob/develop/s3transfer/manager.py#L546-L548

    def shutdown(self, cancel=False, cancel_msg=''):
        ...
        self._shutdown(cancel, cancel, cancel_msg)

    def _shutdown(self, cancel, cancel_msg, exc_type=CancelledError):

After fixing the params locally, attempting to cancel an in-progress s3 file transfer using shutdown(cancel=True) will result in deadlock

sample client code

s3client = boto3.client('s3')
transfer_config = boto3.s3.transfer.TransferConfig(max_concurrency=s3concurrency, multipart_threshold=multipart_threshold, use_threads=use_threads)
manager = boto3.s3.transfer.create_transfer_manager(client=s3, config=transfer_config)
transfer = boto3.s3.transfer.S3Transfer(manager=manager)

class AbortableProgress():
  def __init__(self, s3manager, progress_cb):
    self.manager = s3manager
    self.cb      = progress_cb
    self.start   = datetime.datetime.now()

  def __call__(self, *args, **kwargs):
     if self.cb:
       self.cb(*args, **kwargs)
     if self.manager:
       current = datetime.datetime.now()
       elapsed = (current-self.start).seconds
       if elapsed > 3:
         print("ABORT")
         self.manager.shutdown(cancel=True, cancel_msg="time to abort!")

transfer.download_file(sts['bucket'], sts['key'], local_filename, callback=AbortableProgress(manager, progress_callback))
@tschoonj
Copy link

tschoonj commented Mar 8, 2021

I have fixed the thread deadlock in #179. Hope it gets merged in soon.

@sodul
Copy link

sodul commented Dec 1, 2023

I stumbled on this because today pylint is giving us this error in our code calling shutdown():

Unexpected keyword argument 'cancel_msg' in method call (unexpected-keyword-arg)

Pylint is wrong to call out our code but the internal call to _shutdown() with cancel being passed twice is obviously broken.

This bug has been opened since 2019 with a PR for it, @nateprewitt is this something you can take a look at?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants