Suppress stack trace for KeyboardInterrupt

[dmpetrov]

1. When canceling (Ctrl+C) dvc pull you see Python exception in the output. It is better to avoid it (UX issue).
2. EDITED: it looks like this was fixed in later versions (see my commend below) The command was not actually canceled after the first Ctrl+C - dvc pull was still running. I had to push Ctrl+C many times (with additional exceptions in the output) before it was actually canceled.

Ideally, after Ctrl+C user should see 1-2 lines about the operation cancelation and command exit.

$ dvc pull

date
 39%|███▊      |s3://dvc-private/re780406/2017219 [44:27<1:01:15,   336file/s] 39%|███▊      |s3://dvc-private/re780805/2017219 [44:29<1:25:30,   241file/s] 39%|███▊      |s3://dvc-private/re780867/2017219 [44:29<1:16:04,   271file/s]^CTraceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 427, in result
    self._condition.wait(timeout)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 296, in wait
    waiter.acquire()
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmitry/src/dvc/dvc/remote/base.py", line 738, in cache_exists
^C  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 591, in result_iterator
    future.cancel()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 362, in cancel
    self._condition.notify_all()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 365, in notify_all
    self.notify(len(self._waiters))
KeyboardInterrupt

During handling of the above exception, another exception occurred:

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 40, in _python_exit
    t.join()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
^C

[shcheklein]

@casperdcl can it be something tqdm and the recent change related?

@dmpetrov please, include the DVC version.

[casperdcl]

I doubt it's tqdm specific - this looks more related to passing KeyboardInterrupt to multiple threads. Presumably should have always been an issue?

Maybe can be fixed by subclassing ThreadPoolExecutor?

[dmpetrov]

Right. It was not the last version:

$ dvc -V
0.62.1

After the update to 0.68.1+6147b9 the 2nd issue disappeared. So, only the python stack trace output UI issue needs to be fixed:

$ dvc pull
  5%|▌         |s3://dvc-private/remote/dataset-regis105931/2017219 [06:16<4:14:37,   125file/s]ERROR: interrupted by the user
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 40, in _python_exit
    t.join()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

I'll edit the bug description.

[efiop]

Was playing around with a big dataset and pulling, where I need to interrupt for debugging purposes and this drives me insane.

To solve it, we need to set a signal handler for our ThreadPoolExecutors that will call executor.shutdown(). The unfortunate part is that python has very limited thread support, so we will have to wait until all workers finish(e.g. we'll have to wait until 1 large file is done downloading), as we can't just kill them, unfortunately :( We might be able to use some global variable and check for it from time-to-time in our workers (e.g. write a block -> check if we are aborted -> continue), but that would bring quite a complication to our code. Another possible solution might be to use multiprocessing instead, where we can easilly kill any worker, but it requires non-trivial refactoring, as lots of things are not pickable right now 🙁

So I think we should still at least set the signal handler, so we could communicate with the user more clearly, not show him ugly stacktrace and cancel the rest of the jobs that weren't submitted yet.

EDIT: even CPython's ThreadPoolExecutor is trying hard to wait for threads when python exists https://github.com/python/cpython/blob/3.8/Lib/concurrent/futures/thread.py#L16

[dmpetrov]

@efiop thank you for the investigation.
Another possible solution - upload/download files by smaller blocks.