wormhole send crash zipping large directory

[nick-robo]

When sending a large directory containing video files (zip compression is pointless), wormhole crashes after raising ValueError: Can't close the ZIP file while there is an open writing handle on it. Close the writing handle before closing the zip.

Full Output

main@box ~> wormhole send video_dir/
Building zipfile..
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/wormhole/cli/cli.py", line 122, in _dispatch_command
    yield maybeDeferred(command)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/wormhole/cli/cmd_send.py", line 91, in go
    yield d
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python3/dist-packages/wormhole/cli/cmd_send.py", line 105, in _go
    offer, self._fd_to_send = self._build_offer()
  File "/usr/lib/python3/dist-packages/wormhole/cli/cmd_send.py", line 350, in _build_offer
    raise UnsendableFileError(errmsg)
  File "/usr/lib/python3.7/zipfile.py", line 1265, in __exit__
    self.close()
  File "/usr/lib/python3.7/zipfile.py", line 1798, in close
    raise ValueError("Can't close the ZIP file while there is "
builtins.ValueError: Can't close the ZIP file while there is an open writing handle on it. Close the writing handle before closing the zip.
ERROR: Can't close the ZIP file while there is an open writing handle on it. Close the writing handle before closing the zip.

Details:

• magic-wormhole 0.11.2
• Linux 4.19.0-12-amd64 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux
• Directory size: 51 Gb

[ysig]

@meejah Zipping a directory when sending it sounds like a bad implementation. What should be zipped instead are file names and ideally a --sync option should be given to have a functionality like rsync. Most people want to use wormhole when it's complicated to use scp/rsync or a transferring software.

[meejah]

I do agree with your assessment.

I'm personally going to be putting my effort (further) into the "next gen" transfer protocol (see e.g. #445 ) which won't use Zip at all and allows for multiple transfers, including in both directions. ("Dilation" itself also allows tons of other cool features, e.g. https://github.com/meejah/fowl )

This gives UI authors a ton more flexibility and honours the "hard work" of setting up the connection in the first place. If only I had infinite time ;)

Note that 0.13.0 did fix a problem with streaming that might make the above better. It was a minor change, but meant the receiving side buffered potentially a lot in memory.