OnionShare closes before downloading is complete

[g9so]

When I attempt to share a file using the CLI version of OnionShare 2.0 on Ubuntu 18.04 the file that is received is a truncated version of the original.

Attempting to resend the file will result in a different amount of data truncated.

This bug is only present in the CLI version, the GUI version behaves as expected.

When the --stay-open option is used the whole file will be downloaded correctly but OnionShare will report that the download has completed long before Tor Browser does.

The problem does not appear to occur when sharing vary small files but is consistent when sharing anything around 10 MB or larger.

[mig5]

Hi @g9so , this sounds like #812 - where we found a bug in certain Tor versions that causes exactly this problem, when using v3 onions.

On Ubuntu 18.04, the version of Tor shipped seems to be 0.3.2.10 which likely contains this bug.

You can try one of three things to see if it stops the issue for you:

1. Enable 'Use legacy onions' - you'll use the shorter onion addresses, which aren't affected by this bug, but of course the newer v3 style onions are recommended

2. Use the Tor Project's Apt repository instead of using Ubuntu's older version of Tor. You can add the Tor Project Apt repository by following these instructions (see Option 2): https://www.torproject.org/docs/debian.html.en . If you do this, you should be able to use OnionShare with v3 onions

3. Download Tor Browser and then change your OnionShare settings to use Tor Browser's Tor rather than the system-installed Tor.

Please try either of these and let us know if it fixes it for you. We'll probably need to write some documentation on this issue so long as versions of Tor like 0.3.2.10 remain in popular distributions like Ubuntu.

Cheers

[mig5]

Actually, I just realised 0.3.2.10 is probably too old even for v3 onions, so it's probably not related to that issue at all :( sorry. You could still try a newer upstream version of Tor as above, and see if it helps, in any case.

[g9so]

Hi @mig5

I have upgraded to the latest stable version of Tor from the Tor Project's repository, but the problem still exists for both v2 and v3 onions.

I have done a lot more testing and have found that the bug is actually present in both the CLI and GUI versions.

In both the CLI and GUI versions OnionShare reports that the file download has started and will show progress on how much data has been sent as soon as the .onion/slug/download link is visited but before the save button is pressed in Tor Browser.

In the CLI version as soon as OnionShare reports that the file has been fully downloaded OnionShare closes and whatever has not been received yet is truncated.

In the GUI version OnionShare will report that the file has finished downloading and that sharing has stopped when in fact sharing continues. If OnionShare is closed after it reports that the download is complete and that sharing has stopped but before the download has actually completed the file will be truncated.

[mig5]

hi @g9so ,

In both the CLI and GUI versions OnionShare reports that the file download has started and will show progress on how much data has been sent as soon as the .onion/slug/download link is visited but before the save button is pressed in Tor Browser.

In the GUI version OnionShare will report that the file has finished downloading and that sharing has stopped when in fact sharing continues

These two are not in fact bugs: OnionShare starts sending the data into the Tor network when Download is clicked (whether or not the client has clicked the Tor Browser dialog to save or open is purely client side stuff - the download is already happening in the background as a temporary file).

And there is always a lag between when OnionShare says it's 'completed' sending a share, and what the client side reports, because what it means is that it's sent 100% of the data into the Tor network. The receiving end takes a slight extra time to receive the full file due to that data still traversing the nodes in the Tor circuit, even if OnionShare says it's finished - it can only report on the progress of sending the data into the Tor Network, and vice versa.

If OnionShare is closed after it reports that the download is complete

Did you close it yourself? Sounds unusual.

In my tests on latest Tor, there is no truncating. The file I shared to myself is identical to the original file, on OnionShare 2. At the moment that OnionShare stops serving and the download says it's complete, they are identical files, no truncating. This is the same whether I use the GUI or CLI. On the OnionShare side, I'm using Tor 0.3.5.8 and on the client side I'm using Tor Browser 8.0.6, so Tor 0.3.5.7. I'm not using Ubuntu though.

Anyway thanks for your report, perhaps it's OS specific. See what Micah says :)

[g9so]

The receiving end takes a slight extra time to receive the full file due to that data still traversing the nodes in the Tor circuit, even if OnionShare says it's finished - it can only report on the progress of sending the data into the Tor Network, and vice versa.

To clarify, what I am experiencing is not a slight network delay. OnionShare can send data for more than a minute and accounts for several MB of data after it reports that the download is complete.

Did you close it yourself?

Yes, if I close OnionShare after it reports that the download is complete but before Tor Browser reports that the download is complete the file will be truncated.

From everything that I have seen it appears that the problem is the result of OnionShare overestimating how much data has been sent and consequently killing itself before the file has actually finished downloading.

If OnionShare is configured to to close after the first download then the file will be truncated. If OnionShare is configured to stay open then sharing will continue until the whole file is downloaded without truncation.

[g9so]

I have done some more testing in a VM and can confirm that the issue is also present when using Ubuntu 18.10.

[mig5]

Just noting that I did indeed reproduce this today on Ubuntu 18.04 (but don't think it's OS-specific, nor Tor version specific (reproduced on both vanilla Ubuntu and upstream Tor apt repo).

It's a very similar bug to #812 and I believe it's probably a Tor, not an OnionShare bug.

In #812 the download would truncate once the Onion service stopped. This is similar, except if you leave OnionShare (the app) running (note: I'm not talking about stay_open=True, but just leaving the app running with a stopped service), the download completes even after the Onion service stops.

However, if you shut down OnionShare (the app) entirely, severing its Tor connection, this severs the connection immediately on the client side, even though we'd already stopped and removed the .onion service via Stem. CLI probably reproduces this because the entire app exits when the service stops (but I didn't test CLI, only GUI).

Over to Micah. Probably going to need a similar PoC to last time, except this time also stopping the Tor connection altogether on the server side.

[superboum]

I observed this bug between a Ubuntu 20.04 with the official OnionShare PPA (OnionShare 2.2.ppa1, Tor 0.4.2.7) from one side and Fedora 32 with Tor Browser (version 9.0.2, embedded Tor version is Tor 0.4.1.6 (git-d10abc0929f4941d)). I am using onionshare in CLI mode. I was trying to transfer a 100Mo file created randomly with dd if=/dev/urandom of=rnd.img bs=1M count=100. Each time, the transfer fails around ~90Mo (94Mo, 91Mo and 90Mo respectively).

The --stay-open workaround works.

[mig5]

I don't reproduce it on the develop branch using Tor 0.4.3.5 and the 'CLI' mode.

I shared a 100MB file with poetry run dev_scripts/onionshare ~/out.file --verbose against develop branch.

And I downloaded it with torify wget -O out.file http://onionshare:XXXXXXXXXX@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.onion/download

I diffed the original and the downloaded file and they were identical.

I wonder if it's been 'accidentally' fixed somehow...

[micahflee]

It still seems to be a bug for me in the latest develop branch. Alas. Taking a look now.

[micahflee]

Okay, this bug is incredibly similar to #812, but with one small difference. With that bug, the circuits were closed when the ephemeral onion service was stopped, even if they were still in use (e.g. the server finished serving/uploading the file but the client hasn't finished downloading it). That was fixed.

The problem we're hitting here is when the tor process itself gets killed, it closes all of its circuits, thus breaking the downloading (even if the upload is finished). This problem seems to only exist in the CLI when using bundled tor. If you're using a system tor, or Tor Browser, the tor process doesn't get killed the file is finished sending, so it finishes successfully. And same with the GUI, the tor process remains running even if the onion service stops.

The bottom of cli/onionshare_cli/__init__.py has this:

    finally:
        # Shutdown
        app.cleanup()
        onion.cleanup()

If you just comment out onion.cleanup(), the bug goes away. That function stops the ephemeral onion services and (in the case of bundled tor) kills the tor process -- so there ends up being a dangling tor process after onionshare is closed, which isn't good.

So I think the solution will be to somehow wait until the circuits are complete before killing the tor process...