-
Notifications
You must be signed in to change notification settings - Fork 650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"Stop sharing after first download" doesn't work with v3 onions #812
Comments
I just reproduced this, but only when 'Close after first download' is checked. When unchecked ('stay open' as we call it), the file downloads fully. So somehow the shutdown is occurring too soon in the default mode - which is even more curious. |
I haven't been able to solve this bug. But here's what I've been able to find out, in case it helps you:
(note the ShareModeWeb.b'' - I am printing to common.log(), the value of
Somehow the .onion is closed too soon, despite all bytes having been 'sent' from Flask, and it seems to cut off the download. None of the logic re: when to stop the share has changed (we still only consider a download 'finished' in the UI when the REQUEST_PROGRESS' bytes sent == the file size. Furthermore, the same issue occurs in CLI mode) :/ |
OK, this is wild. I reproduce it on my Onionshare dev VM even when I've checked out the master branch (v 1.3.1) However, I don't reproduce it when using my 1.3.1 MacOS instance (even though in both cases I'm still downloading the file via a Whonix DispVM). This suggests to me it's either:
I've experimented with downgrading Tor and Flask on the Linux machine, to no avail. I'll next try and install 1.3.1 on a separate Linux VM (maybe the upstream Fedora package) to see if I reproduce it 'natively' (ie. with no poisoning of dependencies from my dev machine). |
Hmm, still confused. I don't reproduce it on Fedora 28 with I reproduce it on Mac against develop branch (assumed it was my locally installed pip stuff though). But then I checked out master branch on Mac and ran it, and I don't reproduce it (works fine), so this is inconsistent with my Linux experience.. Also it's not v2/v3 onion specific. |
Wow thanks for all of the troubleshooting. I had no idea this would be such a weird and tricky bug. I'm starting to take a look too. |
I'm reproducing it when using Tor, but not reproducing it when using |
|
Hmm I'll give it some more time soon but I'm actually about to board a flight where I won't have internet. Good luck for tonight though :) |
I find that if I add a In other words, we know that the The core of the problem is that the onion service stops too suddenly and this causes the incomplete file client side. Since nothing has changed in the way we start/stop the onion service, as far as I can tell.. this somehow points to some sort of buffering change in how onion services are served....? :s I am not advocating the I'm surely wrong about all this... but I get consistent results if I keep the onion service running just a bit longer (which also tallies with the fact that we knew 'stay open' mode also works). It all comes down to how long the onion service is available. |
I'm now convinced it's a Tor version or Tor Browser version thing. I just connected an old Tor Browser (7.5.5 - the version we shipped the tor binary from in our 1.3.1 release) as my client (for downloading a share from OnionShare). I reproduced the issue when using Tor Browser 7.5.5 as a client, but with OnionShare itself using a newer ('bundled') Tor on Linux (0.3.4.8). So, then I changed OnionShare itself to use the Tor Browser 7.5.5 running in the background as its tor provider ('automatic') mode, instead of bundled mode. Still using the same Tor Browser 7.5.5 as my client, I got a complete share! I couldn't reproduce the incomplete download issue. Of course, I also got a v2 onion when using 7.5.5 as my connection method, whereas the previous attempt in bundled mode, due to newer Tor, was a v3 onion. But it's not v2 specific. I know this because I then switched OnionShare back to using bundled mode, but with legacy v2 onion mode turned on, and using this new v2 onion address, I reproduced the incomplete file issue again. This is all against develop branch. So what this tells me, is that in between Tor 0.3.3.7 (the version that was shipped in Tor Browser 7.5.5), and Tor 0.3.4.8 (my version on Debian Buster that I use in 'bundled' mode), the way data is sent to/from onion services to the end client (regardless of the client's Tor Browser version) has changed. It suggests that closing an ephemeral onion service (still via the same Stem library version, remember!) changes whether any remaining data flows through to the client. In newer Tor, it immediately is shut off... I'll try and find upstream evidence that corroborates this change, but it will be hard to figure out what to look for.. EDIT argh, no, I cannot be right, because if I downgrade to Tor 0.9.8 on Buster (via jessie-backports repo I had lying around) I still reproduce it.. it would not make sense to reproduce in 0.2.9.8 and 0.3.4.8 but not 0.3.3.7... however, I definitely don't reproduce it once I switch back to 'automatic' mode using TBB 7.5.5 in the background... going slightly mad now, might have to give up. Hope I haven't caused too many rabbit holes for you to dig into. One thing I know for sure, is that adding a sleep before stopping the .onion in either CLI or GUI mode, avoids the issue.. |
I've been distracted for a few weeks, but I'm finally looking at this one again. Ugh, this is a really annoying, but critical, issue. So basically, it appears that with some versions of Tor, as soon as the OnionShare side closes the ephemeral onion service, somehow the Tor network closes it on the Tor Browser as well, even if the download isn't complete. The hack where we And the issue with relying on that hack is that we have no way of learning how long we should sleep. It completely depends on the speed of the data across the Tor network. Like, we could try to make an educated guess maybe based on how long it took to send 100% of the bytes, but if our guess is wrong (like, maybe one of the Tor nodes gets extra congested after OnionShare has sent the last byte) then we might not wait like enough, and the onion connection will shut down early. |
Yep I completely agree. I think fixing it in OnionShare would be unreliable. I tried trawling Tor core release notes and commits to try and pinpoint the cause but didn't succeed. Is there anyone there you might be able to speak to who might know more or recognise this behaviour as being attributed to a change in core? If it's a regression and gets fixed, at least we could document a range of Tor versions that might end up being known to be a problem for OnionShare.. |
So I think the best way forward is to either write an email to the tor-dev listserve, or ask on IRC, and explain the situation as best we can. In preparation of writing that email, I decided to try to make a simple PoC of the bug, and now it's not acting like a bug... Check out this simple python script: https://gist.github.com/micahflee/bade960e96d35007bc5c182a0ca61b56 You need Tor Browser in the background for stem to connect to the tor controller, and you need a system tor so you can run So for example, in one terminal, I start the server and wait for the onion service to start:
In another terminal, I make a GET request to http://127.0.0.1:8080/ (not the onion service) to make sure it works locally:
Over in the first terminal, I see the request and it shuts down:
It downloaded the full 5mb and shut down the server. Now here's trying again, but this time downloading over the onion service. I would expect it to hit this bug, but it doesn't seem too... Back in the first terminal, I run the script again and wait for the onion service:
In the second terminal, I download it over the onion service:
While it was still downloading, in the first terminal, the server already shut down, and in fact the whole program ended, so the ephemeral onion service must have stopped:
Yet in the second terminal, it kept downloading until the full 5mb was received. So, I didn't reproduce it... I'm using Tor Browser 8.0.3, which comes with tor 0.3.4.8. I just changed my
When I load it in Tor Browser, it says the size is 8.1mb, and the compressed zip file size is 7.3mb. Then I download it, and shortly after the OnionShare process stops, the download in Tor Browser stops at 5.5mb, corrupted. Here's the rest of the terminal output:
So, how come I can reproduce it OnionShare with tor 0.3.4.8, but I can't reproduce it in this little script with the same Tor version? OnionShare is using flask, and that script with was |
OK weird.. Do you at least see the same as I did if you download the old Tor Browser 7.5.5 and use that as your Tor connection method in OnionShare - do you get a complete download? Because if it's an OnionShare-specific issue then I'm still confused how I succeeded with the older Tor Browser (older Tor version for the connection) |
I just reproduced this bug from the Let me try with Tor Browser 7.5.5 next. |
So it's kind of hard to find old releases of Tor Browser, but I found them on their github mirror: https://github.com/TheTorProject/gettorbrowser/releases They don't have Tor Browser 7.5.5, but I did try it with Tor Browser 7.5.3, and you're totally right. I don't reproduce the error in that version of Tor... So how do we proceed? OnionShare is kind of a beast of a project now. I think the best way forward is to reproduce the bug, using the current version of Tor, in a small script like that gist. If we can reproduce it, then at least we can properly ask for help. |
Sorry I should've clarified I fetched my old TBB from https://archive.torproject.org/tor-package-archive/torbrowser/ I agree. I will look at your script and try and reproduce (probably not today). It would be interesting if Flask somehow has something to do with it but can't imagine how since it works without Tor. |
Just a note that I had the same experience as you with that PoC. Could not reproduce the issue. Very frustrating. I've tried to re-implement it in Flask in case that's the difference, but my Python-fu is just not as strong. |
I made a branch, 812_troubleshooting, where I'm ripping out almost all of the un-needed logic to reproduce this bug. I can still rip out more and make it even simpler, but so far it only supports CLI mode, only share mode, only connecting using Tor Browser, no longer has a slug. And it reproduced the bug. I'll keep trying to make it even simpler, but thought I'd share the progress. |
I've reproduced the bug in this gist! It turns out, when I start the onion service like this, it works fine: res_nobug = c.create_ephemeral_hidden_service(
{ 80: 8080 },
await_publication=True
) But when I start the onion service like this (which incidentally uses a v3 onion server) it triggers the bug: res_bug = c.create_ephemeral_hidden_service(
{ 80: 8080 },
await_publication=True,
key_type="NEW",
key_content="ED25519-V3"
) It's consistent, every time. So, I think the bug is either in stem's |
Ok, I've updated the gist. Now it runs if sys.argv[1] == "1":
key_type = "NEW"
key_content = "RSA1024"
elif sys.argv[1] == "2":
key_type = "NEW"
key_content = "ED25519-V3"
elif sys.argv[1] == "3":
key_type = "ED25519-V3"
key_content = generate_v3_private_key()[0] For example, to try the
Then, in another terminal, I run
Using Tor Browser 8.0.3, with Tor 0.3.4.8, I've confirmed that these settings don't reproduce the bug:
And these ones do:
With Tor Browser 7.5.5, with Tor 0.3.2.10, making a new
Which makes sense, because Tor 0.3.2.10 doesn't support ephemeral onion services. So, it really seems like this is either a v3 onion service issue, or possibly a stem issue related to v3 onion services. |
Tor Browser 8.0 comes with Tor 0.3.3.9, and it has the same issue. v3 onion services fail. Tor Browser 7.5.6 comes with Tor 0.3.3.7, it has the same issue. Tor Browser 7.5.1 comes with Tor 0.3.2.10, and like with Tor Browser 7.5.5, it doesn't support ephemeral v3 onion services. |
I've tested with stem 1.6.0 and stem 1.7.0, but there's no difference. |
I updated the script to reproduce the issue again -- this time you just pass either I also just wrote a detailed email to the tor-dev list that explains the problem, shows the script that reproduces the bug, and asks for help in solving it: https://lists.torproject.org/pipermail/tor-dev/2018-November/013556.html |
Huh! Ok. I was sure I reproduced this with v2 per #812 (comment) but maybe I was just going cross-eyed by then. |
Yup, and I just tested using Tor 0.3.3.9 (the version that comes in the Ubuntu 18.10 So, while we're waiting to see if anyone from the tor-dev list can help, I think we should maybe try to finish and merge #806, to simplify the onion service code. And then I think we need to make a decision. There's a decent chance that this is an upstream bug, and it needs an upstream patch before it's fixed. If this is the case, should we:
|
Hey hey, thanks for pointing this out. This is now on our radar. We will try to look into this ASAP and get back to you. Give us a few days tho. |
I've responded on the tor-dev@ email thread! The underlying issue in tor is now here: |
Thanks for your detailed response @dgoulet-tor! And thanks @asn-d6, a few days is definitely fine. I'm so glad we finally understand the underlying cause of this bug. Since this is a tor issue, and its milestone is tor 0.3.5.x, I wonder how OnionShare should handle it. If the tor version is less than 0.3.5, OnionShare could disable v3 onion support, and if it's newer, it could enable it. And depending on tor's release schedule, we could maybe wait until there's at least a tor 0.3.5rc release to make the OnionShare 2.0 release, so that at least we can ship tor 0.3.5rc with the Windows and Mac versions of OnionShare. And in the meantime Linux users could install tor from TPO's repo, or run the beta Tor Browser, to get a new enough tor if they want v3 onion services. |
@micahflee I guess tor may implement (as part of the ticket) some GETINFO key that would indicate that RP circuits are not being closed on service removal. It should be much more straightforward than dealing with tor versions. |
@nogoegst this is the PR that fixes the upstream tor bug: torproject/tor#546 I believe OnionShare will have to wait until that PR gets merged and makes it into a new tor release (version 0.3.5). Then, OnionShare can create v3 onion services using the tor 0.3.5 binary, and those onion services shouldn't have this problem. So my idea is to only support v3 onion services is it's using tor 0.3.5+. Am I misunderstanding how this bug fix will work? |
@micahflee all I am just saying is that in case this bugfix gets backported to some version below 0.3.5 it will be easier to detect if there was a flag. It's seems not to be an issue here, version check should suffice. |
I just added Tor's nightly-master debian repository and installed tor (I also tested and I can't reproduce it yet in So yeah, I think the way to resolve this is to disable v3 onion services unless tor is new enough. |
I'm keeping this issue open until there's a Tor Browser release that includes a version of Tor that fixes the v3 bug. When that happens I need to:
|
Blocked waiting for a Tor Browser release. |
Just while using OnionShare (built from the develop branch) to send a file to myself, I'm running into a weird issue. When I try sharing a 899kb jpg to myself, the server stops and the client thinks it's complete, but it only downloaded 250kb. I was able to reproduce this.
The text was updated successfully, but these errors were encountered: