Add V3 onion support without "Persistent onion" for the time being

[Baccount]

The UI for now just needs to disable the possibility to set persistent address and stealth onion service when using v3 Services. IMO v3 should be the default so the Settings could say e.g. @maqp

• Use v3 onion services
  (unchecking will use old, less secure v2 onion services)

• Use a persistent address (v2 only)

• Create stealth onion services (v2 only)*

When v3 is checked, the two settings should be greyed out.

*prop224 currently does not specify stealth client authorization.

#461 (comment)
@mig5 has a branch that does support V3 onions, but there would be some work to do.
https://github.com/mig5/onionshare/tree/prop224

Current problems with Persistent V3 onions

• Nick m just merged a patch that may resolve the issue with the rev counter.

https://gitweb.torproject.org/tor.git/commit/?id=5804ccc9070dc5443f1c6ce565dbf17572812764

Asn said

• Onion service rev counters are useless and actually harmful for scalability
  https://trac.torproject.org/projects/tor/ticket/25552

@maqp said

• I'm still getting the UPLAOAD_REJECTED. This is most likely because HSDirs are still using the latest stable or at best, the lastest alpha release. So it may take days, weeks or months before we know it was fixed.

some of above comments are by @maqp
#461 (comment) #461 (comment)

[micahflee]

I like this idea. I think v3 support without the extra features is a better default than v2 support with extra features. It still looks non-trivial to implement. I started looking at @mig5's branch but it'll take some time to get working. But I think this is a good idea.

[maqp]

The fix for UPLOAD_REJECTED issue has been moved until Tor 0.3.5, so it would appear persistent v3 onion services won't work reliably until December of this year. The recent issue where Stem was unable to create a service at all, already has a fix candidate. Therefore, it's possible to start working on the ephemeral v3 service fairly soon.

[maqp]

The fix for the recent issue is now merged to Tor's master branch.

[Baccount]

@maqp Tor 0.3.4.2 was released with the 25939 fix. And it torproject says that 0.3.4.2 will become available in the torbrowser alpha in the coming weeks. So Onionshare could update to the latest Tor Browser alpha when it is released, and it should finally be possible to implement the feature as described in this ticket.

The patch for the rev counter has been published as a open draft. torproject/tor#150

   Improve revision counter generation in next-gen onion services. Onion
   services can now scale by hosting multiple instances on different hosts
   without synchronization between them, which was previously impossible
   because descriptors would get rejected by HSDirs. Addresses ticket 25552.

@maqp

[asn-d6]

Hey guys, why do you care about torproject/tor#150 ? You should only care about that branch (ticket #25552 in trac) if you plan to have multiple instances of onionshare hosting the same HSv3 from different hosts.

IIUC, onionshare does not care about this scaling use case right now, so perhaps torproject/tor#150 is not something you should care about.

Let me know if I'm mistaken :)

[maqp]

Hi @asn-d6! Currently creating an ephemeral v3 Onion Service multiple times with Stem, using the same expanded private key results to UPLOAD_REJECTED error. IIUC, this is because HSDirs are rejecting the descriptor due to re-used rev counters. It's however possible to recreate the Onion Service using the same private key after one hour (possibly due to descriptor TTL or HSDir rate limiting).

This error seems to take 2-5 tries of descriptor publishing. The issue I see for OnionShare is, start and stop of sharing files (e.g. user forgets to include some documents, or they just want to stop sharing for a moment) under persistent address leads to the UPLOAD_REJECTED issue, and they need to wait for an hour before it's possible to bring the Onion Service online again. This isn't that huge a deal with OnionShare, but instant messaging like TFC and possibly Ricochet are hard to use if user can only login once an hour.

So most likely the issue isn't present just when scaling but also when the persistent onion service goes online and offline repeatedly.

OnionShare has previously been able to create persistent v2 Onion Services, where Tor manages the private key itself. This has however relied on the Onion URL being random when the service is brought up the first time. There has been discussion about implementing a dead man's switch that generates the private key and .onion-address beforehand and shows them to user even before v2 HSDirs learn about the address. It's better if this works without dry-run of launching v2/3 service (for key generation), and if OnionShare can generate the private key itself and derive the to-become .onion address from that. To do that, I've been trying to use Tor's testing code to create a python3 version of key expansion function for v3 services. It seems to work but I'm not sure if it's correct. The output was the same as the original v2 code with random 32-byte inputs. So if this approach is incorrect, that might also explain the UPLOAD_REJECTED issue (assuming there shouldn't be errors since there's no scaling going on).

So could you please advice on this?

[asn-d6]

Hello @maqp!

Hi @asn-d6! Currently creating an ephemeral v3 Onion Service multiple times with Stem, using the same expanded private key results to UPLOAD_REJECTED error. IIUC, this is because HSDirs are rejecting the descriptor due to re-used rev counters. It's however possible to recreate the Onion Service using the same private key after one hour (possibly due to descriptor TTL or HSDir rate limiting).
This error seems to take 2-5 tries of descriptor publishing. The issue I see for OnionShare is, start and stop of sharing files (e.g. user forgets to include some documents, or they just want to stop sharing for a moment) under persistent address leads to the UPLOAD_REJECTED issue, and they need to wait for an hour before it's possible to bring the Onion Service online again. This isn't that huge a deal with OnionShare, but instant messaging like TFC and possibly Ricochet are hard to use if user can only login once an hour.
So most likely the issue isn't present just when scaling but also when the persistent onion service goes online and offline repeatedly.

Hmm! That seems plausible but I still don't understand it enough!

Tor should be writing the revision counter value to its state file and using
that (incremented) next time it starts up, so that it doesn't re-use the same rev
counter. How come that doesn't work in your use case? Are the v3s on the same host?
Is a state file being used?

In any case, #25552 should help with this indeed, but I'm wondering why it's
currently not working.

OnionShare has previously been able to create persistent v2 Onion Services, where Tor manages the private key itself. This has however relied on the Onion URL being random when the service is brought up the first time. There has been discussion about implementing a dead man's switch that generates the private key and .onion-address beforehand and shows them to user even before v2 HSDirs learn about the address. It's better if this works without dry-run of launching v2/3 service (for key generation), and if OnionShare can generate the private key itself and derive the to-become .onion address from that. To do that, I've been trying to use Tor's testing code to create a python3 version of key expansion function for v3 services. It seems to work but I'm not sure if it's correct. The output was the same as the original v2 code with random 32-byte inputs. So if this approach is incorrect, that might also explain the UPLOAD_REJECTED issue (assuming there shouldn't be errors since there's no scaling going on).

Hmm, I'm also not sure I understand this either.

When you say The output was the same as the original v2 code with random 32-byte inputs. you mean with the original v3 code, right? If yes, that seems fine to me.

I also don't see how this could cause the UPLOAD_REJECTED issue.

BTW, what's the use case behind the start-up timer ticket that you cited?

Are you guys on the #tor-dev IRC? Perhaps we can talk about all these things there. I'd love to help you out and resolve all such issues. Also if this is reproducible, we currently have a branch in Tor's #25552 you might want to test out :)

[maqp]

@asn-d6

Are the v3s on the same host?

Yes.

Is a state file being used?

The state file AFAIK isn't being used since the program is calling controller.create_ephemeral_hidden_service. The script launches Tor with config that contains 'AvoidDiskWrites': '1' KV-pair. Changing the value to '0' seems to have no effect on whether the UPLOAD_REJECTED happens or not.

you mean with the original v3 code, right?

Yes, my bad. I meant to write The output [of my Python 3 code] was the same as the original Python 2 code with random 32-byte inputs.

BTW, what's the use case behind the start-up timer ticket that you cited?

The idea behind the startup-timer was a dead man's switch: e.g. a whistleblower could set up a system that publishes documents under pre-announced Onion URL on set date unless they decide to cancel it first.

Are you guys on the #tor-dev IRC?

I've talked about the issue there with some people including arma, but I'm not regularly there. I'll try to be more present there for now.

The bug is reproducible. I've tried to make that as easy as possible with a bash script that sets up the environment and that downloads the python script that reproduces the UPLOAD_REJECTED. You can obtain the latest version of the bash script here and the latest Python script here.

It seems to work reliably (aside issues with dpkg lock during background updates) with e.g. freshly installed Ubuntu 18.04 virtual machine. Please make sure to skim what the code does before running it.

[maqp]

@asn-d6 @Baccount @mig5 Well I'll be. I changed the script to use @asn-d6's bug25552_ope_draft. The new script was able to launch the Onion Service using the same key 100 times before I stopped it. It really seems to fix the issue!

@asn-d6 Are the changes in that branch only in the client, or is the client connected to something like test network or test HSDirs for now?

[asn-d6]

@asn-d6 Are the changes in that branch only in the client, or is the client connected to something like test network or test HSDirs for now?

Good to learn that the branch helps you out!

The changes in the branch are strictly service-side. Only the onion service code/behavior is changed by the branch; the behavior of clients and HSDirs does not change at all. Is that what you asked?

[maqp]

@asn-d6

The changes in the branch are strictly service-side. Only the onion service code/behavior is changed by the branch; the behavior of clients and HSDirs does not change at all. Is that what you asked?

Yes. That's really good to know. It now makes it possible to start developing the feature while waiting for Tor 0.3.5 to release.

I'm really glad this doesn't have to wait for HSDirs to update. Related to that I'm curious, in the case it would have to wait, are changes to HSDirs' code something that can be pushed to maintainers of HSDirs and if not, how quickly do they generally react to new releases?

[asn-d6]

I'm really glad this doesn't have to wait for HSDirs to update. Related to that I'm curious, in the case it would have to wait, are changes to HSDirs' code something that can be pushed to maintainers of HSDirs and if not, how quickly do they generally react to new releases?

I don't really know how quickly HSDirs in particular respond to new releases, but you can see how the network responds here: https://metrics.torproject.org/versions.html

All in all, it would suck if we had to wait for HSDirs to update, but we could potentially speed it up if a patch is particularly important, by backporting fixes to earlier releases and writing blog posts... Let's hope we dont need to do that. :)

[mig5]

Exciting - https://trac.torproject.org/projects/tor/ticket/25552 was merged in yesterday.

I'm somewhat inclined to avoid complicating our logic/UI if we were to do this ticket (allow v3 without persistent mode), and instead just wait a few more months before re-implementing the v3 logic, including self-generating the keys in advance per my prop224 branch (opening up the opportunity for Dead Man's Switch) since it sounds like Stem won't implement that pre-generation idea upstream.

After all, we've waited this long already.. a few more months and we can just implement v3 officially, persistent or not, and with pre-generating keys for future features like Dead Man's Switch/faster UI feedback re: what the URL will be..

In other words I personally don't fully consider v3 'stable' until this fix has landed in stable Tor release. Once it is, we can support it.

[mig5]

Changed my mind on the above because @micahflee said the exact opposite earlier in this ticket and I missed it.

Taken a new pass at v3 onions, this time against the receiver-mode-gui branch since it contains so much other refactoring already. Option to 'use legacy v2 mode' now exists and allows persistence/stealth - otherwise if Tor version supports it, v3 onions will be used for both sharing and receiving.

#731

There are a couple of extra Python dependencies which testers would need to install (see BUILD.md for installing those in the usual way depending on your OS)

[mig5]

v3 support (without persistence or stealth for now) has been merged into develop along with the receiver-mode-gui work!