Tor support

[SamWhited]

Previous discussed in #45.

Current work

• Don't bother checking SRV records for .onion domains
• Use Tor for DNS requests (not applicable since we're only doing .onion addresses for now)
• Remove SOCKS config in favor of Tor only
• Make sure .onion routing is working
• Use Tor only for .onion addresses until SRV lookups are implemented in Tor's DNS
• Don't error on certificate mismatch for .onion; instead, show the warning and let the user choose to pin the invalid cert
• Add Tor support for HTTP connections
• Make sure HTTP connections aren't leaking DNS all over the floor
• Add Tor support for Jingle connections

Possible future improvements

Possible future improvements:

• Add a check to see if Orbot is installed and/or started and display an appropriate error if "Always use Tor" is set or we're connecting to a Tor HS.
• When changing the Tor setting, bounce all non-HS accounts (HS accounts always use Tor).
• Detect Tor proxy and DNS ports instead of hard coding them (Orbot doesn't have an intent for this yet)
• Possibly ensure stream isolation when multiple accounts are in use? This may happen anyways; all the circuit building stuff has changed around since the last time I really dug into Tor in depth.

[SamWhited]

@n8fr8 What's the standard distribution channel for NetCipher? Is there a Maven repo I can pull from? I wanted to experiment with its Orbot integration library later today.

[iNPUTmice]

Is this something we can finish till Sunday the 18th?
I would like to do the feature freeze for 1.0 this weekend (and release the beta)

(No rush if you don't have the time for it you don't have the time for it. But I thought I'd let you know about my plans.)

[SamWhited]

Thanks; I'll see what I can do. I've been busy with a mix of job interviews and work on another project, but shouldn't be too difficult to knock out the rest of my todo list if I just sit down and work on it for a bit.

The only exception would probably be DNS resolution; I still have to look into how that works, but I suspect doing it over Tor won't be possible with minidns.

[iNPUTmice]

doesn't orbot provide a local dns server that can be queried?

[SamWhited]

It may (that would make things a lot easier); I haven't looked into it.

[iNPUTmice]

doesn't look like it unfortunately...
how does Tor resolve domains then? if there are changes required to minidns you can maybe ask @rtreffer for help. afaik he is using conversations himself.

[SamWhited]

If it doesn't provide a separate local DNS serer we'll have to use the SOCKS proxy. SOCKS doesn't support UDP in general, but it has special support for DNS and forwards requests.

[rtreffer]

AFAIK orbot has a DNS port, so it's just a matter of querying another local port (which the client can't right now, but that would be a trivial change).

[SamWhited]

If it does have a DNS port it should be easy enough for us to swap it over; I'm going to be on the road most of today, but I'll check this out tomorrow or later tonight when I get back.

[iNPUTmice]

just looked in the app settings instead of trying to Google that with an unstable connection. it seems to have a dns port on 5400

[SamWhited]

@iNPUTmice @rtreffer Currently the queryDNS function takes the address of the server to query as an argument and the getSRVRecord function handles actually looking up the servers to use (which it then passes to queryDNS). In my mind it makes more sense to handle the lookup of DNS servers in queryDNS as well; this way we can be sure that anything using queryDNS uses the proxy server and we don't have to add more logic elsewhere to get the DNS servers from the system if we were to add more DNS related functions in the future.

Does this make sense, or am I missing something and there's a good reason why queryDNS handles the lookup of the servers initially?

[SamWhited]

I also just realized that the queryDNS function isn't as general purpose as I thought and is only called from getSRVRecord. Making queryDNS private and doing the logic in getSRVRecord would be even cleaner. Doing it this way for now.

[SamWhited]

Notes:

Detect Tor proxy and DNS ports instead of hard coding them

I looked at the Orbot manifest, and unfortunately there doesn't appear to be an intent for this, and getting ahold of the torrc would require root. We could go back to having a general SOCKS proxy option which defaults to the Tor defaults, and a "proxy DNS" port option, but I think it's probably fine just to leave it hardcoded to the Tor defaults.

[n8fr8]

Yeah, we don't have an intent for this yet. Most people use the stock ports, and that's what we do for ChatSecure, as well.

[iNPUTmice]

The DNSHelper was originally a quick and dirty implementation I wrote myself one night to fetch SRV records. When we migrated to using minidns with version 0.1.2 Rene just made the interface compatible with mine. Meaning that DNSHelper interface wasn't meant to be flexible or anything. Just historical growth.
Feel free to do what ever changes you like. I wouldn't even mind getting rid of that static class at all if you think that would be better.

[SamWhited]

Alrighty; that should wrap everything up then. Stress testing time.

Thanks for the info and help all.

[SamWhited]

@rtreffer I misunderstood you before that the client can't query another port. Any chance we could get custom port support added to minidns?

I'm messing about with port forwarding right now, but obviously that's not something that most people will want to setup just to use Tor :)

[rtreffer]

@SamWhited was planning on doing that tonight, especially as it looks like this ticket got some momentum right now :-)
I've a pull request for unit tests waiting, would add that support afterwards and do a release.

[SamWhited]

@rtreffer Sounds great; let me know if you need extra hands or eyes on it.

[rtreffer]

I think I was wrong, it should already be possible with a minor patch.

Attached patch should work. I'm recompiling Conversations right now to
test it (have to set up orbot, too...)

[rtreffer]

@SamWhited I just created SamWhited#1 - compile worked, trying it right now......

[SamWhited]

@rtreffer Awesome; thanks. I'll test too; let me know if / when you're ready for me to merge that.

[rtreffer]

@SamWhited ready to merge / incorperate into this pull request.

But we need a longer DNS timeout for TOR. It looks like the current 5s / query is way to low, the tor client used to allow 300s (at least according to a Mail from 2010).

Working on a patch for minidns.

@iNPUTmice can we switch to the release (maven central) version of minidns?

[SamWhited]

RE our discussion on the sub-PR: I think we should just remove the "Never" option, and let "Only for .onion" essentially BE the "Don't use Tor" option (unless you specify a .onion domain, obviously). So the options (and their behavior) would be:

• Always
  • .onion via Tor
  • Regular via Tor
• Auto
  • .onion via Tor
  • Regular via Tor if Orbot enabled, clearnet otherwise
• Only for .onion (Default)
  • .onion via Tor
  • Regular via clearnet

Regardless of what we call the option, I think @rtreffer is right about NEVER allowing .onions to attempt to resolve with regular DNS as this leaks potentially sensitive information about the user (that they're trying to use Tor). If the user is in a country where Tor is illegal, this could be very bad for them.

[rtreffer]

Just addd a pull request to get rid of the embedded minidns, will create / push a version that can handle timeout changes now. Might take 'till tomorrow for mirrors to copy it.
#886

[iNPUTmice]

I'm fine with leaving the Never option as like you said the only for .onion pretty much covers that.
But I thought we agreed upon not doing the 'Auto' option since that is too dangerous / too unpredictable.

[SamWhited]

@iNPUTmice It seemed like a good idea to me as long as it wasn't the default (it is under expert options afterall), but I keep going back and forth on it. Would you like me to remove it?

Maybe calling it "If Tor is running" or something would make it a bit more clear.

[iNPUTmice]

I honestly don't know... I mean I was the one who originally suggested that. But then I was convinced otherwise and now I can't really picture a use case for that. Either you need / want TOR or you don't...

[iNPUTmice]

Androids nslookup is the busy box nslookup which doesn't do SRV if I'm remembering that correctly from back in the day when I was trying to get SRV support into Conversations

[SamWhited]

Yah, unfortunately that's what it looks like. I'm having trouble getting a Debian chroot working on Android 5 too, so this may be a bit tricky to debug.

[n8fr8]

With ChatSecure, we require the connect server to be manually specified (or we have some pre-baked in) for connecting over Tor. This is mostly because we haven't modified minidns to be able to talk to Orbot/Tor's DNS port ourselves.

[SamWhited]

I'd definitely prefer to get SRV records working (that's going to be very confusing otherwise), but I suspect @iNPUTmice and @rtreffer are correct and Tor's DNS port just doesn't support SRV records at all, and I doubt that doing SOCKs remote name resolution will be an option (I'm trying to figure out how all that works right now; I don't really know much about SOCKS at the protocol level).

@n8fr8 The changes that were made to minidns to allow custom ports / timeouts have been merged upstream (thanks @rtreffer), so you guys should be able to use those too now if you feel so inclined.

[n8fr8]

There are a finite list of XMPP servers that support .onion domains, so having a built-in list is not a bad short-term to achieve the usability you are seeking.

Anyhow found this ticket: https://trac.torproject.org/projects/tor/ticket/7797 and will check with nickm and others if it is truly still open.

[SamWhited]

@n8fr8 Good catch; thanks.

The main problem here isn't with onion domains (which appear to work, since no SRV resolution is required), but with regular domains where the actual server is somewhere else (hence the need for an SRV lookup).

[n8fr8]

Ah right. Yes, we build those in for the common/popular servers, but only use them when Tor is enabled.

[rtreffer]

That's the ticket I originally found. The other ticket ( https://trac.torproject.org/projects/tor/ticket/7829 ) lists places in the code where it could be added. I didn't find references to SRV in those places (see link to dns.c in older comment).

Anyway, I think it would be quite interesting to raise this on jdev: http://mail.jabber.org/mailman/listinfo/jdev (How should tor be handled)

I've also found one Mail saying that - in theory - .onion could appear in SRV replies.... I'm just 100% unsure if anyone is using that.

[SamWhited]

Currently being discussed in #tor-dev on OFTC: https://gitweb.torproject.org/torspec.git/tree/proposals/219-expanded-dns.txt

Unless anyone has a workaround they think would work, I suspect this will be a blocker to Tor support in Conversations.

@rtreffer xmpp.net lists exactly one server with a .onion in the SRV records... so I guess somebody's doing it.

[SamWhited]

§ 0.1 Motivation:

Many applications running over Tor need more than just resolving FQDN to
IPv4 and vice versa. Sometimes to prevent DNS leaks the applications have to
be hacked around to be supplied necessary data by hand (e.g. SRV records in
XMPP).

That last bit amused me...

[rtreffer]

Could this pull request be splitted?
IMHO .onion is the easier part and ready to merge (besides jingles / http which could be disabled on first .onion release) or am I missing s.th.?
And having .onion support would be a great first step towards TOR.

[iNPUTmice]

I guess I would be fine with only supporting TOR for .onion domains since this would also mean less confusing settings (I hate settings) but not having file transfer?!

[SamWhited]

I don't know how useful this would be, but it would be fairly easy as a temporary stopgap for now. I do like fewer settings...

I'll branch and chop out the bits that don't work and we'll see what that leaves us with.

[rtreffer]

The problem with jingle is that it should be a peer to peer connection, disclosing yourself in the process.
This is dangerous on TOR, and should be better left off (or implemented differently, e.g. via a HS).

[SamWhited]

Jingle via HS would be a nifty feature (and creating a HS is actually pretty easy with Orbot)... but I'm getting ahead of myself.

[iNPUTmice]

The problem with jingle is that it should be a peer to peer connection

it uses the servers socks proxy anyway. because cell phones and NAT. so using TOR shouldn't be too much of a problem

[SamWhited]

Made a new branch. Will merge it in here later if we decide to use it. Working on HTTP and Jingle comaptibility now.

[SamWhited]

Got distracted shortly after that last comment; may not be able to get to Jingle/HTTP support tonight after all.

[SamWhited]

Another thing that should probably go in before .onion support is considered "complete": Mismatched certificates should be downgraded from an error to a warning for HS's (show the cert dialog and allow the user to pin the cert like we'd do for a matching self signed certificate). We require TLS, but I don't know of a single certificate authority that will actually issue a cert for a .onion, or any XMPP server that's using a separate self-signed cert for the .onion.

I'm not sure if this is easily possible with MTM or not.

[SamWhited]

Merged in the revert for only using Tor with hidden services and replaced the host name verifier with a no-op version for .onion's as well.

One of the HS's I tested doesn't present a peer certificate which causes the memorizing trust manager to hiccup and we get a security error still; not sure what the best course of action for this is (or if we even care).

[SamWhited]

Added support for proxying HTTP stuff through Tor if the account is a .onion account. It's using the built in SOCKS4 proxy class (which means DNS will be leaked and no onion addresses again). Next step is to figure out where to set a custom resolver so we can point it at Tor's local DNS (which should work fine for this since links will most likely point to addresses that are A, AAAA, or possibly CNAME records).

[mdik]

Regarding DNS lookup (sorry if I missed if this was proposed earlier, these threads are long):

Would it be hard to cache the server IP address on the device? If not I could imagine that you make the very first connection to the server without Tor, and after that, whenever the SRV lookup fails the client defaults on the IP it remembers from previous connections.

This would mean that the casual Tor user would not be bothered at all, because everything would be handled by Conversations itself.

[SamWhited]

DNS lookup was resolved, orbot provides a DNS port.

[mdik]

Ok. So Orbot is more capable than standard Tor in this regard then?

[SamWhited]

I'm not sure if "more capable" is how I'd put it; it's just working around limitations in the Android API / of developing on a phone. Standard Tor just acts as a SOCKS5 proxy which can handle DNS itself, but when developing for Android you might not be able to use the SOCKS proxy that way so they provide you a normal DNS endpoint (that itself works over the SOCKS proxy) for you to use. You could easily setup a local DNS server pointing to Tor on your desktop machine too.

EDIT: What I'm trying to say is that orbot is "standard Tor". It's just bundled into an app with some other useful stuff. You could easily do the same on a desktop; it's a difference in distribution, not in one being "more capable". I hope that makes sense.