Update repos as Tor hidden services (onions)

[andrewdavidwong]

Currently, users have the option of downloading all updates (for both dom0 and TemplateVMs) over Tor. This is good, but the update repos are still regular clearnet servers. It would be even better (from a privacy perspective) if these update repos were Tor hidden services (onions), since then all traffic would stay within the Tor network.

@adrelanos [https://github.com/QubesOS/qubes-issues/issues/1159#issuecomment-241519552]:

b) Accessing repositories over Tor hidden services (onions) will be implemented for Whonix 14. At least for a transitional period (Whonix 14) we will be using both, onions and regular http repositories in apt sources lists. (If that works out great, regular http repositories will be removed in Whonix 15.)

Accessing repositories over Tor hidden services (onions) may be mostly possible for Qubes also.

Debian, The Tor Project, and Whonix now all allow apt-get updating using Tor hidden services (onions).

• https://onion.debian.org
• https://onion.torproject.org

Afaik Fedora does not offer updating using Tor hidden services (onions), but we could post a feature request.

Branched from #1159.

[adrelanos]

This is also better from a security perspective, since this takes out Tor exit relays from the equation. By doing so it removes Tor exit relays ability to exploit hypothetical flaws in apt-get and/or gpg.

This could be the default if someone opts in using Tor by default.

Implementing this ticket requires an official qubes-os.org Tor onion host or mirror. (Not sure if #1352 is enough to track that task.)

[adrelanos]

This is also better from a security perspective, since this takes out Tor exit relays from the equation. By doing so it removes Tor exit relays ability to exploit hypothetical flaws in apt-get and/or gpg.

This could be the default if someone opts in using Tor by default.

Implementing this ticket requires an official qubes-os.org Tor onion host or mirror. (Not sure if #1352 is enough to track that task.)

[andrewdavidwong]

(Not sure if #1352 is enough to track that task.)

No, that's just for the website (mirroring github-pages). Created a separate one: #2266.

[andrewdavidwong]

(Not sure if #1352 is enough to track that task.)

No, that's just for the website (mirroring github-pages). Created a separate one: #2266.

[fortasse]

@andrewdavidwong: I can help with this, I think you discussed with @adrelanos some of the specifics.

What I need:

• The rsync repo(s) to clone from
• The subdomain on kkkkkkkkkk63ava6.onion that each repo corresponds to
• How frequent the polling should be (I think I saw ~hourly on our forums here, does that still seem reasonable?)

Once I have that, I'll get everything set up and tested.

[fortasse]

@andrewdavidwong: I can help with this, I think you discussed with @adrelanos some of the specifics.

What I need:

• The rsync repo(s) to clone from
• The subdomain on kkkkkkkkkk63ava6.onion that each repo corresponds to
• How frequent the polling should be (I think I saw ~hourly on our forums here, does that still seem reasonable?)

Once I have that, I'll get everything set up and tested.

[marmarek]

The rsync repo(s) to clone from

[user@testvm ~]$ rsync ftp.qubes-os.org::
qubes-mirror   	Qubes OS authoritative repo
[user@testvm ~]$ rsync ftp.qubes-os.org::qubes-mirror
drwxrwsr-x              5 2015/07/17 15:04:18 .
drwxrwsr-x             56 2016/09/28 21:52:39 iso
drwxrwsr-x              4 2015/07/17 15:11:50 repo
[user@testvm ~]$ rsync ftp.qubes-os.org::qubes-mirror/repo/
drwxrwsr-x              4 2015/07/17 15:11:50 .
drwxrwsr-x              6 2016/06/01 15:40:44 deb
drwxrwsr-x             12 2016/05/15 23:27:27 yum

I think the whole repo/deb dir can be placed on qubes-deb.kkkkkkkkkk63ava6.onion, and similarly repo/yum on qubes-yum.kkkkkkkkkk63ava6.onion.
Warning - repo/yum is quite large - 75GB, as it contains also template images. If you want, you can exclude them (and probably also exclude unstable subdirectories), then it's down to 11G. Narrowing it down to only r3.x (so, exclude r1 and r2), makes it 6.1GB. But if space is not a concern, it's easier to mirror it all.

How frequent the polling should be (I think I saw ~hourly on our forums here, does that still seem reasonable?)

Hourly polling is ok.

[marmarek]

The rsync repo(s) to clone from

[user@testvm ~]$ rsync ftp.qubes-os.org::
qubes-mirror   	Qubes OS authoritative repo
[user@testvm ~]$ rsync ftp.qubes-os.org::qubes-mirror
drwxrwsr-x              5 2015/07/17 15:04:18 .
drwxrwsr-x             56 2016/09/28 21:52:39 iso
drwxrwsr-x              4 2015/07/17 15:11:50 repo
[user@testvm ~]$ rsync ftp.qubes-os.org::qubes-mirror/repo/
drwxrwsr-x              4 2015/07/17 15:11:50 .
drwxrwsr-x              6 2016/06/01 15:40:44 deb
drwxrwsr-x             12 2016/05/15 23:27:27 yum

I think the whole repo/deb dir can be placed on qubes-deb.kkkkkkkkkk63ava6.onion, and similarly repo/yum on qubes-yum.kkkkkkkkkk63ava6.onion.
Warning - repo/yum is quite large - 75GB, as it contains also template images. If you want, you can exclude them (and probably also exclude unstable subdirectories), then it's down to 11G. Narrowing it down to only r3.x (so, exclude r1 and r2), makes it 6.1GB. But if space is not a concern, it's easier to mirror it all.

How frequent the polling should be (I think I saw ~hourly on our forums here, does that still seem reasonable?)

Hourly polling is ok.

[fortasse]

So it's just qubes-deb.kkkkkkkkkk63ava6.onion and qubes-yum.kkkkkkkkkk63ava6.onion ?

Space is not a concern, we can do the whole thing.

[fortasse]

So it's just qubes-deb.kkkkkkkkkk63ava6.onion and qubes-yum.kkkkkkkkkk63ava6.onion ?

Space is not a concern, we can do the whole thing.

[marmarek]

So it's just qubes-deb.kkkkkkkkkk63ava6.onion and qubes-yum.kkkkkkkkkk63ava6.onion ?

I think so. Then it will be a drop-in replacement for deb.qubes-os.org and yum.qubes-os.org.

[marmarek]

So it's just qubes-deb.kkkkkkkkkk63ava6.onion and qubes-yum.kkkkkkkkkk63ava6.onion ?

I think so. Then it will be a drop-in replacement for deb.qubes-os.org and yum.qubes-os.org.

[fortasse]

Sounds good. I'll start pulling down the rsync, and I'll post here again when I have updates. Thanks for the quick response @marmarek!

[fortasse]

Sounds good. I'll start pulling down the rsync, and I'll post here again when I have updates. Thanks for the quick response @marmarek!

[fortasse]

@marmarek: I don't have a qubes test environment available at the moment, can you check and make sure everything's in order?

qubes-deb.kkkkkkkkkk63ava6.onion and qubes-yum.kkkkkkkkkk63ava6.onion should be up and running. I also have qubes-mirror.kkkkkkkkkk63ava6.onion available as a top-level. As a note, the mirror is only available over hidden service at the moment - do you need it to be accessible over HTTPS as well?

[fortasse]

@marmarek: I don't have a qubes test environment available at the moment, can you check and make sure everything's in order?

qubes-deb.kkkkkkkkkk63ava6.onion and qubes-yum.kkkkkkkkkk63ava6.onion should be up and running. I also have qubes-mirror.kkkkkkkkkk63ava6.onion available as a top-level. As a note, the mirror is only available over hidden service at the moment - do you need it to be accessible over HTTPS as well?

[marmarek]

On Debian stretch I've got:

  Direct connection to .onion domains is blocked by default. If you meant to use Tor remember to use tor+http instead of http.

The VM is behind Whonix Gateway, so tor+http shouldn't be needed (IIUC it would run tor-over-tor). Any idea?

On Whonix Workstation (-> jessie based) it works fine. On Fedora also looks fine.

Hidden service only is ok. We have excellent mirror at mirrors.kernel.org for non-tor traffic. Thanks!

[marmarek]

On Debian stretch I've got:

  Direct connection to .onion domains is blocked by default. If you meant to use Tor remember to use tor+http instead of http.

The VM is behind Whonix Gateway, so tor+http shouldn't be needed (IIUC it would run tor-over-tor). Any idea?

On Whonix Workstation (-> jessie based) it works fine. On Fedora also looks fine.

Hidden service only is ok. We have excellent mirror at mirrors.kernel.org for non-tor traffic. Thanks!

[fortasse]

Hm, from what I can gather that's an apt-generated message trying to keep you from making mistakes?

There's a Debian bug report here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754242. There appears to be a workaround, but I'm not sure it would work here.

[fortasse]

Hm, from what I can gather that's an apt-generated message trying to keep you from making mistakes?

There's a Debian bug report here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754242. There appears to be a workaround, but I'm not sure it would work here.

[fortasse]

Ah, there's a flag you can set. From this Whonix forums post:

In /etc/apt/apt.conf.d/30user set Acquire::BlockDotOnion "false";

[fortasse]

Ah, there's a flag you can set. From this Whonix forums post:

In /etc/apt/apt.conf.d/30user set Acquire::BlockDotOnion "false";

[marmarek]

That helped, now it works :)

[marmarek]

That helped, now it works :)

[fortasse]

Good to know. That's apparently a newer addition to apt, and probably a smart one for the vast majority of users. Good to keep in mind going forward though.

I'll keep an eye on this ticket for the next few days in case something goes wrong, but are we done here for the time being?

[fortasse]

Good to know. That's apparently a newer addition to apt, and probably a smart one for the vast majority of users. Good to keep in mind going forward though.

I'll keep an eye on this ticket for the next few days in case something goes wrong, but are we done here for the time being?

[marmarek]

I think so :)

[marmarek]

I think so :)

[andrewdavidwong]

@fortasse: Thank you! @adrelanos also mentioned that you would be able to handle #1352. What do you think?

[andrewdavidwong]

@fortasse: Thank you! @adrelanos also mentioned that you would be able to handle #1352. What do you think?

[fortasse]

Probably. We can discuss further in #1352.

[fortasse]

Probably. We can discuss further in #1352.

[andrewdavidwong]

Thanks for making these, @fortasse! The next step is to implementing using these for updates in Qubes. I'll make a separate issue for that.

[andrewdavidwong]

Thanks for making these, @fortasse! The next step is to implementing using these for updates in Qubes. I'll make a separate issue for that.

[anonmos1]

Could you please update the hidden service mirrors as yum.qubesos4rrrrz6n4.onion and deb.qubesos4rrrrz6n4.onion ftp.qubesos4rrrrz6n4.onion as I assume this address didn't exist at the time you used whonix's address. though not a big deal it would be more appropriate and less confusing

[anonmos1]

Could you please update the hidden service mirrors as yum.qubesos4rrrrz6n4.onion and deb.qubesos4rrrrz6n4.onion ftp.qubesos4rrrrz6n4.onion as I assume this address didn't exist at the time you used whonix's address. though not a big deal it would be more appropriate and less confusing

[fortasse]

@anonmos1: Done. I also have the old addresses still active in case someone is already using them. Is there any reason to disable the *.kkkkkkkkkk63ava6.onion addresses?

[fortasse]

@anonmos1: Done. I also have the old addresses still active in case someone is already using them. Is there any reason to disable the *.kkkkkkkkkk63ava6.onion addresses?