implement qrexec based updates proxy

[adrelanos]

Quote @marmarek:

have updates proxy running over qrexec instead of TCP/IP, so template will not have its own netvm at all

[adrelanos]

Could this be implemented similar to qubes-download-dom0-updates / qubes-download-dom0-updates.sh?

[adrelanos]

Could this be implemented similar to qubes-download-dom0-updates / qubes-download-dom0-updates.sh?

[marmarek]

In theory yes, but in practice it's a lot more work to do. Because you have various distributions in Template VMs and each of them have different repository format, different signing scheme etc. This mechanism should support all of them...

[marmarek]

In theory yes, but in practice it's a lot more work to do. Because you have various distributions in Template VMs and each of them have different repository format, different signing scheme etc. This mechanism should support all of them...

[adrelanos]

It would be awesome if the qrexec based updates proxy could be implemented with Whonix stream isolation in mind.

https://forums.whonix.org/t/improve-apt-get-stream-isolation

[adrelanos]

It would be awesome if the qrexec based updates proxy could be implemented with Whonix stream isolation in mind.

https://forums.whonix.org/t/improve-apt-get-stream-isolation

[marmarek]

Do you know an application that accept HTTP PROXY stream on stdin and relay it through SOCKS? For HTTP CONNECT requests it should be trivial (just replace one header with another), but that's not all unfortunately.

[marmarek]

Do you know an application that accept HTTP PROXY stream on stdin and relay it through SOCKS? For HTTP CONNECT requests it should be trivial (just replace one header with another), but that's not all unfortunately.

[adrelanos]

Just throwing some thoughts in...

There are polipo and privoxy that can listen for http and forward it to socks. However, these are probably better avoided for security since these are not applications specialized for this simple use case? https://github.com/mischief/http2socks looks great with under 50 lines of code. (untested by me)

Quote https://www.whonix.org/wiki/Tunnels/Connecting_to_Tor_before_a_proxy#Tools

redsocks can also accept "Trans data streams" and can forward them to https, socks4 and socks5 proxies. If you were to use a http proxy (no https, without connect-method), you could access only http sites, no https sites. Rather redsocks can convert UDP DNS queries to TCP DNS queries.

... DNS resolution ...

Search terms are "http2socks" and "http 2 socks". Should I look if there are better choices nowadays?

Maybe we could throw socat into the mix to listen on STDIN and forward it to a http2socks shim?

[adrelanos]

Just throwing some thoughts in...

There are polipo and privoxy that can listen for http and forward it to socks. However, these are probably better avoided for security since these are not applications specialized for this simple use case? https://github.com/mischief/http2socks looks great with under 50 lines of code. (untested by me)

Quote https://www.whonix.org/wiki/Tunnels/Connecting_to_Tor_before_a_proxy#Tools

redsocks can also accept "Trans data streams" and can forward them to https, socks4 and socks5 proxies. If you were to use a http proxy (no https, without connect-method), you could access only http sites, no https sites. Rather redsocks can convert UDP DNS queries to TCP DNS queries.

... DNS resolution ...

Search terms are "http2socks" and "http 2 socks". Should I look if there are better choices nowadays?

Maybe we could throw socat into the mix to listen on STDIN and forward it to a http2socks shim?

[marmarek]

Maybe we could throw socat into the mix to listen on STDIN and forward it to a http2socks shim?

I'm talking about forwarding stdin to socks directly from qrexec service (with the translation described above), instead of some single service, because in qrexec service we have access to $QREXEC_REMOTE_DOMAIN variable, which can be used for stream isolation (passed as socks auth). Using a single http2socks instance would have the same problem as tinyproxy.
But since http2socks is such a short code, maybe it's easy to modify it to get the data from stdin instead of tcp listener? Anyone know Go here?

[marmarek]

Maybe we could throw socat into the mix to listen on STDIN and forward it to a http2socks shim?

I'm talking about forwarding stdin to socks directly from qrexec service (with the translation described above), instead of some single service, because in qrexec service we have access to $QREXEC_REMOTE_DOMAIN variable, which can be used for stream isolation (passed as socks auth). Using a single http2socks instance would have the same problem as tinyproxy.
But since http2socks is such a short code, maybe it's easy to modify it to get the data from stdin instead of tcp listener? Anyone know Go here?

[adrelanos]

Will this work for Whonix TemplateVMs / sys-whonix? With stream isolation? Any changes in Whonix required?

[adrelanos]

Will this work for Whonix TemplateVMs / sys-whonix? With stream isolation? Any changes in Whonix required?

[marmarek]

Policy needs to be adjusted to direct Whonix templates to sys-whonix (instead of sys-net). Task for salt formulas used to install Whonix.

As for stream isolation - by default no - it is still going through the same tinyproxy instance regardless of source VM and tinyproxy don't support stream isolation itself. See above discussion - one possibility is to modify http2socks to support this, but unfortunately I don't know Go...

[marmarek]

Policy needs to be adjusted to direct Whonix templates to sys-whonix (instead of sys-net). Task for salt formulas used to install Whonix.

As for stream isolation - by default no - it is still going through the same tinyproxy instance regardless of source VM and tinyproxy don't support stream isolation itself. See above discussion - one possibility is to modify http2socks to support this, but unfortunately I don't know Go...

[marmarek]

As for changes in Whonix - I think not strictly required.
One possible improvement is port 8082 can no longer be available from outside of Whonix Gateway - all connections are on loopback interface (127.0.0.1). But this is only true for Qubes 4.0, for Qubes 3.2 (which is still supported for some time) it is still required to access tinyproxy from templates.

Another thing is updates proxy is now configured to 127.0.0.1:8082. Probably no change in Whonix required here.

[marmarek]

As for changes in Whonix - I think not strictly required.
One possible improvement is port 8082 can no longer be available from outside of Whonix Gateway - all connections are on loopback interface (127.0.0.1). But this is only true for Qubes 4.0, for Qubes 3.2 (which is still supported for some time) it is still required to access tinyproxy from templates.

Another thing is updates proxy is now configured to 127.0.0.1:8082. Probably no change in Whonix required here.

[qubesos-bot]

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.0-1.fc24 has been pushed to the r4.0 testing repository for the Fedora fc24 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.0-1.fc24 has been pushed to the r4.0 testing repository for the Fedora fc24 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.0-1.fc25 has been pushed to the r4.0 testing repository for the Fedora fc25 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.0-1.fc25 has been pushed to the r4.0 testing repository for the Fedora fc25 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.0-1+deb8u1 has been pushed to the r4.0 testing repository for the Debian jessie template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing jessie-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.0-1+deb8u1 has been pushed to the r4.0 testing repository for the Debian jessie template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing jessie-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.0-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian stretch template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.0-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian stretch template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[davex25]

When I do a sudo dnf update --verbose in a template I upgraded from 3.2, I see this:

"Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f26&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f26&arch=x86_64 [Failed to connect to 127.0.0.1 port 8082: Connection refused]."

I've modified http to be https in the .repo files.

In a 4.0 fedora template (didn't upgrade it from old qubes), sudo dnf update seems to work fine. Both have "proxy=http://127.0.0.1:8082/" in /etc/dnf/dnf.conf, and .repo files look the same to me.

Where should I look next?

[davex25]

When I do a sudo dnf update --verbose in a template I upgraded from 3.2, I see this:

"Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f26&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f26&arch=x86_64 [Failed to connect to 127.0.0.1 port 8082: Connection refused]."

I've modified http to be https in the .repo files.

In a 4.0 fedora template (didn't upgrade it from old qubes), sudo dnf update seems to work fine. Both have "proxy=http://127.0.0.1:8082/" in /etc/dnf/dnf.conf, and .repo files look the same to me.

Where should I look next?

[marmarek]

Do you have qubes-agent-* packages updated to 4.0.* version, or still at 3.2 there?

[marmarek]

Do you have qubes-agent-* packages updated to 4.0.* version, or still at 3.2 there?

[davex25]

I think 3.2? Not sure how to tell, or how to upgrade them.

[davex25]

I think 3.2? Not sure how to tell, or how to upgrade them.

[marmarek]

Not sure how to tell,

rpm -q qubes-core-agent qubes-core-vm

If that's 3.2.x, you don't have qrexec based proxy there, so you need to enable network for this template (set netvm to sys-firewall in template settings).
To upgrade to 4.0 packages, in the template, as root:

1. Copy /etc/yum.repos.d/qubes-r3.repo to /etc/yum.repos.d/qubes-upgrade.repo
2. Replace every r3.2 with r4.0 in it.
3. Download and import R4 signing key, verify its fingerprint, should be: 5817 A43B 283D E5A9 181A 522E 1848 792F 9E27 95E9
4. Update: dnf update.
5. Install qubes-vm-recommended metapackage, if you don't have it already.
6. Remove now unneded /etc/yum.repos.d/qubes-upgrade.repo.

After that, shutdown the template and set netvm back to none.

Short version:

sed -e 's:r3\.2:r4.0:g' /etc/yum.repos.d/qubes-r3.repo > /etc/yum.repos.d/qubes-upgrade.repo
wget https://keys.qubes-os.org/keys/qubes-release-4-signing-key.asc
gpg --with-fingerprint qubes-release-4-signing-key.asc
# here compare the fingerprint, continue only if matches
rpm --import qubes-release-4-signing-key.asc
dnf update
dnf install qubes-vm-recommended
rm /etc/yum.repos.d/qubes-upgrade.repo

You might hit a problem with qubes-core-agent package during update. Running update the second time fixes it.

[marmarek]

Not sure how to tell,

rpm -q qubes-core-agent qubes-core-vm

If that's 3.2.x, you don't have qrexec based proxy there, so you need to enable network for this template (set netvm to sys-firewall in template settings).
To upgrade to 4.0 packages, in the template, as root:

1. Copy /etc/yum.repos.d/qubes-r3.repo to /etc/yum.repos.d/qubes-upgrade.repo
2. Replace every r3.2 with r4.0 in it.
3. Download and import R4 signing key, verify its fingerprint, should be: 5817 A43B 283D E5A9 181A 522E 1848 792F 9E27 95E9
4. Update: dnf update.
5. Install qubes-vm-recommended metapackage, if you don't have it already.
6. Remove now unneded /etc/yum.repos.d/qubes-upgrade.repo.

After that, shutdown the template and set netvm back to none.

Short version:

sed -e 's:r3\.2:r4.0:g' /etc/yum.repos.d/qubes-r3.repo > /etc/yum.repos.d/qubes-upgrade.repo
wget https://keys.qubes-os.org/keys/qubes-release-4-signing-key.asc
gpg --with-fingerprint qubes-release-4-signing-key.asc
# here compare the fingerprint, continue only if matches
rpm --import qubes-release-4-signing-key.asc
dnf update
dnf install qubes-vm-recommended
rm /etc/yum.repos.d/qubes-upgrade.repo

You might hit a problem with qubes-core-agent package during update. Running update the second time fixes it.

[davex25]

Ok. I did have to run the update for qubes-core-agent twice as you said, but it appeared to install correctly.

After I finished that install, the template vm started behaving like sudo was disabled (great power great responsibility blah blah prompt). I stupidly restarted the template vm at that point, and now when I start it I am unable to run anything (e.g. from a launcher, or from a dom0 terminal using "qvm-run fedora-26-mytemplate gnome-terminal"). So for now I seem to be locked out of that VM.

The guest-fedora-26-mytemplate.log file from /var/log/xen mentions starting the Qubes remote exec agent. Not sure what else to look for in there, or if there are other logs to look at.

[davex25]

Ok. I did have to run the update for qubes-core-agent twice as you said, but it appeared to install correctly.

After I finished that install, the template vm started behaving like sudo was disabled (great power great responsibility blah blah prompt). I stupidly restarted the template vm at that point, and now when I start it I am unable to run anything (e.g. from a launcher, or from a dom0 terminal using "qvm-run fedora-26-mytemplate gnome-terminal"). So for now I seem to be locked out of that VM.

The guest-fedora-26-mytemplate.log file from /var/log/xen mentions starting the Qubes remote exec agent. Not sure what else to look for in there, or if there are other logs to look at.

[marmarek]

Have you installed qubes-vm-recommended?

[marmarek]

Have you installed qubes-vm-recommended?

[davex25]

I believe I did, had to do a couple other things (e.g. comment out the /etc/dnf/dnf.conf proxy), so I did not copy/paste your short version commands verbatim. But not totally sure and can't get back in to check it.

[davex25]

I believe I did, had to do a couple other things (e.g. comment out the /etc/dnf/dnf.conf proxy), so I did not copy/paste your short version commands verbatim. But not totally sure and can't get back in to check it.

[davex25]

It looks like I might not have done the qubes-vm-recommended install. I had another similar template VM that I just upgraded per your instructions, and it seems to be working ok now.

I may have to just delete that other template I tried to upgrade, unless I can find some way to access a terminal on it.

[davex25]

It looks like I might not have done the qubes-vm-recommended install. I had another similar template VM that I just upgraded per your instructions, and it seems to be working ok now.

I may have to just delete that other template I tried to upgrade, unless I can find some way to access a terminal on it.

[marmarek]

Try qvm-run -u root TEMPLATE_NAME xterm
You may also try to access it using sudo xl console TEMPLATE_NAME from dom0 terminal.

[marmarek]

Try qvm-run -u root TEMPLATE_NAME xterm
You may also try to access it using sudo xl console TEMPLATE_NAME from dom0 terminal.

[davex25]

"sudo xl console" is what I was trying to do. Thank you, seems all is good now.

[davex25]

"sudo xl console" is what I was trying to do. Thank you, seems all is good now.

[davex25]

Seems like I spoke too soon. I did install the qubes-vm-recommended on the upgraded VM, but when I run qvm-run TEMPLATE_NAME xterm, the command still hangs and nothing else happens. Currently the only way I can access that VM is via xl console.

[davex25]

Seems like I spoke too soon. I did install the qubes-vm-recommended on the upgraded VM, but when I run qvm-run TEMPLATE_NAME xterm, the command still hangs and nothing else happens. Currently the only way I can access that VM is via xl console.

[marmarek]

Check status of qubes-gui-agent and qubes-qrexec-agent services there.

[marmarek]

Check status of qubes-gui-agent and qubes-qrexec-agent services there.

[davex25]

Qubes-gui-agent was not enabled or running. Now I can launch stuff from dom0.

[davex25]

Qubes-gui-agent was not enabled or running. Now I can launch stuff from dom0.

[davex25]

Seems there are still problems with updating this template. This one was still on fedora 25, when I tried to do an upgrade to fedora 26 like this:

sudo dnf --releasever=26 --setopt=cachedir=/mnt/removable --best --allowerasing distro-sync --verbose

I get this message:

Cannot download 'http://yum.qubes-os.org/r4.0/current/vm/fc26': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried. Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

[davex25]

Seems there are still problems with updating this template. This one was still on fedora 25, when I tried to do an upgrade to fedora 26 like this:

sudo dnf --releasever=26 --setopt=cachedir=/mnt/removable --best --allowerasing distro-sync --verbose

I get this message:

Cannot download 'http://yum.qubes-os.org/r4.0/current/vm/fc26': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried. Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

[davex25]

I tried turning off the localhost proxy in dnf.conf, and using sys-net for NetVM and same thing.

Strange thing is that I set a VM to use this template, and ran the same dnf command as above, and it seemed to have no troubles with dependency resolution and upgrade. But obviously I want to upgrade the actual template.

[davex25]

I tried turning off the localhost proxy in dnf.conf, and using sys-net for NetVM and same thing.

Strange thing is that I set a VM to use this template, and ran the same dnf command as above, and it seemed to have no troubles with dependency resolution and upgrade. But obviously I want to upgrade the actual template.

[marmarek]

Check firewall settings of the template (vm settings -> firewall tab).

[marmarek]

Check firewall settings of the template (vm settings -> firewall tab).

[davex25]

Firewall looks OK. When I switch the upgraded template's NetVM from (none) to sys-net, an IP address is shown for that template in the Qubes Manager, but if I type ip route in a terminal, nothing is shown and I can't ping anything. On the non-upgraded 4.0 template VM, ip route shows what I'd expect ("default via ", etc). So it seems like some network service might not be running in my upgraded template. After some time a route did get assigned (not sure how), but when I noticed that I tried switching back to (none) and then sys-net, and no route again.

The other problem I ran into was that the upgraded qubes-r4.repo file had http instead of https in it. Your documentation for upgrading from 3.2 to 4.0 should include a sed for that in the steps.

[davex25]

Firewall looks OK. When I switch the upgraded template's NetVM from (none) to sys-net, an IP address is shown for that template in the Qubes Manager, but if I type ip route in a terminal, nothing is shown and I can't ping anything. On the non-upgraded 4.0 template VM, ip route shows what I'd expect ("default via ", etc). So it seems like some network service might not be running in my upgraded template. After some time a route did get assigned (not sure how), but when I noticed that I tried switching back to (none) and then sys-net, and no route again.

The other problem I ran into was that the upgraded qubes-r4.repo file had http instead of https in it. Your documentation for upgrading from 3.2 to 4.0 should include a sed for that in the steps.