[Contribution] qvm-expose-port

[niccokunzmann]

Qubes OS version:

R3.2

Affected component(s):

dom0

---

Steps to reproduce the behavior:

Expected behavior:

I would like to expose a port from a VM to the outside.

Actual behavior:

I can read a long documentation and may fail.

General notes:

I implemented a script qvm-expose-port to do the work for me and
published it on GitHub.

I am offering it to you and would like to know if it is useful to add this to Qubes.
I am using it because I develop servers I would like to share.

---

Related issues:

[andrewdavidwong]

Thanks for your contribution! Please see the Package Contributions page for the package contribution procedure.

[andrewdavidwong]

Thanks for your contribution! Please see the Package Contributions page for the package contribution procedure.

[tasket]

Not sure if its complementary, but @jpouellet also has a port-forwarding script:

"Forwards a specified port to a specified VM, auto-detecting its NetVM chain. (Qubes OS)"
https://gist.github.com/jpouellet/d8cd0eb8589a5b9bf0c53a28fc530369

[tasket]

Not sure if its complementary, but @jpouellet also has a port-forwarding script:

"Forwards a specified port to a specified VM, auto-detecting its NetVM chain. (Qubes OS)"
https://gist.github.com/jpouellet/d8cd0eb8589a5b9bf0c53a28fc530369

[niccokunzmann]

@tasket Thanks! Posting this clarifies that I should use qvm-run -u root as pointed out in niccokunzmann/qvm-expose-port#1

Also, I like to point out that they know

• how to recurse
• how to get the IP without grep

[niccokunzmann]

@tasket Thanks! Posting this clarifies that I should use qvm-run -u root as pointed out in niccokunzmann/qvm-expose-port#1

Also, I like to point out that they know

• how to recurse
• how to get the IP without grep

[jpouellet]

If people actually want this, then IMO we should integrate it properly rather than as a fragile script in an optional package, and make it portable across different potential NetVMs/ProxyVMs (e.g. not make assumptions about the use or even availability of iptables).

I believe the correct way to implement this would be via a new "forward" action type for firewall rules. We already have infrastructure to handle reading/writing these rules, and the rules' intent are already provided to guests in a mostly-guest-OS-agnostic way via qubesdb, and we already have infrastructure in the linux agent for detecting changes to this exposed information and triggering idempotent iptables updates from them.

Some open questions are then:

• how to handle the case where multiple VMs both have rules requesting the same port be forwarded to them? One possible solution may be for adding a forwarding rule for a VM to remove all rules for the same port on other VMs with the same NetVM, but it's not clear to me that this is the best way.
• how to express mappings from one port to a different one (e.g. 80->8080)? The firewall rules at the moment only have a single port range field (internally referred to as dstports). Should we overload the semantics of a port range to mean lower bound is the port in the ingress NetVM side and the upper bound is the port on the VM providing the service? Should we add another field instead?

[jpouellet]

If people actually want this, then IMO we should integrate it properly rather than as a fragile script in an optional package, and make it portable across different potential NetVMs/ProxyVMs (e.g. not make assumptions about the use or even availability of iptables).

I believe the correct way to implement this would be via a new "forward" action type for firewall rules. We already have infrastructure to handle reading/writing these rules, and the rules' intent are already provided to guests in a mostly-guest-OS-agnostic way via qubesdb, and we already have infrastructure in the linux agent for detecting changes to this exposed information and triggering idempotent iptables updates from them.

Some open questions are then:

• how to handle the case where multiple VMs both have rules requesting the same port be forwarded to them? One possible solution may be for adding a forwarding rule for a VM to remove all rules for the same port on other VMs with the same NetVM, but it's not clear to me that this is the best way.
• how to express mappings from one port to a different one (e.g. 80->8080)? The firewall rules at the moment only have a single port range field (internally referred to as dstports). Should we overload the semantics of a port range to mean lower bound is the port in the ingress NetVM side and the upper bound is the port on the VM providing the service? Should we add another field instead?

[jpouellet]

There is also the issue of direct recursive forwarding up the NetVM chain not being appropriate for all VMs. For example, if some VM is behind a ProxyVM which e.g. transparently tunnels all traffic at layer 2, then a port forward on that ProxyVM's NetVM does not make sense since the ports to be forwarded only exist on the overlay network (inside the proxy tunnel). Some way of denoting that forwarding should stop is required. It seems the introduction of a qvm-features flag would be a good way to express this. ProxyVMs which only act as firewalls (e.g. sys-firewall) could set this flag to mean "establish further forwarding firewall rules in NetVMs upstream of me", or not set it to not do so.

I believe the least-surprising-to-users way to introduce this would be a positive feature saying "do forward upstream" rather than a negative feature saying "do not foward upstream", and setting the feature by default on sys-firewall. This way people with existing ProxyVMs do not suddenly get ports forwarded from their NetVM to their ProxyVM as a surprise, but must be aware of this feature and manually opt in on the ProxyVMs for which it makes sense for them.

Following from this, creating a forwarding rule in a particular VM could automatically create corresponding forwarding rules in upstream NetVMs so long as they have the "auto-port-forward" (or "propagate-port-mappings", or whatever) flag set. This would eliminate the need for a separate port-forwarding tool altogether, and allow easy integration into firewall-management GUIs such as qubes-manager without duplicating netvm-recursion logic, and keeping all semantics contained in the core firewall handling logic.

[jpouellet]

There is also the issue of direct recursive forwarding up the NetVM chain not being appropriate for all VMs. For example, if some VM is behind a ProxyVM which e.g. transparently tunnels all traffic at layer 2, then a port forward on that ProxyVM's NetVM does not make sense since the ports to be forwarded only exist on the overlay network (inside the proxy tunnel). Some way of denoting that forwarding should stop is required. It seems the introduction of a qvm-features flag would be a good way to express this. ProxyVMs which only act as firewalls (e.g. sys-firewall) could set this flag to mean "establish further forwarding firewall rules in NetVMs upstream of me", or not set it to not do so.

I believe the least-surprising-to-users way to introduce this would be a positive feature saying "do forward upstream" rather than a negative feature saying "do not foward upstream", and setting the feature by default on sys-firewall. This way people with existing ProxyVMs do not suddenly get ports forwarded from their NetVM to their ProxyVM as a surprise, but must be aware of this feature and manually opt in on the ProxyVMs for which it makes sense for them.

Following from this, creating a forwarding rule in a particular VM could automatically create corresponding forwarding rules in upstream NetVMs so long as they have the "auto-port-forward" (or "propagate-port-mappings", or whatever) flag set. This would eliminate the need for a separate port-forwarding tool altogether, and allow easy integration into firewall-management GUIs such as qubes-manager without duplicating netvm-recursion logic, and keeping all semantics contained in the core firewall handling logic.

[jpouellet]

OTOH, I suspect such a mechanism would be R4+ only since in R3.2 the firewall rules were not provided in a guest-agnostic format (although I suppose they could be shoved into the provided iptables rules anyway) and IIRC qube feature flags did not exist. So, I see nothing wrong with wanting to package up a short script without forward-compatibility guarantees for R3.2 users.

[jpouellet]

OTOH, I suspect such a mechanism would be R4+ only since in R3.2 the firewall rules were not provided in a guest-agnostic format (although I suppose they could be shoved into the provided iptables rules anyway) and IIRC qube feature flags did not exist. So, I see nothing wrong with wanting to package up a short script without forward-compatibility guarantees for R3.2 users.

[jpouellet]

Shoutout to @Joeviocoe since (although I have not reviewed/audited it) his script is probably a better starting point than mine.

[jpouellet]

Shoutout to @Joeviocoe since (although I have not reviewed/audited it) his script is probably a better starting point than mine.

[niccokunzmann]

@jpouellet Since I am new to the code, I would appreciate more links to the source and implementation you are writing about. I can I agree because I do not know what you are talking about but I am never sure even if I found code. Can you create links to these?

• #4028 (comment)

  • We already have infrastructure to handle reading/writing these rules

  • qubesdb - (probably a repository)

Answers:

how to handle the case where multiple VMs both have rules requesting the same port be forwarded to them?

I would expect an error with no change being done, if I request to forward a port which is already forwarded.

how to express mappings from one port to a different one (e.g. 80->8080)?

This is not an important use case for now in my opinion. If one desires to re-map, one can get a proxy. The basic use-case is to allow people to expose ports in general. Once there are more people who have port conflicts, they can join in creating this more advanced case. However, if we create database entries and such, we can already keep in mind that the incoming port is not the outgoing port.

[niccokunzmann]

@jpouellet Since I am new to the code, I would appreciate more links to the source and implementation you are writing about. I can I agree because I do not know what you are talking about but I am never sure even if I found code. Can you create links to these?

• #4028 (comment)

  • We already have infrastructure to handle reading/writing these rules

  • qubesdb - (probably a repository)

Answers:

how to handle the case where multiple VMs both have rules requesting the same port be forwarded to them?

I would expect an error with no change being done, if I request to forward a port which is already forwarded.

how to express mappings from one port to a different one (e.g. 80->8080)?

This is not an important use case for now in my opinion. If one desires to re-map, one can get a proxy. The basic use-case is to allow people to expose ports in general. Once there are more people who have port conflicts, they can join in creating this more advanced case. However, if we create database entries and such, we can already keep in mind that the incoming port is not the outgoing port.

[jpouellet]

for qubesdb, see:

• https://www.qubes-os.org/doc/vm-interface/
• https://github.com/QubesOS/qubes-core-qubesdb
• $ qubesdb-list /
• $ qubesdb-read /qubes-ip (as a random example)

It's basically a hierarchical key-value store with roughly equivalent semantics to xenstore. If you're not familiar with it, it's pretty simple. Only thing that may be non-obvious is that updates are passed as messages, so you can observe writes even if they are the same value as was there previously. This behavior can be used for generic notification of events without polling at some interval or relying on another out-of-band mechanism, for example to notify that some other set of non-atomic changes has been completed and state is now consistent and can be acted on, for example in the case of firewall rules spread over multiple key/value pairs in the store.

Other functionality is built on top of this, for example getting the firewall rules from entries under /qubesdb-firewall/${vm_ip}/... in R4 (or /qubes-iptables-domainrules/${domain} in R3, still present for backwards compat in R4 but deprecated) and being notified that the rules have been updated and that the vm should install handle them somehow (e.g. install them into the kernel via iptables) via watching for empty writes to /qubesdb-firewall/${vm_ip} (or write of reload to /qubesdb-iptables in R3).

---

as for infrastructure for handling firewall rules, it's nothing special. I'm just referring to the fact that we already have a qvm-firewall tool, rules accessible under qubesadmin.Qubes().domains[somevm].firewall.rules, stored in /var/lib/qubes/appvms/somevm/firewall.xml (for now, maybe to be merged into qubes.xml at some point), modifyable via the qubes-manager gui (albeit with some limitations), etc.

---

Useful entry-points for reading the relevant code would be:

• the agent (not-dom0) side, responsible for grabbing rules from qubesdb and putting them into iptables/nftables is at https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubesagent/firewall.py
• the core qubes handling of per-appvm firewall rule state involves many moving parts but is mostly here: https://github.com/QubesOS/qubes-core-admin/blob/master/qubes/firewall.py
• the dom0 (or theoretically other AdminVM) qvm-firewall command line tool is here: https://github.com/QubesOS/qubes-core-admin-client/blob/master/qubesadmin/tools/qvm_firewall.py
• the firewall tab of the Qubes manager gui is here: https://github.com/QubesOS/qubes-manager/blob/master/qubesmanager/firewall.py

Thanks for being interested :)

[jpouellet]

for qubesdb, see:

• https://www.qubes-os.org/doc/vm-interface/
• https://github.com/QubesOS/qubes-core-qubesdb
• $ qubesdb-list /
• $ qubesdb-read /qubes-ip (as a random example)

It's basically a hierarchical key-value store with roughly equivalent semantics to xenstore. If you're not familiar with it, it's pretty simple. Only thing that may be non-obvious is that updates are passed as messages, so you can observe writes even if they are the same value as was there previously. This behavior can be used for generic notification of events without polling at some interval or relying on another out-of-band mechanism, for example to notify that some other set of non-atomic changes has been completed and state is now consistent and can be acted on, for example in the case of firewall rules spread over multiple key/value pairs in the store.

Other functionality is built on top of this, for example getting the firewall rules from entries under /qubesdb-firewall/${vm_ip}/... in R4 (or /qubes-iptables-domainrules/${domain} in R3, still present for backwards compat in R4 but deprecated) and being notified that the rules have been updated and that the vm should install handle them somehow (e.g. install them into the kernel via iptables) via watching for empty writes to /qubesdb-firewall/${vm_ip} (or write of reload to /qubesdb-iptables in R3).

---

as for infrastructure for handling firewall rules, it's nothing special. I'm just referring to the fact that we already have a qvm-firewall tool, rules accessible under qubesadmin.Qubes().domains[somevm].firewall.rules, stored in /var/lib/qubes/appvms/somevm/firewall.xml (for now, maybe to be merged into qubes.xml at some point), modifyable via the qubes-manager gui (albeit with some limitations), etc.

---

Useful entry-points for reading the relevant code would be:

• the agent (not-dom0) side, responsible for grabbing rules from qubesdb and putting them into iptables/nftables is at https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubesagent/firewall.py
• the core qubes handling of per-appvm firewall rule state involves many moving parts but is mostly here: https://github.com/QubesOS/qubes-core-admin/blob/master/qubes/firewall.py
• the dom0 (or theoretically other AdminVM) qvm-firewall command line tool is here: https://github.com/QubesOS/qubes-core-admin-client/blob/master/qubesadmin/tools/qvm_firewall.py
• the firewall tab of the Qubes manager gui is here: https://github.com/QubesOS/qubes-manager/blob/master/qubesmanager/firewall.py

Thanks for being interested :)