jool and jool_siit modules simultaneity

[edmont]

Tested with issue223 branch and root user.

If one of the modules is loaded after the other one, a userspace application would fail to execute.

root@pi:~# modprobe jool
root@pi:~# modprobe jool_siit
root@pi:~# jool --pool6
  (empty)
root@pi:~# jool_siit --pool6
Error: You're speaking to NAT64 Jool using the SIIT Jool application.
(Error code: 22)
root@pi:~# modprobe -r jool
root@pi:~# jool_siit --pool6
Could not send the request to Jool (is it really up?).
Netlink error message: Unspecific failure (Code -1)
root@pi:~# modprobe -r jool_siit
root@pi:~# modprobe jool_siit
root@pi:~# modprobe jool
root@pi:~# jool --pool6
Error: You're speaking to SIIT Jool using the NAT64 Jool application.
(Error code: 22)
root@pi:~# jool_siit --pool6
  (empty)

Is this known? How is it supposed to configure both modules simultaneously?

[ydahhrk]

It is certainly an error, although it's one I had been willing to stomach, to be honest.

What happens is nobody has come up with a reason why anyone would want to have a NAT64 and a SIIT in the same namespace.

So essentially, both translators register to the same Netlink protocol family (there are only 32 available, after all), and as you see, they catfight over the requests from userspace. In the end, the kernel is the one who decides who handles them.

It's bad design. I have developed the opinion that merely using Netlink is, in and of itself, bad design.

But the design was (I think) completely fixed during the 3.5 development. How did we fix it? From the root: We don't use Netlink anymore. But this is the kind of extreme refactoring I'd rather not do unless we're bracing for at least a "middle number" version upgrade (which is what is happening as we move on from 3.4.x to 3.5.x).

So I don't think this can be fixed in the 3.4 code, unless you manually compile a different Netlink protocol family depending on which translator you're building. Switching to the (soon to be) 3.5 code would be less messy.

But do you really have a reason to hold two translators in the same namespace? I'd like to hear the rationale, because Jool hasn't been coded considering such a use case so far.

Even in version 3.4, the translators will not interfere with each other as long as you modprobe them in different namespaces:

$ sudo modprobe jool
$ sudo ip netns add blue
$ sudo ip netns exec blue modprobe jool_siit
$ jool --pool6
  (empty)
$ sudo ip netns exec blue jool_siit --pool6
  (empty)

Jool 3.5:

$ sudo modprobe jool
$ sudo modprobe jool_siit
$ jool -6
  (empty)
$ jool_siit -6
  (empty)

[toreanderson]

But do you really have a reason to hold two translators in the same namespace? I'd like to hear the rationale, because Jool hasn't been coded considering such a use case so far.

I've been playing with the idea of integrating Jool into OpenStack Neutron in order to better support IPv6-only VMs. A virtual router in Neutron lives inside a private network namespace and provides routing between the tenant's virtual overlay networks and the Internet.

In IPv4, outbound access to the Internet is provided with standard IPTables SNAT. Inbound access from the Internet to selected virtual machines is handled with IPTables DNAT. This concept they call «Floating IP Addresses».

Extending this concept to IPv6 and Jool would mean using NAT64 for outbound access to the Internet, while using SIIT-DC for the inbound access (one EAM per Floating IP Address).

[edmont]

Sounds good.

One example where the two translators are in the same namespace: a host with three interfaces, one in IPv4 which is common for both translators, the other two in IPv6. And shared ip(6)tables rules.

Other example, maybe a bit weird: a normal SIIT translator used to access to the IPv6 part from the IPv4 local network, plus the stateful translator used by the IPv6 hosts to access IPv4 Internet (with a different pool6).

Anyway, I guess that can be achieved with namespaces, although I am not any master :)

[kpfleming]

There is a lot of work going on in the OpenStack community to move away from 'floating IP addresses' and NAT, so it might not be a good idea to replicate that model with IPv6.

[ydahhrk]

Extending this concept to IPv6 and Jool would mean using NAT64 for outbound access to the Internet, while using SIIT-DC for the inbound access (one EAM per Floating IP Address).

Hmmm... I'm not following. What's with the asymmetric translation? If v4 addres a is translated by NAT64 into v6 B, and in the opposite direction SIIT decides that the v4 equivalent of B is c, then I expect the packet to end up lost. If you need to go out of your way forcing the two translators to translate the same, what's stopping you from ditching one of them?

SIIT and NAT64 translate both source and destination, so I always thought of them as "S"NAT and "D"NAT combined.

(I guess I'm missing the point of a "floating" address; I don't see the difference between that and a normal NAT that "hides" public addresses instead of private ones.)

Other example, maybe a bit weird: a normal SIIT translator used to access to the IPv6 part from the IPv4 local network, plus the stateful translator used by the IPv6 hosts to access IPv4 Internet (with a different pool6).

I didn't understand your other example, but this sounds very reasonable. This is what I'm picturing here:

+---------------+   ::1 +--------+ 192.168.0.0/24 +----------+
| 2001:db8::/96 |-------| Router |----------------| Internet |
+---------------+       +--------+ .1          .8 +----------+
                         .1 |
                            |
                    +--------------+
                    | 192.0.2.0/24 |
                    +--------------+

It seems to work fine: (Tested in both 3.4 and 3.5)

sudo modprobe jool pool6=64:ff9b::/96
sudo modprobe jool_siit
sudo jool_siit -ea 192.0.2.0/24 2001:db8:4::/120
sudo jool_siit -ea 203.0.113.0/24 2001:db8:6::/120
sudo sysctl -w net.ipv4.conf.all.forwarding=1

Tests:

$ # Ping from the IPv6 network to the IPv4 network
$ ping6 2001:db8:4::8

$ # Ping from the IPv6 network to somebody on the Internet
$ ping6 64:ff9b::192.168.0.8

$ # Ping from the IPv4 network to the IPv6 network
$ ping 203.0.113.8

$ # Ping from the IPv4 network to somebody on the Internet
$ ping 192.168.0.8

It works fine because the SIIT address ranges and the NAT64 address ranges do not collide. Not sure why you suggested a pool6 in the SIIT though. It does, however, show another bug:

$ dmesg
SIIT Jool WARNING (pool6_find): The IPv6 pool is empty.

That warning is completely unnecessary.

There is a lot of work going on in the OpenStack community to move away from 'floating IP addresses' and NAT, so it might not be a good idea to replicate that model with IPv6.

What's the alternative? Any keywords I might look up?

[toreanderson]

Hmmm... I'm not following. What's with the asymmetric translation? If v4 addres a is translated by NAT64 into v6 B, and in the opposite direction SIIT decides that the v4 equivalent of B is c, then I expect the packet to end up lost. If you need to go out of your way forcing the two translators to translate the same, what's stopping you from ditching one of them?

You might have 100s of VMs and just a handful of IPv4 addresses. Thus you need NAT64 due to its N:1 quality, i.e., allowing every single one of the VMs to access IPv4-only resources on the Internet.

Further assume that a couple of your VMs might be running web servers and thus need to be accessed from IPv4-only clients on the Internet. Here you need 1:1 and thus SIIT.

This is the concept for IPv4:

iptables -t nat -I POSTROUTING -s 10.0.0.0/8 -j SNAT --to 192.0.2.1
iptables -t nat -I PREROUTING -d 192.0.2.2 -j DNAT --to 10.1.2.3

So here all the VMs in 10.0.0.0/8 can reach the Internet, sharing 192.0.2.1 as the external public adresss. At the same time the VM 10.1.2.3 can be accessed from Internet clients connecting to the external public address 192.0.2.2.

So I have assumed that in order to copy this concept to IPv6 and Jool I'd need NAT64 to replace the SNAT rule and SIIT-EAM for the DNAT rule.

[edmont]

It seems to work fine: (Tested in both 3.4 and 3.5)

This is how it goes for me:

# modprobe jool pool6=64:ff9b::/96
# modprobe jool_siit
# jool_siit -ea 192.0.2.0/24 2001:db8:4::/120
Error: You're speaking to NAT64 Jool using the SIIT Jool application.
(Error code: 22)

Not sure why you suggested a pool6 in the SIIT though.

What if the host on the Internet is 8.8.8.8? What I mean is something like:
jool_siit -ea 0.0.0.0/0 2001:db8:4::/96
Or maybe:
jool_siit -pool6 2001:db8:4::/96

It does, however, show another bug

That one made my syslog grow up to 3 GB already :)

[ydahhrk]

Further assume that a couple of your VMs might be running web servers and thus need to be accessed from IPv4-only clients on the Internet. Here you need 1:1 and thus SIIT.

Mayhap you're missing the point of static BIB entries? They are just web servers. One port.

sudo jool --bib --add --tcp 2001:db8::1#80 192.0.2.2#80

"The webserver at [2001:db8::1]:80 should be available from outside as 192.0.2.2:80."

Error: You're speaking to NAT64 Jool using the SIIT Jool application.
(Error code: 22)

Try it like this then:

sudo modprobe jool_siit
sudo jool_siit -ea 192.0.2.0/24 2001:db8:4::/120
sudo jool_siit -ea 203.0.113.0/24 2001:db8:6::/120
sudo modprobe jool pool6=64:ff9b::/96
sudo sysctl -w net.ipv4.conf.all.forwarding=1

What if the host on the Internet is 8.8.8.8? What I mean is something like:

You don't need any of that, as far as I can tell.

$ # From the IPv6 network
$ ping6 64:ff9b::8.8.8.8

$ # From the IPv4 network
$ ping 8.8.8.8

That one made my syslog grow up to 3 GB already :)

Then it's a rather critical bug!

Update: https://github.com/NICMx/Jool/tree/issue224

---

I'm starting to realize the documentation does not seem to be very clear. Any suggestions?

By the way: Our guests network discovered a kernel panic in the 3.5 code. Will report and fix through the course of the day.

[toreanderson]

Mayhap you're missing the point of static BIB entries? They are just web servers. One port.

The «Floating IP» address concept in OpenStack is a layer-3 DNAT. That is, all protcols, all ports. Using static BIB entries to achieve that would be extremely impractical. And even if you did go ahead and preconfigure all the hundreds of thousands of possible static BIB entries, non-TCP/UDP protocols would still not work. And for TCP and UDP I think you'd still end up keeping state for no reason at all (i.e., session table entries for the IPv4-client initiated connections).

[edmont]

Try it like this then:

# modprobe jool_siit
# jool_siit -ea 192.0.2.0/24 2001:db8:4::/120
# modprobe jool pool6=64:ff9b::/96

Well, that doesn't bark, but any other subsequent command...

# jool --pool6
Error: You're speaking to SIIT Jool using the NAT64 Jool application.
(Error code: 22)

[ydahhrk]

The «Floating IP» address concept in OpenStack is a layer-3 DNAT. That is, all protcols, all ports. Using static BIB entries to achieve that would be extremely impractical. And even if you did go ahead and preconfigure all the hundreds of thousands of possible static BIB entries, non-TCP/UDP protocols would still not work. And for TCP and UDP I think you'd still end up keeping state for no reason at all (i.e., session table entries for the IPv4-client initiated connections).

Thanks, I get it now.

Well, that doesn't bark, but any other subsequent command...

I know; the point was to work around it by finishing configuring one module before moving on to the other one.

Ok so the point is that there are perfectly valid reasons to hold a NAT64 and a SIIT in the same namespace simultaneously. Unfortunately, Netlink is not going anywhere in Jool 3.4 and there is no way around its petty family pool.

So I'm letting you guys decide which Netlink family the sockets should be bound to.

This is the jist of it:

sudo modprobe jool_siit nl_family=30
sudo jool_siit --nl-family 30 -ea 192.0.2.0/24 2001:db8:4::/120
sudo jool_siit --nl-family 30 -ea 203.0.113.0/24 2001:db8:6::/120

sudo modprobe jool pool6=64:ff9b::/96 nl_family=31
sudo jool --nl-family 31 ...

sudo sysctl -w net.ipv4.conf.all.forwarding=1

All userspace commands need to match the netlink family its respective module was modprobed with.

The Netlink family needs to be a number between 0 and 31. You shold not use a number that has been already assigned.

• Here's a list of taken numbers in kernel 4.6: http://lxr.free-electrons.com/source/include/uapi/linux/netlink.h?v=4.6#L8
• Here's a list of taken numbers in kernel 4.5: http://lxr.free-electrons.com/source/include/uapi/linux/netlink.h?v=4.5#L8
• Here's a list of taken numbers in kernel 4.4: http://lxr.free-electrons.com/source/include/uapi/linux/netlink.h?v=4.4#L8

And so on and so on.

Again, all of this will be left behind by Jool 3.5 since it doesn't need it.

[mkg20001]

Is there any example of using the changes in a8d6a3f ?

[ydahhrk]

@mkg20001 No, brah.

That code is so old, it doesn't even compile in current kernels. But more importantly, Jool doesn't use Netlink directly anymore; that flag doesn't make sense.

Why do you want it? If you're trying to implement your own kernel-to-userspace channel, use Generic Netlink instead. No need to choose a Netlink family then.