Static IPs for jails

[spusuf]

The current documentation for the advanced network setup is a little hard to understand. I will be pointing my reverse proxy and port forwards to services in these jails as soon as I can figure out how to assign a unique static IP or NAT to each jail.

[Jip-Hop]

Why do you need static IPs for the jails? Can't you use the default host networking? A service inside the jail can bind to a port on the host IP address, e.g. 8080, which you can then point your reverse proxy to.

If you must it is possible to give the jail a static IP following the steps in the advanced networking setup. I hope you succeed if you try :)

[spusuf]

A reason why is for example NGINX as a reverse proxy. I want my router to port forward multiple ports (including 80, 443, 8123, etc) to the NGINX jail's IP with those ports intact (80-80, 443-443, 8123-8123). Not translating from port 80 to whatever port the service is running.

[Jip-Hop]

I don't know why you must keep the same port numbers when doing port forwarding. But if that's a requirement, and you want to stick with the default (host) networking for the jail, you could move the TrueNAS SCALE web gui away from port 80 and 443 to e.g. 8080 and 8443. That would free up port 80 and 443 for your nginx server inside the jail.

Just for fun you could then use the nginx reverse proxy to still access your TrueNAS web gui through port 80/443.

[Jip-Hop]

Then one last thing you can do before going into the 'advanced networking setup' is to try this suggestion: https://github.com/Jip-Hop/jailmaker#networking

If you add and additional alias (IP addresses) to the main networking interfere of your TrueNAS SCALE server, then that one will have all the ports free and you can bind to that IP addresses from inside your jail. No need yet for individual static IPs per jail.

[spusuf]

Just to add to this, there are many services that run on the IP address of the host. So for these apps you would connect to the IP of the jail, for the web portal, but if thats the same as the truenas interface then the app fails to run.

[chocology]

I dont know if this would help any of you looking to run services without changing for example port 80 or 443, I followed this youtube guide and as a result, I am able to create a bridge and run numerous containers with Jailmaker on my TNS without changing any ports on TNS.

I feel is we can incorporate this into the guidelines of Jailmaker, it would extend the abilities quite a lot. https://www.youtube.com/watch?v=7clQw132w58

[spusuf]

I have switched to using macvlan and it performs flawlessly and basically the same as TrueNAS Core's jails. It's WAY easier than setting up bridges. Check my answer.

[Lockszmith-GH]

I'm sorry @spusuf, while I agree with you about the 'easier to setup', as all it requires is a copy-paste to the config, it is far from 'flawless'.
The Jail cannot be accessed by the host.

If what you run is only outwards facing - you are right, no concern, and that's why you can describe it as 'performs flawlessly'.

But if you want to run a reverse proxy in the Jail, and you would want it to serve the TrueNAS front end, or vice-versa, if you would like a reverse proxy installed as a TrueNAS SCALE app to serve a service in a Jail, macvlan isn't the choice.

On the other hand, setting up a bridge, not only allows that type of communication, it also allows communicating with a VM.

It makes the system more versatile.

So 'easier', does not automatically means 'better'.

Don't get me wrong, bridges aren't perfect either, I just feel they are easier to manage.

[Lockszmith-GH]

I see your thread below, and no, the static IP does not work to communicate with the host - at least not without extra routing definitions.

[spusuf]

The process of creating, managing, and troubleshooting a bridge is much more time intensive than one line config by two or three orders of magnitude. The only use cases I've heard for it is:

• Serve the TrueNAS frontend through a nested reverse proxy (this is discouraged by TrueNAS, and could still be achieved using a port forward and a certificate installed to the truenas UI)
• Host a DHCP server inside the jail (is that necessary AND a common use case?)
• Home Assistant running in a jail wont be able to monitor temps of the host (there are http api calls which should be used)

[Lockszmith-GH]

any communication with the host will fail. So if for example you are running prometheious or loki inside a jail, it can't monitor the host.

Yes, DNS and DHCP are part of it, but I would agree those might be a bit nieche, as other solutions might already be deployed.

But, runing Wireguard inside the jail to access the local network, while blocking the TrueNAS WebUI ports from the world - a safer solution then just a reverse proxy - not possible either.

Take a look at Guacamole or Remotely, those are great in a jail, as long as you can actually access everything from within them.

I agree that it really depend on what the jail is doing, but it's more common than you think to need that communication.

[spusuf]

Here's how to run Static IPs for jails/containers WITHOUT using bridges. This is identical to the behavior of TrueNAS Core's jails when a Static IP is set.

Add --network-macvlan=eno1 --resolv-conf=bind-host to user flags either during creation or after by editing the config.
Run nano /etc/systemd/network/mv-dhcp.network (you will need to install the nano editor, or you can use a remote file browser like WinSCP to edit the file manually).
Change the contents to:

[Match]
Virtualization=container
Name=mv-*

[Network]
DHCP=false
Address=192.168.X.X/24 
Gateway=192.168.X.X

Remembering to replace the X's with your desired IP and Gateway's (Router) IP. For an example it was Address=192.168.1.10 Gateway=192.168.1.1

Thats it. No bridges, or anything. Utilises the TrueNAS network interface to designates a Static IP (even when conflicting, yes will stop you connecting from TrueNAS if conflicting)

[ghost]

Is the procedure for using an alias similar? I.e setting truenas scale to have two alias IPs (say 192.168.1.10 and 192.168.1.11 and binding the jail to the second one(192.168.1.11)?

[spusuf]

Much harder to set up an alias and bind IMO. MAC VLAN is pretty simple and I've never had to troubleshoot something not working, but I have when using the dual interfaces in TrueNAS.
Test it for yourself, but the procedure I've mentioned is functionally identical to how jails work in TrueNAS Core. Setting two IPs to one adapter seems like more of a hack than a proper implementation of a networking stack, but that's just my opinion.

[Jip-Hop]

Downside of the macvlan network is that by default the jail can't reach the TrueNAS host and vice versa. So if you for some reason need to do this, a bridge would still be better. Never the less macvlan is a nice option. Glad to hear you worked it out. Feel free to improve the advanced networking documentary.

[spusuf]

Downside of the macvlan network is that by default the jail can't reach the TrueNAS host and vice versa.

Can you provide any reason this would be needed? The whole point of containers is running services that can't touch the host system. And TrueNAS recommends not installing ANYTHING AT ALL to the Linux OS underneath it. The only reason nspawn containers are viable is because they can't corrupt the TrueNAS install.

[Jip-Hop]

"Anything that is available to the public internet?" The hypothetical DNS server I mentioned is LAN accessible only...

[Jip-Hop]

I haven't tried but I'd expect using macvlan would also prevent communication between the jail and Apps hosted on TrueNAS. So if you want to use them side by side and connect them together I think you'd need the bridge. Haven't tried this myself though as I don't have Apps enabled.

[spusuf]

Ah sorry, I assumed everyone is like me and runs their DNS on their router to prevent unnecessary traffic.

Also macvlan would not prevent communication with apps, you simply use the static IP instead of 127.0.0.1 (localhost). This has been tested and works.

[Jip-Hop]

Nice! Then macvlan should work well in most cases :)

[spusuf]

It would be nice to have an option in the jail install script to switch between dhcp and macvlan as needed. Instead of googling the github page, going to advanced networking and copying the macvlan config arguments.

[mrstux]

Been testing this a bit today.

I had assumed that macvlan wouldn't work if the primary interface was involved in a bridge. This is not true.

As I see it, the only reason to use network-bridge is if you want the jail to have connectivity to the host. Network connectivity is just fine to the LAN (and WAN).

You can use your router to assign fixed IP via DHCP to the jail, as systemd containers use a hash of the container name for the MAC address, thus as long as your jail's name doesn't change, the MAC won't change.

If you don't use a router as your DHCP server, then you will want to use a fixed IP address.

I really don't see much cause to use host networking in a systemd jail... unless you are wanting to add services to the host per se, many of us are wanting to add services to our network, and it doesn't need to be on the same IP as the NAS (host).

So, the only reason to use bridge mode networking in the jail is if if you NEED to access the NAS/Host IP... for SMB, or iSCSI, or some other reason... say SSH, but that'd be a security hole.

I can't think of a good reason, when you already have bind mounts.

[Jip-Hop]

So you recommend to update the docs? Feel free! Yes I think using macvlan is appropriate in most cases.

[spusuf]

Correct, MACVLAN is sufficient for services which need a dedicated IP, and the default pass through is sufficient for ports under the same IP as host. That's what I've recommended and explained in my version of the docs (located in the docs folder)., but I havent updated them in a while due to switching to vanilla Debian instead of truenas so I cannot guarantee they are 100% up to date.

ONLY reason for bridge is container to host networking which still seems niche.

[mrstux]

did some more testing.

I think if you already have a bridge setup on the external interface, ie so that VMs can contact the host (which is necessary in order to access storage in the VM), then macvlan will not work, and you must use the bridge method.

edit: You would see this error in the journal when attempting to use an interface for macvlan which is already a member of a bridge

Apr 25 19:29:40 truenas .ExecStartPre[2425404]: PRE_START_HOOK
Apr 25 19:29:40 truenas systemd-nspawn[2425406]: Failed to add new macvlan interfaces: Device or resource busy
Apr 25 19:29:40 truenas systemd-nspawn[2425408]: Parent died too early
Apr 25 19:29:40 truenas systemd[1]: jlmkr-docker.service: Main process exited, code=exited, status=1/FAILURE

[mrstux]

I demonstrate setting up Static IPs for both MAC-VLAN and bridge networking in this video

https://youtu.be/S0nTRvAHAP8?t=313