Skip to content

Home

Jump to bottom
Ryan Cox edited this page Mar 19, 2025 · 7 revisions

spank_iso_netns - Create an Isolated Network Namespace for Slurm Jobs

Overview

spank_iso_netns.c is a SPANK plugin that provides network namespace isolation for Slurm jobs. This plugin creates a separate network namespace for a job, enabling isolated network environments with custom configurations while maintaining controlled communication channels with the outside world.

Installation

Compile with:

$ gcc -I/usr/local/src/slurm -fPIC -shared -o spank_iso_netns.so spank_iso_netns.c

Install the .so file in your Slurm library directory (or anywhere else, if you want) then point to it in Slurm's plugstack.conf:

optional /path/to/your/slurm/spank_iso_netns.so setup_script=/path/to/your/spank_iso_netns/setup_script

Install the setup script and modify it if needed. It includes some example code for a veth pair that is commented out since it is not well tested and may not be desirable.

Other things to install:

  • node.js (needed for the included HTTP proxy, if you choose to use that)
  • socat >= 1.8.0 (needed for outbound "proxies" other than the HTTP proxy)
    • If you have RHEL 9.4 or Debian 12 or earlier, socat is not new enough. You must install a version with ACCEPT-FD support. Anything in 1.8.x or newer should do.

How to Use From sbatch

Simple Example

To enable the plugin with 3 TCP listeners:

$ sbatch --iso-netns-listeners=3 myjob.sh

Proxy Configuration (Optional)

Setting up specific proxy addresses. This is a one-to-one mapping of source to destination IP:port (i.e. access 192.168.1.1:8080 by contacting 192.168.1.1:8080).

$ sbatch --iso-netns-listeners=3 --iso-netns-proxies="tcp@192.168.1.1:8080,tcp@10.0.0.1:443" myjob.sh

A simple HTTP proxy is available using:

sbatch --iso-netns-listeners=3 --iso-netns-proxies="127.0.0.1:8080=httpproxy,1.2.3.4:443" --iso-netns-http-proxy-conf=/path/to/spank_iso_netns/proxy.conf.d/test-rc.conf,/path/to/spank_iso_netns/proxy.conf.d/test2-star.conf

Specify an IP and port to listen on inside the namespace (e.g. 127.0.0.1:8080), specify to use the httpproxy, and specify where to pull the proxy configuration from with --iso-netns-http-proxy-conf. It can be comma-separated if you need to combine multiple configs. Each config allows connections to the destination using * as a wildcard. Place one destination per line. If you want example.com to be allowed, specify that as a line. *.example.com would allow subdomains as well. * opens everything up.

An example file is:

*.example.com
host.example.net
*.something.example.net

The HTTP proxy is not well-tested at the moment.

Additional Parameters (Optional)

--iso-netns-options can be used to pass optional information to the setup script or through to the job. The plugin does not use this information and instead just passes it through.

Example:

$ sbatch --iso-netns-listeners=3 --iso-netns-options="I like 🥓,super_not_secret_option,What is this for❓"

Output Environment Variables

The following environment variables are set.

SPANK_ISO_NETNS_LISTENING_FD_0
SPANK_ISO_NETNS_LISTENING_PORT_0
SPANK_ISO_NETNS_LISTENING_PROTO_0

0 in this example is for the first listener. Listeners are numbered sequentially starting from 0.

These are the file descriptor number, port, and protocol for that listener.

How It Works

Network Namespace Isolation

The core functionality revolves around Linux network namespaces. Network namespaces provide isolation of the network stack, allowing each namespace to have its own:

  • Network interfaces
  • Routing tables
  • Firewall rules
  • Socket ports (e.g. TCP 445)

When a Slurm job starts with this plugin enabled, the job gets its own network namespace, effectively isolating it from the host system's network.

Inside of the network namespace, you end up with complete isolation. Here is an example using 7lbd where only three ports (and some qemu UDP ports) are open in the namespace:

# ip addr
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
# ip route
# nft list ruleset
# ss -nltp
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port    Process                                                                         
LISTEN    0         50               127.0.0.1:445              0.0.0.0:*        users:(("smbd",pid=2677469,fd=28))
LISTEN    0         5                  0.0.0.0:4822             0.0.0.0:*        users:(("guacd",pid=2677962,fd=4))                                    
LISTEN    0         1                  0.0.0.0:3389             0.0.0.0:*        users:(("qemu-kvm",pid=2677917,fd=16))
#

These three ports in the example are inaccessible outside of the namespace. The ports cannot be accessed even by the same user on the same host unless the user is somehow placed inside the network namespace. That means that the user who submitted the job has no way of accessing the ports from the three processes listed above unless there is something to proxy those connections.

Listeners (For Inbound Connections)

Let's say you want a port inside the namespace to be made available outside of the namespace. There are several approaches that could have been chosen, such as creating a veth NIC pair with one NIC inside and one NIC outside the namespace, then routing between the two. That has more complexity but is certainly a possibility. However, I wanted something that didn't involve messing with routing or firewall rulesets.

The approach that I chose is to use what I call "listeners". In the sbatch parameters, the user can specify how many listeners they need. The plugin opens ports on the outside of the job then creates a new network namespace. This is the magic. The port exists in the outer namespace, but the file descriptor that refers to it is now in the new namespace. The plugin never closes the file descriptor so that it's still open when the batch job runs. File descriptors are preserved at fork and exec time unless you take steps to prevent it or close them. Therefore, when your target program runs, it has access to the same file descriptor that was opened outside the namespace. Your program will know what file descriptor to use since the plugin sets environment variables such as:

SPANK_ISO_NETNS_LISTENING_FD_0=13
SPANK_ISO_NETNS_LISTENING_PORT_0=44857
SPANK_ISO_NETNS_LISTENING_PROTO_0=tcp

0 is the number of the listener in this example. In this example, you have file descriptor number 13, with TCP port 44857. If you have multiple programs needing listeners, you can choose which of the listeners you want to assign to what program. It was assumed that you'll want a random port since there are presumably many jobs running and you don't want conflicts. The plugin binds to port 0 and lets the kernel choose the port.

Now, how do you use this file descriptor in your program? It depends! Quite a few programs take a file descriptor number (13 in the example above) in lieu of an IP and port to listen on. Key terms to search for are "fd", "file descriptor", "inetd", "listen", "systemd socket activation", "sd_listen_fds", and others (though you have to watch out for the difference between programs that are expecting an already accept()ed socket or one that is only listen()ing). Sometimes there are undocumented features that you have to read the source code for. For example, stunnel supports sd_listen_fds() but it's completely undocumented at the time of this writing

UNRELATED: To save you some trouble should you ever use sd_listen_fds(), just know that bash's exec forks before exec. That means LISTEN_PID won't be correct when you call your target program. A workaround is something like this example: exec perl -e '$ENV{LISTEN_FDS}=1; $ENV{LISTEN_PID}=$$; exec("stunnel", $ENV{stunnelconf_path}). You can set up fds with exec 3>&-; exec 3>&"$SPANK_ISO_NETNS_LISTENING_FD_0"-.

If your program supports it, great! Otherwise you may need an intermediary like socat or others.

To clarify how this works, look at the following pseudo-code that is typical of TCP connections:

socket()
bind()
listen()
while (accept()) {
  ...do stuff...
}

First you bind to a port, then you listen on it, then you accept connections and do stuff with them. A typical program is starting from scratch and needs to do all of these steps. spank_iso_netns only does:

socket()
bind()
listen()

Your program can then pick up where spank_iso_netns left off by using the same file descriptor and doing the remainder:

while (accept()) {
  ...do stuff...
}

You just need to figure out how to pass the file descriptor number to your program. If your program doesn't support it, you need to investigate alternative approaches such as proxying with socat, stunnel (for encrypted connections), a web proxy, or something else.

Proxies (For Outbound Connections)

Isolation is great and all, but you may need external connections for license servers or other resources. veth NIC pairs would certainly be an option and can be done using the setup script. However, the preferred way is to use the --iso-netns-proxies option. For example, --iso-netns-proxies="tcp@192.168.1.1:8080,tcp@10.0.0.1:443"

What this does may be confusing, but it's just the opposite direction of the listeners. The plugin opens a port in the inner namespace then reenters the outer namespace to launch socat using that file descriptor. Now, socat is on the outside and can make requests to the specified IP address while still listening on the inside.

socat only makes connections to a specific IP and port. What's fun is that socat is also listening on the target IP and port using that file descriptor that was opened in the inner namespace.

When you need to talk to 10.0.0.1:443 from inside the namespace, simply talk to 10.0.0.1:443. That's where socat's listening file descriptor is listening. When socat proxies the connection, it then talks out from outside that namespace to 10.0.0.1:443. This makes it so you don't have to do any mapping. This trick works because of ip route add local 0.0.0.0/0 dev lo being run in the setup script so that the kernel responds on all IPv4 IP addresses.

It's also worth noting that we run sysctl -w net.ipv4.ip_unprivileged_port_start=0 inside the inner namespace so that (ONLY in the namespace) unprivileged processes can listen to any port. That also helps for programs like smbd so that they can listen on well-known ports.

HTTP Proxy (For Outbound Connections)

In addition to the proxy approach above, there is also a simple HTTP proxy. Tinyproxy seems like a good proxy for this purpose, but it does not support passing in a file descriptor at the time of this writing. It would probably be relatively easy to add this support, but an LLM was quicker. The problem with proxies is that there are several good but complicated ones that would work, and there are thousands of simple ones online. Finding a decent one that also supported listening on an existing file descriptor exhausted my relatively long attention span. An LLM was easy.

The included node.js HTTP proxy was 100% written with an LLM. It is also not well-tested at the moment. Buyer beware.

Setup Script

The plugin runs a script you specify in plugstack.conf. The plugin calls the script from various points and passes one of these parameters as the only argument to the script:

  • pre-ns-creation
    • Runs just prior to namespace creation
  • post-ns-creation-outer
    • Runs just after namespace creation in the original namespace
  • post-ns-creation
    • Runs just after post-ns-creation-outer in the new namespace
  • task-exit
    • Run any cleanup tasks

post_ns_creation_outer sets up any proxies. To reiterate, these are proxying from the inner namespace to the outside.

post-ns-creation sets up the new namespace by setting lo up with an IP. sets sysctl parameters, etc.

Security

The plugin handles privilege management by:

  1. Running setup script with appropriate privileges
  2. Dropping privileges when executing user-context tasks
  3. Managing capabilities to ensure isolation

Testing

To test if it's working, submit a job to request a certain number of listeners. Inside the job, run ip addr, ip route, etc. ip addr should only show interface lo and ip route should be empty. You can also run the following to test the listeners.

Inside the job:

computenode$ echo $SPANK_ISO_NETNS_LISTENING_PORT_0 
48781
computenode$ socat ACCEPT-FD:$SPANK_ISO_NETNS_LISTENING_FD_0 STDIO

Outside the job (different host, same host, or wherever):

loginnode$ nc computenode 48781

Once both sessions are ready, type hello into one of the sessions. If it's working, hello should show up on the other end. If socat complains about ACCEPT-FD not being a valid parameter, you need a newer socat.

Clone this wiki locally