[Bug] starting container with vpn takes a lot of time

[harmbuisman]

Describe the bug
Starting a container on a node that has vpn enabled takes about 3 minutes more than starting it on the same node with vpn disabled.

To Reproduce
Steps to reproduce the behavior:
With vpn_subnet in the node config, start the node, run a task. Use the portal UI to see start and finish time of the task.

Comment out the vpn_subnet line in the config. Repeat the above.
See that without the vpn configured the task takes about 3 minutes less.

[frankcorneliusmartin]

I observe the same behaviour. I started to add some debug logging to look more into the issue: https://github.com/vantage6/vantage6/tree/bugfix/slow-start-algorithm-with-vpn

It seems that the occupied_ports it taking long to compute:

2023-06-21 13:25:13 - task (2731)    - DEBUG    - Setup port forwarder
2023-06-21 13:25:13 - vpn_manager    - DEBUG    - Getting IP address of algorithm container
2023-06-21 13:25:13 - vpn_manager    - DEBUG    - Finding exposed ports of algorithm container
2023-06-21 13:25:13 - vpn_manager    - DEBUG    - Finding occupied ports... (using the VPN client container)
2023-06-21 13:25:43 - vpn_manager    - DEBUG    - VPN client container reported back
2023-06-21 13:25:43 - vpn_manager    - DEBUG    - Occupied ports: [49152, 49152, 49153, 49153, 49154, 49154, 49155, 49155, 49156, 49156, 49157, 49157]
2023-06-21 13:25:43 - vpn_manager    - DEBUG    - Assigning port 49158 to algorithm port
2023-06-21 13:25:43 - vpn_manager    - DEBUG    - Assigning port 49159 to algorithm port

I also observed that the first algorithm run (on a fresh restart) is rather quick. Every succeeding call seems to be slower. This call I made 2 iterations later (notice the time difference)

2023-06-21 13:27:49 - vpn_manager    - DEBUG    - Finding occupied ports... (using the VPN client container)
2023-06-21 13:28:39 - vpn_manager    - DEBUG    - VPN client container reported back
2023-06-21 13:28:39 - vpn_manager    - DEBUG    - Occupied ports: [49152, 49152, 49153, 49153, 49154, 49154, 49155, 49155, 49156, 49156, 49157, 49157, 49158, 49158, 49159, 49159, 49160, 49160, 49161, 49161]
2023-06-21 13:28:39 - vpn_manager    - DEBUG    - Assigning port 49162 to algorithm port
2023-06-21 13:28:39 - vpn_manager    - DEBUG    - Assigning port 49163 to algorithm port

And one itteration later:

2023-06-21 13:34:05 - vpn_manager    - DEBUG    - Finding occupied ports... (using the VPN client container)
2023-06-21 13:35:05 - vpn_manager    - DEBUG    - VPN client container reported back
2023-06-21 13:35:05 - vpn_manager    - DEBUG    - Occupied ports: [49152, 49152, 49153, 49153, 49154, 49154, 49155, 49155, 49156, 49156, 49157, 49157, 49158, 49158, 49159, 49159, 49160, 49160, 49161, 49161, 49162, 49162, 49163, 49163]

seems to be 10 secs per container:

2023-06-21 13:35:13 - vpn_manager    - DEBUG    - Finding occupied ports... (using the VPN client container)
2023-06-21 13:36:23 - vpn_manager    - DEBUG    - VPN client container reported back

[frankcorneliusmartin]

iptables can be a slow command, thus a possible semi-solution to randomize port selection and check if its free.

[frankcorneliusmartin]

From the iptables MAN

-L, --list [chain]
List all rules in the selected chain. If no chain is selected, all chains are listed. As every other iptables command, it applies to the specified table (filter is the default), so NAT rules get listed by
iptables -t nat -n -L
Please note that it is often used with the -n option, in order to avoid long reverse DNS lookups.