[Bug]: root-controller always points to port 30006

[emmuhamm]

Description

When booting up a session, it is possible to accidentally connect to someone else's session. This is because the root controller always points to port 30006, so if theres a stray session somewhere it will connect to someone else's session instead.

It is unclear why the port number always resolves to 30006.

Steps to reproduce

1. On a new shell, set up the nightly NFD_DEV_251117_A9
2. Clone the drunc repository (on develop, fee48f2 as of the time of reporting)
3. Open a drunc session (eg drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-1x1-config emir-test)
4. Run boot. The drunc session may need to be exited and reinitialised a few times before the issue can be observed.

It is more likely to observe this when there is a stray controller session elsewhere

Version and environment

NFD_DEV_251117_A9

Impact

Small/Isolated

Anything else?

No response

[wanyunSu]

The root-controller port can not be 0 for k8s as it's used as a NodePort service (port range 30000–32767).

The port for root-controller is given in the configuration : https://github.com/DUNE-DAQ/daqsystemtest/blob/develop/config/daqsystemtest/ccm.data.xml#L174

We need to have a better way to assign an available port for root-controller.

[PawelPlesniak]

Could we dynamically assign something in that range? Otherwise, we should add a check to ensure that the port you are intending to use is free, and if not then a pointer to the tool that is needed to change the port would be very useful

[bieryAtFnal]

Michal, with a little help from me, wrote a script in the daqconf repo to set_rc_controller_port. It calls out to a Python function that is defined in daqconf/python/daqconf.

I mention that users should run this script in Step 4 on the software-instruction-page (here) to help them avoid port clashes like the one that I think is being discussed in this Issue.

The creation of this script was part of DUNE-DAQ/daqconf#608.

The python script is called from the integrationtest infrastructure as mentioned in DUNE-DAQ/integrationtest#131.

I mention this because the daqconf/scripts/daqconf_set_rc_controller_port script and/or the daqconf/python/daqconf/set_rc_controller_port.py function may be useful as part of addressing this Issue.

[bieryAtFnal]

If the script is not given a new port value, it randomly assigns a value in the allowed K8s range:

daqconf_set_rc_controller_port local-1x1-config config/daqsystemtest/example-configs.data.xml --help
Usage: daqconf_set_rc_controller_port [OPTIONS] OKS_SESSION_NAME OKSFILE
                                      [RC_PORT]

  Script to set the value of the RC Controller Service port used by the
  specified Session in the specified OKS database file. If the new port is not
  specified, it is set to a  random available k8s NodePort.

Options:
  --help  Show this message and exit.

[PawelPlesniak]

Thank you for pointing us to the right direction here Kurt, Michal had told us about this script but not pointed it to us. The associated PR should run a check that validates whether or not the current configuration's port is in use on the given host, and if it is, then throw an error and exit the shell, making it clear what the issue is and telling the user to use daqconf_set_rc_controller_port.

[PawelPlesniak]

The formal solution for this agreed upon in the RC technical meeting on 11.12 is to allow the RC to change the configuration file on boot calls, adjusting the configuration value for the used port number so it will be correctly published to the run registry. The solution provided in #741 is temporary.
Note - on further consideration, this will require confirmation. E.g. we currently use CVMFS to distribute the configuration files in the instance that we are not using a local copy, which is a read only file system. As such we will need to decide whether it is reuqired to have this value published in the configuration file, or if we are OK with it being published incorrectly.