Controller restore procedure gets stuck

[nrauso]

The restore procedure from backup for the nethsecurity-controller app cannot be completed.
The process gets stuck indefinitely during the database import step restore-module/30restore_database

The restore action repeatedly expects the timescale container to exist and be running, but at this stage no container is available.

Relevant logs:

May 06 10:15:26 rl1 agent@nethsecurity-controller1[41205]: task/module/nethsecurity-controller1/8b6433e2-5e9f-4822-b196-f879e6ea93bb: restore-module/30restore_database is starting
May 06 10:15:26 rl1 agent@nethsecurity-controller1[41205]: Error: no container with name or ID "timescale" found: no such container
May 06 10:15:31 rl1 agent@nethsecurity-controller1[41205]: Error: no container with name or ID "timescale" found: no such container
May 06 10:15:36 rl1 agent@nethsecurity-controller1[41205]: Error: no container with name or ID "timescale" found: no such container
May 06 10:15:41 rl1 agent@nethsecurity-controller1[41205]: Error: no container with name or ID "timescale" found: no such container
May 06 10:15:46 rl1 agent@nethsecurity-controller1[41205]: Error: no container with name or ID "timescale" found: no such container

At this point, the timescale container is not started:

~]# runagent -m nethsecurity-controller1 podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

The restore action remains indefinitely blocked:

~]# runagent -m nethsecurity-controller1 ps xf
    PID TTY      STAT   TIME COMMAND
...
  41205 ?        Ssl    0:00  \_ /usr/local/bin/agent --agentid=module/nethsecurity-controller1 --actionsdir=/usr/local/agent/actions --actionsdir=/home/ne
  42839 ?        S      0:00  |   \_ /bin/sh /home/nethsecurity-controller1/.config/actions/restore-module/30restore_database
  43278 ?        S      0:00  |       \_ sleep 5

A temporary manual workaround is possible, but it requires several manual steps:

• Kill the blocked 30restore_database action
• Temporarily modify the timescale systemd unit to avoid starting its dependencies and start it manually
• Re-run the 30restore_database action manually
• Stop timescale and restore the default 'timescale` systemd unit
• Run configure-module to put the restored controller into production

This workaround allows the restore to complete, but it should not be required during a normal restore procedure.

Components

• nethsecurity-controller:2.2.3

[gsanchietti]

The bugs is composed by 3 different parts:

• a leftover of a901e690dea9869c921c0a216ada65e2fd3f76e1: was causing a loop inside the configure-module preventing the start of the database
  Relevant log for failed restore:

  May 07 14:03:29 rl1 agent@nethsecurity-controller6[646995]: Traceback (most recent call last):
  May 07 14:03:29 rl1 agent@nethsecurity-controller6[646995]:   File "/home/nethsecurity-controller6/.config/actions/configure-module/20configure", line 93, in <module>
  May 07 14:03:29 rl1 agent@nethsecurity-controller6[646995]:     'lets_encrypt': request["lets_encrypt"],
  May 07 14:03:29 rl1 agent@nethsecurity-controller6[646995]:                     ~~~~~~~^^^^^^^^^^^^^^^^
  May 07 14:03:29 rl1 agent@nethsecurity-controller6[646995]: KeyError: 'lets_encrypt'
  

• the same regression was preventing the restore of the application inside a machine different from the original one
  Relevant log for fariled restore inside another machine:

  May 07 14:13:47 rl1 agent@traefik1[611547]: 2026-05-07T14:13:45Z unable to generate a certificate for the domains [nethcontroller.nethesis.it]: error: one or more domains had a problem:
  May 07 14:13:47 rl1 agent@traefik1[611547]: [nethcontroller.nethesis.it] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:tls :: 188.166.63.28: remote error: tls: unrecognized name
  

• the Timescale database was updated during time, but the extension was not upgraded inside the database schema: this was preventing the restore of backup done on a machine installed with an old controller version and then updated to the last one.
  When restoring the dump, there was a conflict between Timescale extension version that was causing a failure during database dump import.
  This issue can be reproduced by installing and old version and updating it to the last one:

  # timescale 2.16.1
  add-module ghcr.io/nethserver/nethsecurity-controller:1.1.0
  # timescale 2.21.1
  api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.1.0","instances":["nethsecurity-controller1"]}'
  # timescale 2.22.1
  api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.1.2","instances":["nethsecurity-controller1"]}'
  # timescale 2.31.1
  api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.1.3","instances":["nethsecurity-controller1"]}'
  # same timescale, latest controller stable release
  api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.2.4","instances":["nethsecurity-controller1"]}'
  # Execute the backup
  

[gsanchietti]

Testing image: ghcr.io/nethserver/nethsecurity-controller:2.2.5-dev.1

Test case A — New installation -> backup & restore

1. Prepare fresh controller: add-module ghcr.io/nethserver/nethsecurity-controller:2.2.5-dev.1
2. Configure the instance: make sure to enable Let's Encrypt toggle
3. Connect one unit so DB contains registration records, wait for some data to be sent to the Controller
4. Configure and execute the backup from Backup and Restore page of cluster-admin
5. Restore to a new/clean instance using the Backup and Restore page of cluster-admin
6. The backup should be restored, check the Network DPI metrics inside Grafana

Test case B — Old installation (multi-step update) -> backup & restore
Goal: reproduce older/backups created by older releases and ensure restore finishes after extensions/format fixes.

1. Create an “old” controller (source) and update it to latest stable:

# timescale 2.16.1
add-module ghcr.io/nethserver/nethsecurity-controller:1.1.0
# timescale 2.21.1
api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.1.0","instances":["nethsecurity-controller1"]}'
# timescale 2.22.1
api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.1.2","instances":["nethsecurity-controller1"]}'
# timescale 2.31.1
api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.1.3","instances":["nethsecurity-controller1"]}'
# same timescale, latest controller stable release
api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/nethsecurity-controller:2.2.4","instances":["nethsecurity-controller1"]}'

2. Configure and execute the backup from Backup and Restore page of cluster-admin
3. Restore to a new/clean instance using the Backup and Restore page of cluster-admin
4. The backup should be restored, check the Network DPI metrics inside Grafana

[nrauso]

All test cases VERIFIED
Nice job @gsanchietti!