have netbox-restore script restart necessary services or set necessary permissions

[mmguero]

The documentation says that:

Note that some of the data in the NetBox database is cryptographically signed with the value of the SECRET_KEY environment variable in the ./netbox/env/netbox-secret.env environment file. A restored NetBox backup will not work if this value is different from when it was created.

From what I can tell this isn't actually true: the data itself isn't encrypted, just some of the session handle information or cookies or whatever. A GitHub thread (also this one) I found seems to indicate this as well.

However, it does appear that some services need to be restarted for this to take effect, either Logstash or NetBox or both, to make sure that logstash logs back in successfully, otherwise it appears you end up with a 403 error from Logstash.

So this bug is to:

1. Determine exactly what needs to be restarted after the netbox-restore command to make sure logstash "logs in" again with the token successfully
2. Make the netbox-restore script restarts what it needs to restart
3. Fix the documentation to not say that about the SECRET_KEY

Here's the netbox reference

[mmguero]

It might not be an issue with SECRET_KEY or SUPERUSER_API_TOKEN at all, it might be something to do with table permissions after the restore.

[mmguero]

The issue actually seems to be with the SUPERUSER_API_TOKEN logstash is using for the connection:

logstash@logstash:~$ curl -v -H "Authorization: Token $SUPERUSER_API_TOKEN" -H "Accept: application/json; indent=4" -L "http://netbox:8080/netbox/api/dcim/sites/?limit=1&name=Celestic&offset=0"
*   Trying 172.27.11.10:8080...
* TCP_NODELAY set
* Connected to netbox (172.27.11.10) port 8080 (#0)
> GET /netbox/api/dcim/sites/?limit=1&name=Celestic&offset=0 HTTP/1.1
> Host: netbox:8080
> User-Agent: curl/7.68.0
> Authorization: Token WJxp5b3iWniMLCpk9mwHUJGms7Qg3dZyXYbWc_k3
> Accept: application/json; indent=4
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Type: application/json
< Vary: HX-Request, Cookie, origin
< Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS
< X-Request-ID: 28a7a674-b605-4556-859e-4fe91efa7353
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Cross-Origin-Opener-Policy: same-origin
< X-Frame-Options: SAMEORIGIN
< Content-Length: 33
< Server: Unit/1.31.1
< Date: Mon, 13 Nov 2023 17:01:48 GMT
< 
{
    "detail": "Invalid token"
}
* Connection #0 to host netbox left intact

Both containers do have the right environment variables. But somehow restarting netbox as we restore the database it's not getting picked up.

[mmguero]

A little more issue contaxt:

Before restore:

$ curl -v -H "Authorization: Token $SUPERUSER_API_TOKEN" -H "Accept: application/json; indent=4" -L "http://netbox:8080/netbox/api/dcim/sites/?limit=1&name=Celestic&offset=0"
*   Trying 172.27.12.10:8080...
* TCP_NODELAY set
* Connected to netbox (172.27.12.10) port 8080 (#0)
> GET /netbox/api/dcim/sites/?limit=1&name=Celestic&offset=0 HTTP/1.1
> Host: netbox:8080
> User-Agent: curl/7.68.0
> Authorization: Token WJxp5b3iWniMLCpk9mwHUJGms7Qg3dZyXYbWc_k3
> Accept: application/json; indent=4
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: HX-Request, Cookie, origin
< Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS
< X-Request-ID: 3bfb566e-4d16-456b-8628-bbdb0345a5a3
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Cross-Origin-Opener-Policy: same-origin
< X-Frame-Options: SAMEORIGIN
< Content-Length: 1114
< Server: Unit/1.31.1
< Date: Mon, 13 Nov 2023 17:08:09 GMT
< 
{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 1,
            "url": "http://netbox:8080/netbox/api/dcim/sites/1/",
            "display": "Celestic",
            "name": "Celestic",
            "slug": "celestic",
            "status": {
                "value": "active",
                "label": "Active"
            },
            "region": null,
            "group": null,
            "tenant": null,
            "facility": "",
            "time_zone": null,
            "description": "",
            "physical_address": "",
            "shipping_address": "",
            "latitude": null,
            "longitude": null,
            "comments": "",
            "asns": [],
            "tags": [],
            "custom_fields": {},
            "created": "2023-11-13T17:08:02.286598Z",
            "last_updated": "2023-11-13T17:08:02.286608Z",
            "circuit_count": 0,
            "device_count": 0,
            "prefix_count": 0,
            "rack_count": 0,
            "virtualmachine_count": 0,
            "vlan_count": 0
        }
    ]
* Connection #0 to host netbox left intact

after restore:

$ curl -v -H "Authorization: Token $SUPERUSER_API_TOKEN" -H "Accept: application/json; indent=4" -L "http://netbox:8080/netbox/api/dcim/sites/?limit=1&name=Celestic&offset=0"
*   Trying 172.27.12.10:8080...
* TCP_NODELAY set
* Connected to netbox (172.27.12.10) port 8080 (#0)
> GET /netbox/api/dcim/sites/?limit=1&name=Celestic&offset=0 HTTP/1.1
> Host: netbox:8080
> User-Agent: curl/7.68.0
> Authorization: Token WJxp5b3iWniMLCpk9mwHUJGms7Qg3dZyXYbWc_k3
> Accept: application/json; indent=4
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Type: application/json
< Vary: HX-Request, Cookie, origin
< Allow: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS
< X-Request-ID: 2de89ac9-1aeb-4ef0-a37a-add12c6d0eb3
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Cross-Origin-Opener-Policy: same-origin
< X-Frame-Options: SAMEORIGIN
< Content-Length: 33
< Server: Unit/1.31.1
< Date: Mon, 13 Nov 2023 17:12:44 GMT
< 
{
    "detail": "Invalid token"
* Connection #0 to host netbox left intact

[mmguero]

For my tracking purposes, going to ask in the netbox slack forum with this:

Hi, I'm working on an issue I've got in my project dealing with replicating netbox (ie., backup/restore). Starting a thread here to keep it all under one discussion.

I've read Replicating NetBox and as far as I can tell I'm doing the right things to do the restore.

To give a little bit of background to how netbox is running, I have netbox running in a Docker container, based on the netboxcommunity/netbox image. The main netbox service and rqworker and housekeeping services are running under a supervisord process. Everything works as it should as far as NetBox features go, no issues just running as normal.

It's when I go to do a restore of another database that I start to run into issues. From what I've read on GitHub, there's nothing in the secret key or tokens or anything in the actual database itself, just session cookies, so I don't think it's an issue with having a different secret key.

The behavior I have it boiled down to is pretty much this:

1. Run a curl command against the NetBox API using the $SUPERUSER_API_TOKEN token, no error (200 OK)
2. Do my database restore process (more info on this below)
3. Run the same curl command against the NetBox API using the $SUPERUSER_API_TOKEN token, error 403 Forbidden with "detail": "Invalid token"

My restore process looks like this:

1. Stop the netbox processes (main, rqworker and housekeeping)
2. Drop the postgreSQL database
3. Create a new netbox database
4. GRANT ALL PRIVILEGES on DATABASE netbox TO netbox
5. Load the backed-up database dump from a previous psql command
6. Start back up the netbox processes
7. manage.py migrations

After the restore I am able to hit the UI and I see all of my data restored as it should be. However, when I attempt to run the curl command with $SUPERUSER_API_TOKEN I get the 403 error and Invalid token message as I've described.

[mmguero]

Verbose output:

/scripts/netbox-restore -v --netbox-restore ~/Desktop/celestic_netbox_backup_v23.11.0.gz 
...
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox', 'supervisorctl', 'stop', 'netbox:*']() returned 0: ['netbox:housekeeping: stopped', 'netbox:main: stopped', 'netbox:worker: stopped', '']
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox', 'bash', '-c', 'pgrep -f /usr/local/bin/netbox_init.py | xargs -r kill']() returned 143: []
Error interrupting netbox_init.py: []
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox-postgres', 'dropdb', '-U', 'netbox', 'netbox', '--force']() returned 0: []
Error dropping NetBox database: []
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox-postgres', 'createdb', '-U', 'netbox', 'netbox']() returned 0: []
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox-postgres', 'psql', '-U', 'netbox', '-c', 'GRANT ALL PRIVILEGES ON DATABASE netbox TO netbox']() returned 0: ['GRANT', '']
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox-postgres', 'psql', '-U', 'netbox'](--
-- PostgreSQL database dump
--

-- Dumped from database version 15.4
-- Dumpe...) returned 0: [...]
  ['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox', 'supervisorctl', 'start', 'netbox:housekeeping', 'netbox:main', 'netbox:worker']() returned 0: ['netbox:housekeeping: started', 'netbox:main: started', 'netbox:worker: started', '']
['docker', 'compose', '--profile', 'malcolm', '-f', '/home/user/devel/github/Malcolm/userdev.yml', 'exec', '-T', '-u', '1001:1001', 'netbox', '/opt/netbox/netbox/manage.py', 'migrate']() returned 0: [" loaded config '/etc/netbox/config/configuration.py'", " loaded config '/etc/netbox/config/extra.py'", " loaded config '/etc/netbox/config/logging.py'", " loaded config '/etc/netbox/config/plugins.py'", 'Operations to perform:', '  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless', 'Running migrations:', '  No migrations to apply.', '']

[mmguero]

I'm wondering if it has to do with the users_token table in the database dump... if the SUPERUSER_API_TOKEN "now" isn't the same one it was in the original backup, it's probably not getting updated/recreated

[mmguero]

Turns out that is the problem. I had to do this:

Yes, that is the issue. I had to do this:

psql -U netbox -c "UPDATE users_token SET key='blahblahblah' WHERE ID IN (SELECT ID FROM auth_user WHERE is_superuser = 'true' ORDER BY date_joined LIMIT 1)"

I'll have to work something like that into my restore process.

The SUPERUSER_API_TOKEN apparently only gets used if there's nothing in the database already.