Entries not showing on the front-end after updating image from 2.9.4 to 2.9.9 (database overwritten) #1402
Description
Issue Summary
After updating the NPM image and redeploying my Docker-Compose stack, my previous login details were reset to the admin@example.com / changeme defaults, which was the first flag something wasn't right.
And yup, none of existing configuration are showing in the NPM front-end. I have approximately 15 proxy hosts and a redirect set, but obviously the front-end has no idea of this.
Existing reverse proxy routing all appear to be working fine, i.e. I can reach my hosted web services from external networks using the subdomain routes that I configured in NPM previously. I just can't see the entries on the NPM front-end.
Investigation
The Docker stack's mount folder data/nginx/proxy_host directory still contains all of the conf files. And while all the routes are currently working (thankfully), I obviously can't make any changes to the existing configs, and I'm assuming it might cause problems with Certbot updates.
I checked the container logs and it seems like NPM couldn't find/connect to the existing database (logs provided below). I opened up the database container port, and connected to it using DBeaver. Low and behold, it's an entirely fresh database with no entries.
Hypothesis
So what I gather from all this is that the instance of Nginx running in the NPM container works fine because all my proxy_host .conf files are still in the data directory, and NPM just points Nginx to those directories. Whereas NPM keeps all the user config in the database, and simply generates the proxy host .conf files to the data directory.
Unfortunately I didn't make a backup before the update. So I'm guessing that my best course of action is just to grab all the .conf files, open them up in an editor, and manually enter all entries in a fresh NPM instance.
Ironically, that would have taken me far less time to gather and write up all this information.
Conclusion
- Something went wrong with my update procedure to the point where my entire database was cleared.
- Considering I didn't change anything in my Docker-Compose file, and simply updated an image and restarted the stack (like I always do), it's a bit of a problem.
- It's either an issue with NPM, or I have a weird docker stack setup that was asking for trouble.
- This would be far less of an issue if NPM was able to import existing proxy host
.conffiles either from the front-end GUI, or from thedata/nginx/proxy_hostafter it builds a new database file.
Steps
Running an existing NPM installation with 2.9.4, I updated NPM using Portainer:
- Pulled the
latestimage of NPM. - Removed the existing NPM and DB containers.
- Redeployed the NPM stack.
- Pointed my browser to the NPM host on my LAN with port
81. - Logged in using NPM default credentials (because the previous ones didn't work).
- None of the proxy host entries are shown.
Deployment Specifics
- OS: Ubuntu 20.04.3 LTS aarch64
- Host: Raspberry Pi 4 Model B Rev 1.2
- Kernel: 5.4.0-1042-raspi
- Docker: version 20.10.7, build f0df350
- Docker-Compose: version 1.29.2, build unknown
- Docker management: Portainer 2.6.3 from remote host, linked with Portainer agent.
docker-compose.yml
version: '2'
services:
app:
image: 'jc21/nginx-proxy-manager:latest'
restart: unless-stopped
ports:
- '80:80'
- '81:81'
- '443:443'
environment:
DB_MYSQL_HOST: "db"
DB_MYSQL_PORT: 3306
DB_MYSQL_USER: secret
DB_MYSQL_PASSWORD: secret
DB_MYSQL_NAME: secret
volumes:
- /opt/nginx/data:/data
- /opt/nginx/letsencrypt:/etc/letsencrypt
db:
image: 'jc21/mariadb-aria:latest'
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: secret
MYSQL_DATABASE: secret
MYSQL_USER: secret
MYSQL_PASSWORD: secret
volumes:
- /opt/nginx/data/mysql:/var/lib/mysql
Digging around
After having a poke around in the NPM container, it certainly seems like NPM couldn't find the existing database, and just created a new one while keeping all the Nginx files intact.
NPM container log
...
2021-09-13T02:13:49.643099972Z [9/13/2021] [2:13:49 AM] [Global ] › ℹ info Generating MySQL db configuration from environment variables
2021-09-13T02:13:49.651550929Z [9/13/2021] [2:13:49 AM] [Global ] › ℹ info Wrote db configuration to config file: ./config/production.json
2021-09-13T02:13:52.401363725Z [9/13/2021] [2:13:52 AM] [Migrate ] › ℹ info Current database version: none
2021-09-13T02:13:52.492621960Z [9/13/2021] [2:13:52 AM] [Migrate ] › ℹ info [initial-schema] Migrating Up...
2021-09-13T02:13:52.525306175Z [9/13/2021] [2:13:52 AM] [Migrate ] › ℹ info [initial-schema] auth Table created
2021-09-13T02:13:52.564035773Z [9/13/2021] [2:13:52 AM] [Migrate ] › ℹ info [initial-schema] user Table created
...
Database log
2021-09-13T04:44:43.460256328Z [i] pre-init.d - processing /scripts/pre-init.d/01_secret-init.sh
2021-09-13T04:44:43.463819047Z [i] mysqld not found, creating....
2021-09-13T04:44:43.475643761Z [i] MySQL directory already present, skipping creation
2021-09-13T04:44:43.517391021Z 2021-09-13 4:44:43 0 [Note] /usr/bin/mysqld (mysqld 10.4.15-MariaDB) starting as process 1 ...
2021-09-13T04:44:43.585569811Z 2021-09-13 4:44:43 0 [Note] Plugin 'InnoDB' is disabled.
2021-09-13T04:44:43.585653217Z 2021-09-13 4:44:43 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-09-13T04:44:43.588433561Z 2021-09-13 4:44:43 0 [Note] Server socket created on IP: '::'.
2021-09-13T04:44:43.591583583Z 2021-09-13 4:44:43 0 [Warning] 'user' entry '@533783d7cd26' ignored in --skip-name-resolve mode.
2021-09-13T04:44:43.591894171Z 2021-09-13 4:44:43 0 [Warning] 'proxies_priv' entry '@% root@533783d7cd26' ignored in --skip-name-resolve mode.
2021-09-13T04:44:43.598983906Z 2021-09-13 4:44:43 0 [Note] Reading of all Master_info entries succeeded
2021-09-13T04:44:43.599044090Z 2021-09-13 4:44:43 0 [Note] Added new Master_info '' to hash table
2021-09-13T04:44:43.599062109Z 2021-09-13 4:44:43 0 [Note] /usr/bin/mysqld: ready for connections.
2021-09-13T04:44:43.599079145Z Version: '10.4.15-MariaDB' socket: '/run/mysqld/mysqld.sock' port: 3306 MariaDB Server
Docker container bind mount check
CLIing into the container, I can navigate to a complete set of proxy host entries in the mounted host directory. I confirmed the bind was correct using touch test from the container, and the file showing up in the host's directory.
Here's the proxy_host directory print out:
[root@docker-0b51012b4021:/data/nginx/proxy_host]# ls -la
total 68
drwxr-xr-x 2 root root 4096 Sep 13 03:45 .
drwxr-xr-x 9 root root 4096 Jul 11 17:37 ..
-rw-r--r-- 1 root root 1537 Sep 13 02:56 1.conf
-rw-r--r-- 1 root root 1596 Sep 13 02:56 10.conf
-rw-r--r-- 1 root root 1264 Sep 13 02:56 11.conf
-rw-r--r-- 1 root root 1552 Sep 13 02:56 12.conf
-rw-r--r-- 1 root root 1605 Sep 13 02:56 13.conf
-rw-r--r-- 1 root root 1607 Sep 13 02:56 14.conf
-rw-r--r-- 1 root root 1594 Sep 13 02:56 15.conf
-rw-r--r-- 1 root root 1593 Sep 13 02:56 2.conf
-rw-r--r-- 1 root root 1587 Sep 13 02:56 3.conf
-rw-r--r-- 1 root root 1595 Sep 13 02:56 4.conf
-rw-r--r-- 1 root root 1595 Sep 13 02:56 5.conf
-rw-r--r-- 1 root root 1533 Sep 13 02:56 6.conf
-rw-r--r-- 1 root root 1591 Sep 13 02:56 7.conf
-rw-r--r-- 1 root root 1591 Sep 13 02:56 8.conf
-rw-r--r-- 1 root root 1591 Sep 13 02:56 9.conf
Container Nginx config
The /etc/nginx/nginx.conf in the container appears to be referencing the proxy files generated by the previous NPM container, which is why the proxies are still working:
# Files generated by NPM
include /etc/nginx/conf.d/*.conf;
include /data/nginx/default_host/*.conf;
include /data/nginx/proxy_host/*.conf;
include /data/nginx/redirection_host/*.conf;
include /data/nginx/dead_host/*.conf;
include /data/nginx/temp/*.conf;