Recovering from breaking update (cannot load certificate)

[lorenzopicoli]

---

Update: I think this happens if you forgot to set the letsencrypt folder as a volume and tried to upgrade to another version

---

You might be seeing an error like this this:

[nginx ] nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-3/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/npm-3/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib)

The bad news is that you may have lost your SSL certificates: https://github.com/NginxProxyManager/nginx-proxy-manager/releases/tag/v2.10.4

If you don't have a snapshot of your docker container you just lost all of your SSL certificates.

This error message is happening because the services tries to read from these files (that were probably there at some point) on startup and when it doesn't find it, it can't start.

Now the good news is that I managed to get it working and I'll try to explain what I did here.

Update docker compose

In your docker compose file, create a new volume. This is the change that broke your installation. Your volumes section should look like this:

    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

Make sure that is done, the image tag is updated to whatever version you were trying to upgrade to. Now go ahead and restart the docker container.

Get the server to start

Now to get the server to start, we'll create the folders that it expects to exist. You lost the SSL certs so you can't fully recover those files, but we can put some dummy information in there to get the server to start.

``

> sudo docker exec -it bd4f15e863cf  /bin/bash
> cd /etc/letsencrypt
> mkdir live
> cd live

Now in another terminal window, get the logs from the container:

sudo docker logs --follow e2629901592b

You should see something like this:

[nginx ] nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-3/fullchain.pem": BIO_new_file() failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/etc/letsencrypt/live/npm-3/fullchain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib)

The important part of this message is the npm-x part.

Keep this window open. Now back to the previous window, run mkdir npm-x where x is the same number you see in the logs.

> cd npm-x
> openssl genrsa > privkey.pem
> openssl req -new -x509 -key privkey.pem > fullchain.pem

As soon as you answer the questions you should see that the error message is gone on the other terminal window. If it's not gone it should have at least changed to another npm-y folder. Duplicate the folder for each and every folder that the server throws an error for. I had to do it for 20 folders.

Once you get to a point where all the folders are back in place you'll see: Server started which means you should have access to the web ui again.

Getting certificates again

Now to get your certificates back, go to the SSL tab and delete all your old entries, they are gone. Re generate all of your certificates.

You MUST go back to the hosts tab and for each one of them go on edit and make sure that your NEW SSL certificate is selected otherwise it won't work

Cleaning up

Now go back in the container and run ls -halt in the /etc/letsencrypt/live folder. You will be able to see the folders and when they were created. Simply delete the ones that you created initially. I recommend you cd into each one of them and make sure it only contains privkey.pem and fullchain.pem, if it contains other information like a README file, don't delete the folder.

If you screw up and delete something you shouldn't, you can see which host was affected by installing sqlite and checking the DB in sqlite3 /data/database.sqlite with select id, domain_names from certificate WHERE is_deleted = 0; .

Making sure it all works

Restart the pod and make sure it starts properly

[Ed1ks]

Nginx proxy manager should handle this automated, as this can stop a whole production server.

[brunomueller93]

So I had to follow your procedure after manually saving a dying server.

I moved all docker-compose.yml files manually from Server A to Server B with a flash drive and also made sure to copy all of the docker volumes.

In my docker-compose.yml I have the following volumes for NPM:

volumes:
- /media/raid/docker/docker/nginx-manager/data:/data
- /media/raid/docker/docker/nginx-manager/letsencrypt:/etc/letsencrypt

I think that I copied everything with sudo rsync but I'm not 100% sure. Anyways, my folders in /media/raid/docker/docker/nginx-manager/letsencrypt/live/npm-x/ were empty. So maybe I did not use sudo for copying and I did not have access to those subfolders, so they were left empty while copying.

Anyway, I also agree that NPM should have the ability to recover from such an event and be able to continue without these files. It should not handle this error with not starting.

[IIPoliII]

Thanks for this guide !

[aikiharp]

@lorenzopicoli thank you so much! Fortunately I had bind mounts for the folders, but NPM was looking for npm-2 when I only had npm-1. I renamed npm-1 to npm-2, and then it was looking for npm-1! I copied npm-2 and renamed it npm-1 and no more errors. What a relief! I spent all night trying to figure this out.

[scandala2334]

@aikiharp Thank you, thank you, thank you. This one little paragraph pointed me in the right direction. Thanks for taking the time to write this comment.

[glarue]

Chiming in many months later to say that this comment saved me as well, although in my case (from the logs) NPM was looking for /etc/letsencrypt/live/npm-12 and I only had npm-2. From within /etc/letsencrypt/live I did cp -r npm-2 npm-12 to be safe and that solved the issue.

[InteractionEngineer]

Wow, thanks from my side as well! After deleting some entries and trying to get another ssl cert via a DNS chellange, my instance crashed, bootlooped and looked for npm-22.

nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-23/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-22/fullchain.pem, r) error:10000080:BIO routines::no such file)

After I faked that it was searching for npm-23. Copying one of the existing npm-folders and naming them accordingly brought my proxymanager back to life - fully functional again, but somewhat concerning.

[ssuygur]

thanks a lot man! you saved me

[dmalanij]

Thanks a lot @lorenzopicoli, you rock!

Your post saved me lots of time, as I was struggling with ngynx not starting due to the missing .pem files. I don't think my case is due to the upgrade, as I've the letsencrypt mapped and all; I'm actually coming back from a hard drive failure so lost files is not surprising (weird for me is that npm-x folders were recovered with the README on them, but no sign of the certificates 🤷).

Thanks again!

[Yrobot]

Thanks a lot, it is very helpful.