Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx installer not properly reloading configuration #7422

Open
Ackis opened this issue Oct 4, 2019 · 7 comments

Comments

@Ackis
Copy link

commented Oct 4, 2019

Detailed conversation could be found here:
https://community.letsencrypt.org/t/cannot-renew-create-a-new-cert-when-i-had-no-issue-previously/103040/

My operating system is (include version):

Ubuntu 18.04.3 LTS
Nginx 1.17.4 (note, this isn't the version of nginx that's part of Ubuntu's repo's)

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

I ran this command and it produced this output:

sudo certbot renew --cert-name ackis.duckdns.org --debug-challenges --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ackis.duckdns.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ackis.duckdns.org
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges
Attempting to renew cert (ackis.duckdns.org) from /etc/letsencrypt/renewal/ackis.duckdns.org.conf produced an unexpected error: Failed authorization procedure. ackis.duckdns.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/7bO_DNxtDDyO_hPdRJcpGWEJHaLTRwtTsMWpWtQREDE [174.3.126.96]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - -  - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: ackis.duckdns.org
   Type:   unauthorized
   Detail: Invalid response from
   https://ackis.duckdns.org/.well-known/acme-challenge/7bO_DNxtDDyO_hPdRJcpGWEJHaLTRwtTsMWpWtQREDE
   [174.3.126.96]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Certbot's behavior differed from what I expected because:

Certbot should have renewed the certificate.

Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

# HTTP server - redirect to HTTPS
server {
        listen                                  80;
        listen                                  [::]:80;
        server_name                             www.ackis.duckdns.org ackis.duckdns.org;

        location / {
                return                          301 https://ackis.duckdns.org$request_uri;
        }

        # Workaround LE and certbot not working with nginx
        location /.well-known/acme-challenge/ {
                                                root /var/www/letsencrypt;
        }

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";
}

# www domain - redirect to domain without www
server {
        listen                                  443 ssl;
        listen                                  [::]:443 ssl;
        server_name                             www.ackis.duckdns.org;

        location / {
                return                          301 https://ackis.duckdns.org$request_uri;
        }

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        ssl_certificate                         /etc/letsencrypt/live/www.ackis.duckdns.org/fullchain.pem;
        ssl_certificate_key                     /etc/letsencrypt/live/www.ackis.duckdns.org/privkey.pem;

        server_tokens                           off;
        etag                                    off;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";
}

server {
        listen                                  443 ssl default_server;
        listen                                  [::]:443 ssl;
        server_name                             ackis.duckdns.org;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        ssl_certificate                         /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key                     /etc/letsencrypt/live/ackis.duckdns.org/privkey.pem; # managed by Certbot

        server_tokens                           off;
        etag                                    off;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";

        location ~* /\.\./ {
                deny                            all;
                return                          404;
        }

        location ~* "^(?:.+\.(?:htaccess|make|txt|test|markdown|md|engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Re$
                return                          404;
        }

        location = /favicon.ico {
                try_files                       /favicon.ico =204;
        }

        location / {
                root                            /var/www/internet;
                index                           index.html;
        }
        location /nginx_status {
                access_log                      syslog:server=localhost,tag=nginx_access_admin,severity=info;
                error_log                       syslog:server=localhost,tag=nginx_error_admin;
                allow                           192.168.0.0/24;
                deny                            all;

                auth_basic                      "Restricted access";
                auth_basic_user_file            /etc/nginx/auth/admin.htpasswd;

                stub_status                     on;
        }

}

@alexzorin

This comment has been minimized.

Copy link
Collaborator

commented Oct 4, 2019

Nginx 1.17.4 (note, this isn't the version of nginx that's part of Ubuntu's repo's)

Was it installed from https://nginx.org/en/linux_packages.html#Ubuntu or elsewhere?

@Ackis

This comment has been minimized.

Copy link
Author

commented Oct 4, 2019

Nginx 1.17.4 (note, this isn't the version of nginx that's part of Ubuntu's repo's)

Was it installed from https://nginx.org/en/linux_packages.html#Ubuntu or elsewhere?

Yes, it's from their official repo.

@joohoi

This comment has been minimized.

Copy link
Member

commented Oct 4, 2019

Thanks for opening the issue. I wonder if there's something different in the systemd service that gets installed with the nginx version from Nginx repositories. I'm thinking of anything that would cause the nginx restarts to silently fail.

@alexzorin

This comment has been minimized.

Copy link
Collaborator

commented Oct 4, 2019

I tried to reproduce this with Ubuntu 18.04.3, nginx 1.17.4 from nginx mainline, Certbot 0.31.0 from PPA, using the posted configuration (with the domain name substituted and chopped-off regex fixed). Everything worked OK.

Would you be able to zip up your entire /etc/nginx directory and upload it? From your thread, I believe there are a large number of other virtualhosts on the server also. They could be having an influence.

e.g. Certbot reloads nginx (indirectly) via SIGHUP, which is asynchronous. Your nginx takes a long time to reload its full configuration. So long that the challenge requests arrive before the new config is actually applied. The challenges fail.

I'd also like to see whether there are nginx packages from multiple locations installed:

dpkg-query -l "*nginx*"
@Ackis

This comment has been minimized.

Copy link
Author

commented Oct 5, 2019

dpkg-query -l "*nginx*"

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version        Architecture   Description
+++-===================-==============-==============-===========================================
ii  nginx               1.17.4-1~bioni amd64          high performance web server
rc  nginx-common        1.10.3-0ubuntu all            small, powerful, scalable web/proxy server
un  nginx-core          <none>         <none>         (no description available)
ii  nginx-doc           1.14.0-0ubuntu all            small, powerful, scalable web/proxy server
un  nginx-extras        <none>         <none>         (no description available)
un  nginx-full          <none>         <none>         (no description available)
un  nginx-light         <none>         <none>         (no description available)
ii  python-certbot-ngin 0.31.0-1+ubunt all            transitional dummy package
ii  python-certbot-ngin 0.23.0-1       all            Nginx plugin documentation for Certbot
ii  python3-certbot-ngi 0.31.0-1+ubunt all            Nginx plugin for Certbot

And my nginx config files are here: nginx.tar.gz

@beret

This comment has been minimized.

Copy link

commented Oct 9, 2019

I've faced similar issues on my deployment (16.04 LTS, NGINX PPA, Certbot PPA)
Details here: https://community.letsencrypt.org/t/unable-to-renew-cert-using-nginx-plugin-fresh-cert-creation-succeeds/102843
My pastebin links, converted to attachments: certbot issue 7422 - nginx -t.txt
certbot debug output with split root domain blocks - http tls.txt

Since reporting that issue, I reviewed my certbot per-site configuration files. I removed some pre and post hooks for legacy certificates, and unified them to all use the nginx validation and installer plugins. However the impacted certificates, including my test-case in that thread, were already inline with the modern plugins and no special hooks.

I've been able to complete those outstanding renewals, but I had to rerun certbot renew repeatedly as it churned through the renewals, failing to complete some renewals with similar failures consistent with improperly reloaded server configs.

In my test cases, I've been able to make renewals succeed just by running systemctl reload nginx repeatedly in a second window.

@Ackis

This comment has been minimized.

Copy link
Author

commented Oct 10, 2019

My nginx reloads instantly, but seems to take over a minute to restart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.