New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Acme.sh certificate renewal is not complete #495

Closed
ErisDS opened this Issue Oct 10, 2017 · 9 comments

Comments

3 participants
@ErisDS
Member

ErisDS commented Oct 10, 2017

I'm having problems with sites going offline with certificate errors. Acme.sh is correctly renewing the certificates, but nginx must be restarted afterwards, and I don't think that has been configured.

I'm not sure why we haven't had a tonne more reports of this!

Refs:

Update from @ErisDS 23rd Oct:

We're starting to see more people affected by this. To aid search-ability this manifests itself as NET::ERR_CERT_DATE_INVALID errors on your site.

The certificate has renewed, but nginx needs to be restarted to pick up the new certs:

The solution is to run sudo systemctl restart nginx on your box.


Task list

(Taken from my comment below)

  • 1. Build out the new version
  • 2. Write the documentation for how to manually migrate your site to the new version
    • My expectation is the simple version would be "run ghost setup:ssl, then delete X files" or similar, and then there'd be an extended version explaining the exact steps for anyone with a customised system.
  • 3. Release the new version as a minor?

At this point, all new sites are safe, all old ones can migrate.

  • 4. Write a check that can determine if a blog is using the old ssl setup, make it as robust and simple as possible. E.g. use fs.exists.
  • 5. Add a message to as many places as possible shouting about the need to update, linking to the docs.
  • 6. Release as a patch update

At this point, all old blogs will learn about the need to migrate. They will be able but it's not easy.

  • 7. Add a new command: ghost migrate:ssl or something equally relatively sensible and future-proof.
  • 8. Make that command do the necessary migration steps
    - e.g. it runs ghost setup:ssl and then deletes the old files, or maybe does a more advanced set of steps to be more compatible?
  • 9. Update the docs with the shiny new command
  • 10. Maybe update the messages every where with the new command?

At this point, all old blogs have an easy way to migrate.

  • 11. Deal with any fall out, e.g. bugs in the check, or improvements to the migrate command to ensure these are robust.
  • 12. Implement a system to auto-migrate

@ErisDS ErisDS added bug rogue labels Oct 10, 2017

@benaubin

This comment has been minimized.

Show comment
Hide comment

benaubin commented Oct 21, 2017

@acburdine acburdine added this to Next Release in Ghost-CLI Oct 21, 2017

@acburdine

This comment has been minimized.

Show comment
Hide comment
@acburdine

acburdine Oct 23, 2017

Member

So yeah, this bug is pretty bad. I apologize for not catching this one sooner.

After doing some research - I've discovered an unfortunate reality: We need sudo permissions to restart nginx.

There are several different commands that can be run to reload nginx that I found. They are:

  • service reload nginx / systemctl nginx reload (for our intent they're essentially the same)
  • nginx -s reload
  • kill -HUP <nginx process id>

None of the above commands works without sudo permissions, specifically because it requires messing with the nginx process, which the user that runs the acme script won't be able to do without sudo. Also with sudo access, it makes the cron script difficult because it requires a password to be typed in.

As such, there are two different solutions that I can see:

  1. Run acme as the root user & set it up as such.

This solution is probably the easier one to implement for new installations, however it would be a bit of a nightmare for updating existing Ghost-CLI installations. It's also potentially more of a security issue.

  1. Add a line to the sudoers file manually that allows sudo to be executed without a password ONLY for the nginx reload command.

This would potentially be a lot easier to migrate older installations to - and would alleviate the problem of running things as root. However, it would require modification of the sudoers file which I bet most people might be a bit fearful of. We could do it automatically, but it's still a bit scary.

This is just my results from the research I've done into this, the reason I'm posting it is I want to make sure there aren't any other ways of accomplishing this that don't involve major changes with sudo.

@sebgie I'm gonna ping you on this one as well, have any other ideas?

Member

acburdine commented Oct 23, 2017

So yeah, this bug is pretty bad. I apologize for not catching this one sooner.

After doing some research - I've discovered an unfortunate reality: We need sudo permissions to restart nginx.

There are several different commands that can be run to reload nginx that I found. They are:

  • service reload nginx / systemctl nginx reload (for our intent they're essentially the same)
  • nginx -s reload
  • kill -HUP <nginx process id>

None of the above commands works without sudo permissions, specifically because it requires messing with the nginx process, which the user that runs the acme script won't be able to do without sudo. Also with sudo access, it makes the cron script difficult because it requires a password to be typed in.

As such, there are two different solutions that I can see:

  1. Run acme as the root user & set it up as such.

This solution is probably the easier one to implement for new installations, however it would be a bit of a nightmare for updating existing Ghost-CLI installations. It's also potentially more of a security issue.

  1. Add a line to the sudoers file manually that allows sudo to be executed without a password ONLY for the nginx reload command.

This would potentially be a lot easier to migrate older installations to - and would alleviate the problem of running things as root. However, it would require modification of the sudoers file which I bet most people might be a bit fearful of. We could do it automatically, but it's still a bit scary.

This is just my results from the research I've done into this, the reason I'm posting it is I want to make sure there aren't any other ways of accomplishing this that don't involve major changes with sudo.

@sebgie I'm gonna ping you on this one as well, have any other ideas?

@acburdine

This comment has been minimized.

Show comment
Hide comment
@acburdine

acburdine Oct 23, 2017

Member

After a conversation with @sebgie on Slack, we've decided to go with option 1.

Acme.sh has an option to set the certs up in a location other than the home directory - for new installs it will install all the certs to /etc/letsencrypt rather than ~/.acme.sh. We will also run acme.sh as root, which fixes any permissions issues we have with nginx.

This will require building out some of the abilities to migrate installation structures, as moving existing ssl setups with Ghost-CLI will require a bit of extra work.

Member

acburdine commented Oct 23, 2017

After a conversation with @sebgie on Slack, we've decided to go with option 1.

Acme.sh has an option to set the certs up in a location other than the home directory - for new installs it will install all the certs to /etc/letsencrypt rather than ~/.acme.sh. We will also run acme.sh as root, which fixes any permissions issues we have with nginx.

This will require building out some of the abilities to migrate installation structures, as moving existing ssl setups with Ghost-CLI will require a bit of extra work.

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 23, 2017

Member

This will require building out some of the abilities to migrate installation structures, as moving existing ssl setups with Ghost-CLI will require a bit of extra work.

I totally get that we need to do this sometime, and that it's not a small amount of work.

For now, how does this approach sound:

    1. Build out the new version
    1. Write the documentation for how to manually migrate your site to the new version
      • My expectation is the simple version would be "run ghost setup:ssl, then delete X files" or similar, and then there'd be an extended version explaining the exact steps for anyone with a customised system.
    1. Release the new version as a minor?

At this point, all new sites are safe, all old ones can migrate.

    1. Write a check that can determine if a blog is using the old ssl setup, make it as robust and simple as possible. E.g. use fs.exists.
    1. Add a message to as many places as possible shouting about the need to update, linking to the docs.
    1. Release as a patch update

At this point, all old blogs will learn about the need to migrate. They will be able but it's not easy.

    1. Add a new command: ghost migrate:ssl or something equally relatively sensible and future-proof.
    1. Make that command do the necessary migration steps
      • e.g. it runs ghost setup:ssl and then deletes the old files, or maybe does a more advanced set of steps to be more compatible?
    1. Update the docs with the shiny new command
    1. Maybe update the messages every where with the new command?

At this point, all old blogs have an easy way to migrate.

    1. Deal with any fall out, e.g. bugs in the check, or improvements to the migrate command to ensure these are robust.
    1. Implement a system to auto-migrate

If it's possible, joining 1-6 together would be ideal, but I'd rather ship something that can be done manually as soon as physically possible, than delay at all on making migrations easier/not manual.

Also - if you can provide a bullet-point list of commands that result in a migration, then I can write the docs 😊

Member

ErisDS commented Oct 23, 2017

This will require building out some of the abilities to migrate installation structures, as moving existing ssl setups with Ghost-CLI will require a bit of extra work.

I totally get that we need to do this sometime, and that it's not a small amount of work.

For now, how does this approach sound:

    1. Build out the new version
    1. Write the documentation for how to manually migrate your site to the new version
      • My expectation is the simple version would be "run ghost setup:ssl, then delete X files" or similar, and then there'd be an extended version explaining the exact steps for anyone with a customised system.
    1. Release the new version as a minor?

At this point, all new sites are safe, all old ones can migrate.

    1. Write a check that can determine if a blog is using the old ssl setup, make it as robust and simple as possible. E.g. use fs.exists.
    1. Add a message to as many places as possible shouting about the need to update, linking to the docs.
    1. Release as a patch update

At this point, all old blogs will learn about the need to migrate. They will be able but it's not easy.

    1. Add a new command: ghost migrate:ssl or something equally relatively sensible and future-proof.
    1. Make that command do the necessary migration steps
      • e.g. it runs ghost setup:ssl and then deletes the old files, or maybe does a more advanced set of steps to be more compatible?
    1. Update the docs with the shiny new command
    1. Maybe update the messages every where with the new command?

At this point, all old blogs have an easy way to migrate.

    1. Deal with any fall out, e.g. bugs in the check, or improvements to the migrate command to ensure these are robust.
    1. Implement a system to auto-migrate

If it's possible, joining 1-6 together would be ideal, but I'd rather ship something that can be done manually as soon as physically possible, than delay at all on making migrations easier/not manual.

Also - if you can provide a bullet-point list of commands that result in a migration, then I can write the docs 😊

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 23, 2017

Member

I've updated the top issue to make it more searchable. I have also added a section to the troubleshooting guide: https://docs.ghost.org/v1/docs/troubleshooting#section-net-err_cert_date_invalid, and a red callout on the install page: https://docs.ghost.org/v1/docs/install.

Hopefully this will help anyone who runs into this issue (had 2 today in #help on https://slack.ghost.org) find the answer quickly.

Let me know if there's anywhere else I should update or any further docs I can write.

Member

ErisDS commented Oct 23, 2017

I've updated the top issue to make it more searchable. I have also added a section to the troubleshooting guide: https://docs.ghost.org/v1/docs/troubleshooting#section-net-err_cert_date_invalid, and a red callout on the install page: https://docs.ghost.org/v1/docs/install.

Hopefully this will help anyone who runs into this issue (had 2 today in #help on https://slack.ghost.org) find the answer quickly.

Let me know if there's anywhere else I should update or any further docs I can write.

@acburdine acburdine moved this from Next Release to In Progress in Ghost-CLI Oct 24, 2017

acburdine added a commit to acburdine/Ghost-CLI-1 that referenced this issue Oct 24, 2017

fix(ssl): rework how we use acme.sh
closes TryGhost#495
- install acme.sh as root in order to make the nginx reload script work correctly
- catch domain verification error better than we were doing previously

acburdine added a commit to acburdine/Ghost-CLI-1 that referenced this issue Oct 24, 2017

fix(ssl): rework how we use acme.sh
closes TryGhost#495
- install acme.sh as root in order to make the nginx reload script work correctly
- catch domain verification error better than we were doing previously

acburdine added a commit to acburdine/Ghost-CLI-1 that referenced this issue Oct 29, 2017

fix(ssl): rework how we use acme.sh
closes TryGhost#495
- install acme.sh as root in order to make the nginx reload script work correctly
- catch domain verification error better than we were doing previously

acburdine added a commit to acburdine/Ghost-CLI-1 that referenced this issue Oct 29, 2017

fix(ssl): rework how we use acme.sh
closes TryGhost#495
- install acme.sh as root in order to make the nginx reload script work correctly
- catch domain verification error better than we were doing previously

acburdine added a commit to acburdine/Ghost-CLI-1 that referenced this issue Oct 30, 2017

fix(ssl): rework how we use acme.sh
closes TryGhost#495
- install acme.sh as root in order to make the nginx reload script work correctly
- catch domain verification error better than we were doing previously

@acburdine acburdine closed this in 6f30109 Oct 30, 2017

@acburdine acburdine moved this from In Progress to Done, pending release in Ghost-CLI Oct 30, 2017

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin commented Nov 1, 2017

nice!

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Nov 1, 2017

Member

@acburdine should we maybe move the list of tasks out of this issue now and into a new one?

Member

ErisDS commented Nov 1, 2017

@acburdine should we maybe move the list of tasks out of this issue now and into a new one?

@ErisDS ErisDS reopened this Nov 10, 2017

@kirrg001 kirrg001 referenced this issue Nov 10, 2017

Merged

Bump knex-migrator to version 3.1.1 #9199

12 of 12 tasks complete

@acburdine acburdine moved this from Done, pending release to In Progress in Ghost-CLI Nov 10, 2017

@acburdine acburdine referenced this issue Nov 14, 2017

Merged

CLI Migrations #536

4 of 4 tasks complete

acburdine added a commit to acburdine/Ghost-CLI-1 that referenced this issue Nov 16, 2017

@acburdine acburdine closed this in 3c60d89 Nov 16, 2017

@acburdine

This comment has been minimized.

Show comment
Hide comment
@acburdine

acburdine Nov 16, 2017

Member

Reopening this (again) so we can make sure to remember to do the docs updates 😅

Member

acburdine commented Nov 16, 2017

Reopening this (again) so we can make sure to remember to do the docs updates 😅

@acburdine acburdine reopened this Nov 16, 2017

@acburdine acburdine moved this from In Progress to Done, pending release in Ghost-CLI Nov 16, 2017

@acburdine

This comment has been minimized.

Show comment
Hide comment
@acburdine

acburdine Nov 23, 2017

Member

Docs have been updated, blog post published with more information about this bug & how to fix, so this is (finally 😛) done.

Member

acburdine commented Nov 23, 2017

Docs have been updated, blog post published with more information about this bug & how to fix, so this is (finally 😛) done.

@acburdine acburdine closed this Nov 23, 2017

@acburdine acburdine moved this from Done, pending release to Released in Ghost-CLI Nov 23, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment