Implement a "script" plugin and/or fix manual for scriptability #2782

Closed
aug-riedinger opened this Issue Apr 7, 2016 · 19 comments

Projects

None yet

9 participants

@aug-riedinger

I guess I'm not the first one to raise the issue, but I haven't found anything yet by searching.

I'm trying to automate VM deployment with a letsencrypt certificate. So I have tried to remove user interaction as much as possible and reached:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos

But if I add the -n option, I get:

PluginError('Running manual mode non-interactively is not supported',)

I'm guessing the point is that manual mode does require user interaction hence cannot be fully done in non-interactive mode. At least during the file-validation step.

Which is why I'd like to discuss splitting the command in two steps:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos --generate-file /home/user

which would create the file /home/user/.well-known/acme-challenge/buCgmexKDQypJfWE4-XmUfP1cpILzsBR4UWRE_o8iuc with the expected content and then running

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos --validate

would finish the process.

Would this make sense? Did I miss something?

Another option would be to build allow a script to be executed between the file generation step and the verification step:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos --deploy-verification ./deploy-verification.sh
@bmw
Contributor
bmw commented Apr 7, 2016

This certainly does make sense. #2610 also suggests the script approach, but there are benefits to having two different client invocations as well. We certainly want to implement something like this in the future.

@bmw bmw added this to the 1.0.0 milestone Apr 7, 2016
@bmw bmw added the enhancement label Apr 7, 2016
@joohoi joohoi was assigned by pde Apr 7, 2016
@pde
Member
pde commented Apr 7, 2016

@joohoi is actually working on this, in part to make support for DNS challenges practical. I don't think he's decided on a a design yet, though some of the options under discussion include:

  • Run a "Step 1" command that gives you a challenge, and then a "Step 2" command when you've deployed it. Challenge: how to serialise and save state so that the client knows where to pick up from.
  • Call the client with two hook functions, one of which is to be run when a challenge is ready, the other when it has succeeded. Challenge: does this mean that the user has to write three separate scripts, one for each hook and one to call the client with the hooks?

Another option that @bmw and I talked about recently is:

  • Implement hooks, but encourage the user write a fancy script that the client can itself invoke, something like this:
#!/bin/bash

DeployChallenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # do stuff
}

CleanupChallenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # undo stuff
}

if [ -z "$WITHIN_LE_CLIENT" ] ; then
   letsencrypt certonly --script --script-challenge HTTP01 --challenge-hook ". $0 ; DeployChallenge"  --cleanup-hook ". $0 ; CleanupChallenge"
   # deploy cert
fi

That's not terrible, but it has the problems that the client invocation is a bit long and weird. And would even need extra stuff ($0 -> \"$0\") to work if the script had a space in its path. Which leads to:

  • Do the same thing as above, but bake in the bash function names and add some syntactic sugar. So the script would look like:
#!/bin/bash

DeployHTTP01Challenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # do stuff
}

CleanupHTTP01Challenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # undo stuff
}

if [ -z "$WITHIN_LE_CLIENT" ] ; then
   letsencrypt certonly --script --script-challenge HTTP01 --script-hooks "$0"
   # deploy cert
fi
@pde pde changed the title from Manual + Non Interactive mode to Implement a "script" plugin and/or fix manual for scriptability Apr 7, 2016
@bmw
Contributor
bmw commented Apr 8, 2016

The original post in #2610 should definitely be read for additional context as it was @brianmhunt and that post that gave us the idea for something like this in the first place.

pde and I have been thinking about this issue a lot over the past couple of days and I wanted to provide my thoughts.

For starters, I don't think doing this should involve two client invocations. That is first suggestion that pde posted. Doing this feels clunky, would require some major changes in the client, and I can't think of anything you could do in this approach that you couldn't do with passing commands/scripts to the client.

Next, between the options of passing one or two scripts to the client, I think I like the option of two scripts better. If we put everything in one large script, we have to have a LE defined way of separating out the deploy and cleanup code (such as the bash functions with specific names). This seems more complicated and less intuitive than just having separate deploy/cleanup commands/scripts.

Furthermore, as suggested by pde, having the client take two commands still provides a way to write a single script for the manual plugin that can be easily shared with others. The large script can define bash functions, create temporary files, etc.

Lastly, I'd like to suggest that we don't provide multiple ways to do this, unless there is a real benefit. We could provide something like --manual-deploy-command, --manual-cleanup-command, and --manual-combined-command, but by doing this we further complicate our command line and I'm unaware of anything you could do with one approach that you couldn't do with the other.

If anyone else has thoughts on this, I'd love to hear them. If this is done right, I think it could be a huge benefit to our users with more complex server setups.

@aug-riedinger

I'm glad to read that I'm not the only one having this need. ๐Ÿ˜„

@brianmhunt I'll have a look at your blog post to find an intermediary solution before this issue gets completed.

I'm not fully aware of how the client works, but as far as I'm concerned, having a script as an option of a CLI doesn't seem like a common pattern. It means the user has to create an external file to set its automated instructions. If I were to tutorialize the two options, steps would be:

  1. Deploy a file to the correct file
  2. Create a script with the above steps and make it executable
  3. Run letsencrypt certonly --deploy-script ./deploy-script.sh

This sounds more complicated than having a tutorial that says:

  1. Run letsencrypt certonly ...
  2. Go set your file in the server (which could be done by a script or manually depending on user prefrence)
  3. Run letsencrypt validate

Being a Rubyist, the first pattern felt like a decent solution, but from a shell perspective, I think I like the second one better.

But if it is not possible, and if you consider that anyways, if we are there, it is that we want to fully automate then both shall work.

@brianmhunt

Thanks everyone for the conversation and giving life to this issue! :) I'm cool with any proposal, so long as it gives end-to-end automation.

As @pde suggested, I'm putting the gist of my issue from #2610 below โ€“ not because I favour my method to the others (I don't have a solid enough grasp of the others to really think of all the edge cases), but just for ease of reference.

Cheers!


Right now the --manual plugin requires user interaction, but there are some use cases where it needn't.

For example, when using --manual with Google App Engine, one can automate that entire process, but it's quite convoluted.

The process could be vastly simplified if, instead of using manual input, the --manual plugin took a shell command, that I would imagine could work as follows:

  1. User calls letsencrypt --manual --shell-command "./upload-script --server=example.com" -d example.com -d www.example.com
  2. Let's Encrypt calls upload-script, and pipes the challenges to the standard input, something like:
example.com challenge-filename-1 challenge-1 
www.example.com challenge-filename-2 challenge-2

It would be trivial to generalize this with another argument like --shell-pipe-format with options such as csv, tsv, json, etc.

  1. Once the shell command completes Let's Encrypt runs the challenge.

Example

In the case of Google App Engine, a simplistic upload-script might look like (and my bash is a little rusty, but I hope it conveys the point):

# Write the challenge files to a place where AppEngine will read/publish them
CHALLENGE_PATH = "app/acme/challenges"
while read line
do
  CHALLENGE=$line[2]
  CHALLENGE_FILE="$CHALLENGE_PATH$line[1]"
  echo CHALLENGE > CHALLENGE_FILE
while done < "${1:-/dev/stdin//:/ }"

# Publish to AppEngine
appcfg.py update app

Once the script completes (and presuming that acme/challenges publishes to ./well-known/...), Let's Encrypt should validate successfully against the App Engine.

I find this nicely abstracts the validation process.

Compare this to the fragile monstrosities I have created to accomplish the same that carefully read and interpret the output of Let's Encrypt.

@joohoi
Member
joohoi commented Apr 9, 2016

Thanks for a good discussion everyone!

@pde @bmw I think it could be beneficial for the users, if we'd implement the hooking method that lukas2511/letsencrypt.sh uses. It's basically using one bash script with predefined function names for each needed scripted action. I think it's well thought, and would allow users to write interoperable scripts, and use which ever client they want. We'd provide the path to script on command line.

Example hook script here: hook.sh.example

We could easily provide example scripts for the most common operations as well. That would make the learning curve for users pretty much nonexistent.

@kuba
Contributor
kuba commented Apr 10, 2016
@brianmhunt

@kuba While letsencrypt-external will solve the issue for some scenarios, it comes with the constraint about servers of "as long as [the server validating] shares the same external IP address [as the server that will user the certificate]." That constraint can never be satisfied with providers such as Google AppEngine.

@aug-riedinger

@kuba I tried to install Letsencrypt-external but installation failed.

Here's the issue: marcan/certbot-external#1

@aug-riedinger

I finally made it through using the --webroot option:

sudo /opt/letsencrypt/letsencrypt-auto certonly --webroot -w /vagrant/www/current/public -d example.com -m contact@example.com -t --manual-public-ip-logging-ok --agree-tos

Using nginx, here's how I configured it:

upstream rails_app {
  server                localhost:3000;
}

server {

  listen 443 ssl;
  server_name           example.com;

  ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

  location / {
    proxy_pass          http://rails_app;
    proxy_read_timeout  90;
  }

}

server {
  listen                80;
  server_name           example.com;

  # Necessary for Let's Encrypt Domain Name ownership validation
  location /.well-known/acme-challenge/ {
    root /vagrant/www/current/public/.well-known/acme-challenge/;
  }
  return 301 https://$host$request_uri;
}

The location /.well-known/acme-challenge/ block makes the folder publicly accessible, hence validation is made automatically.

@silbe
silbe commented Apr 29, 2016

FWIW, #2053 is also related. I'm currently using a small plug-in to invoke scripts that deploy the challenge to the actual HTTP servers. This is similar to the "hooks" approach.

The two invocations approach proposed by @aug-riedinger is interesting because it allows the steps to be completely decoupled. There is no need for the letsencrypt client to stay running until the challenges have been deployed to all servers.

@joohoi joohoi was unassigned by aug-riedinger May 6, 2016
@joohoi joohoi was assigned by pde May 11, 2016
@pde
Member
pde commented May 13, 2016 edited

@marcan also appears to have been working on something similar:

https://github.com/marcan/letsencrypt-external

(edit: didn't see that Kuba had already mentioned that)

@joohoi
Member
joohoi commented May 13, 2016

@pde letsencrypt-external is currently tied to tls-sni-01, and providing just an external way to deal with the validation for it. As I see this task, we should take this to one step lower, ie: providing an option to use every challenge currently available, and working from there.

It also currently lacks all tests, so the question goes to @marcan : Would you like to work on your plugin to implement tests, and make it work with all the challenges available, the aim would be inclusion to the default installation.

@pde
Member
pde commented May 13, 2016

We ended up with fancy script hooks implemented for renewal (certbot --help renew), so the facilities in hooks.py should be extensible to assist this plugin too.

@marcan
marcan commented May 13, 2016

I'm interested in improving the plugin and adding support for the other challenges, though I don't know how much time I'll be able to invest in that for the next couple of months; I have lots of things coming up. But one of them is a talk about Let's Encrypt so I may be able to use that as an excuse to spend some time on it :).

@joohoi
Member
joohoi commented May 15, 2016 edited

Taken @marcan time constraints, need to support all challenge types and the fact that we already have a hook implementation in place to reuse (ie. most, if not all of the code would need to be rewritten anyway), I think I'll start from the scratch.

@bmw
Contributor
bmw commented Sep 28, 2016

I think we should consider what @jsha wrote here.

@ph4r05
ph4r05 commented Oct 26, 2016 edited

Just in case anybody is interested. I modified manual.py plugin so it produces JSON-only stdout, e.g., challenges to solve. After challenge is solved, caller sends '\n' to stdin so certbot moves forward. It can be used with pipes when caller solves the challenges. It works with DNS, HTTP and TLS-SNI challenges.

Checkout the readme:
https://github.com/EnigmaBridge/certbot-external-auth

I use it for DNS-01 domain validation in our project.

EDIT01: the plugin is also slightly based on letsencrypt-external
EDIT02: the plugin now supports also handler mode as @marcan implemented in letsencrypt-external
EDIT03: I implemented TLS-SNI to the manual.py plugin, so I can create a PR into the Certbot directly if you deem code OK.
EDIT04: DNS hooks are compatible with Dehydrated DNS Hook scripts - former letsencrypt.sh. Installer is supported too.

@joohoi
Member
joohoi commented Jan 21, 2017 edited

This was shipped with Certbot 0.10.0, functionality added to manual plugin to enable scripting the authentication and cleanup.

manual:
  Authenticate through manual configuration or custom shell scripts. When
  using shell scripts, an authenticator script must be provided. The
  environment variables available to this script are $CERTBOT_DOMAIN which
  contains the domain being authenticated, $CERTBOT_VALIDATION which is the
  validation string, and $CERTBOT_TOKEN which is the filename of the
  resource requested when performing an HTTP-01 challenge. An additional
  cleanup script can also be provided and can use the additional variable
  $CERTBOT_AUTH_OUTPUT which contains the stdout output from the auth
  script.

  --manual-auth-hook MANUAL_AUTH_HOOK
                        Path or command to execute for the authentication
                        script (default: None)
  --manual-cleanup-hook MANUAL_CLEANUP_HOOK
                        Path or command to execute for the cleanup script
                        (default: None)
  --manual-public-ip-logging-ok
                        Automatically allows public IP logging (default: Ask)
@joohoi joohoi closed this Jan 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment