Implement a "script" plugin and/or fix manual for scriptability

[augnustin]

I guess I'm not the first one to raise the issue, but I haven't found anything yet by searching.

I'm trying to automate VM deployment with a letsencrypt certificate. So I have tried to remove user interaction as much as possible and reached:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos

But if I add the -n option, I get:

PluginError('Running manual mode non-interactively is not supported',)

I'm guessing the point is that manual mode does require user interaction hence cannot be fully done in non-interactive mode. At least during the file-validation step.

Which is why I'd like to discuss splitting the command in two steps:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos --generate-file /home/user

which would create the file /home/user/.well-known/acme-challenge/buCgmexKDQypJfWE4-XmUfP1cpILzsBR4UWRE_o8iuc with the expected content and then running

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos --validate

would finish the process.

Would this make sense? Did I miss something?

Another option would be to build allow a script to be executed between the file generation step and the verification step:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m contact@example.com -d example.com -t --manual-public-ip-logging-ok --agree-tos --deploy-verification ./deploy-verification.sh

[bmw]

This certainly does make sense. #2610 also suggests the script approach, but there are benefits to having two different client invocations as well. We certainly want to implement something like this in the future.

[pde]

@joohoi is actually working on this, in part to make support for DNS challenges practical. I don't think he's decided on a a design yet, though some of the options under discussion include:

• Run a "Step 1" command that gives you a challenge, and then a "Step 2" command when you've deployed it. Challenge: how to serialise and save state so that the client knows where to pick up from.
• Call the client with two hook functions, one of which is to be run when a challenge is ready, the other when it has succeeded. Challenge: does this mean that the user has to write three separate scripts, one for each hook and one to call the client with the hooks?

Another option that @bmw and I talked about recently is:

• Implement hooks, but encourage the user write a fancy script that the client can itself invoke, something like this:

#!/bin/bash

DeployChallenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # do stuff
}

CleanupChallenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # undo stuff
}

if [ -z "$WITHIN_LE_CLIENT" ] ; then
   letsencrypt certonly --script --script-challenge HTTP01 --challenge-hook ". $0 ; DeployChallenge"  --cleanup-hook ". $0 ; CleanupChallenge"
   # deploy cert
fi

That's not terrible, but it has the problems that the client invocation is a bit long and weird. And would even need extra stuff ($0 -> \"$0\") to work if the script had a space in its path. Which leads to:

• Do the same thing as above, but bake in the bash function names and add some syntactic sugar. So the script would look like:

#!/bin/bash

DeployHTTP01Challenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # do stuff
}

CleanupHTTP01Challenge() {
   CHALLENGE_URL=$1
   CHALLENGE_CONTENT=$2
   # undo stuff
}

if [ -z "$WITHIN_LE_CLIENT" ] ; then
   letsencrypt certonly --script --script-challenge HTTP01 --script-hooks "$0"
   # deploy cert
fi

[bmw]

The original post in #2610 should definitely be read for additional context as it was @brianmhunt and that post that gave us the idea for something like this in the first place.

pde and I have been thinking about this issue a lot over the past couple of days and I wanted to provide my thoughts.

For starters, I don't think doing this should involve two client invocations. That is first suggestion that pde posted. Doing this feels clunky, would require some major changes in the client, and I can't think of anything you could do in this approach that you couldn't do with passing commands/scripts to the client.

Next, between the options of passing one or two scripts to the client, I think I like the option of two scripts better. If we put everything in one large script, we have to have a LE defined way of separating out the deploy and cleanup code (such as the bash functions with specific names). This seems more complicated and less intuitive than just having separate deploy/cleanup commands/scripts.

Furthermore, as suggested by pde, having the client take two commands still provides a way to write a single script for the manual plugin that can be easily shared with others. The large script can define bash functions, create temporary files, etc.

Lastly, I'd like to suggest that we don't provide multiple ways to do this, unless there is a real benefit. We could provide something like --manual-deploy-command, --manual-cleanup-command, and --manual-combined-command, but by doing this we further complicate our command line and I'm unaware of anything you could do with one approach that you couldn't do with the other.

If anyone else has thoughts on this, I'd love to hear them. If this is done right, I think it could be a huge benefit to our users with more complex server setups.

[augnustin]

I'm glad to read that I'm not the only one having this need. 😄

@brianmhunt I'll have a look at your blog post to find an intermediary solution before this issue gets completed.

I'm not fully aware of how the client works, but as far as I'm concerned, having a script as an option of a CLI doesn't seem like a common pattern. It means the user has to create an external file to set its automated instructions. If I were to tutorialize the two options, steps would be:

1. Deploy a file to the correct file
2. Create a script with the above steps and make it executable
3. Run letsencrypt certonly --deploy-script ./deploy-script.sh

This sounds more complicated than having a tutorial that says:

1. Run letsencrypt certonly ...
2. Go set your file in the server (which could be done by a script or manually depending on user prefrence)
3. Run letsencrypt validate

Being a Rubyist, the first pattern felt like a decent solution, but from a shell perspective, I think I like the second one better.

But if it is not possible, and if you consider that anyways, if we are there, it is that we want to fully automate then both shall work.

[brianmhunt]

Thanks everyone for the conversation and giving life to this issue! :) I'm cool with any proposal, so long as it gives end-to-end automation.

As @pde suggested, I'm putting the gist of my issue from #2610 below – not because I favour my method to the others (I don't have a solid enough grasp of the others to really think of all the edge cases), but just for ease of reference.

Cheers!

---

Right now the --manual plugin requires user interaction, but there are some use cases where it needn't.

For example, when using --manual with Google App Engine, one can automate that entire process, but it's quite convoluted.

The process could be vastly simplified if, instead of using manual input, the --manual plugin took a shell command, that I would imagine could work as follows:

1. User calls letsencrypt --manual --shell-command "./upload-script --server=example.com" -d example.com -d www.example.com
2. Let's Encrypt calls upload-script, and pipes the challenges to the standard input, something like:

example.com challenge-filename-1 challenge-1 
www.example.com challenge-filename-2 challenge-2

It would be trivial to generalize this with another argument like --shell-pipe-format with options such as csv, tsv, json, etc.

1. Once the shell command completes Let's Encrypt runs the challenge.

Example

In the case of Google App Engine, a simplistic upload-script might look like (and my bash is a little rusty, but I hope it conveys the point):

# Write the challenge files to a place where AppEngine will read/publish them
CHALLENGE_PATH = "app/acme/challenges"
while read line
do
  CHALLENGE=$line[2]
  CHALLENGE_FILE="$CHALLENGE_PATH$line[1]"
  echo CHALLENGE > CHALLENGE_FILE
while done < "${1:-/dev/stdin//:/ }"

# Publish to AppEngine
appcfg.py update app

Once the script completes (and presuming that acme/challenges publishes to ./well-known/...), Let's Encrypt should validate successfully against the App Engine.

I find this nicely abstracts the validation process.

Compare this to the fragile monstrosities I have created to accomplish the same that carefully read and interpret the output of Let's Encrypt.

[joohoi]

Thanks for a good discussion everyone!

@pde @bmw I think it could be beneficial for the users, if we'd implement the hooking method that lukas2511/letsencrypt.sh uses. It's basically using one bash script with predefined function names for each needed scripted action. I think it's well thought, and would allow users to write interoperable scripts, and use which ever client they want. We'd provide the path to script on command line.

Example hook script here: hook.sh.example

We could easily provide example scripts for the most common operations as well. That would make the learning curve for users pretty much nonexistent.

[kuba]

One link: https://github.com/marcan/letsencrypt-external

[brianmhunt]

@kuba While letsencrypt-external will solve the issue for some scenarios, it comes with the constraint about servers of "as long as [the server validating] shares the same external IP address [as the server that will user the certificate]." That constraint can never be satisfied with providers such as Google AppEngine.

[augnustin]

@kuba I tried to install Letsencrypt-external but installation failed.

Here's the issue: marcan/certbot-external#1

[augnustin]

I finally made it through using the --webroot option:

sudo /opt/letsencrypt/letsencrypt-auto certonly --webroot -w /vagrant/www/current/public -d example.com -m contact@example.com -t --manual-public-ip-logging-ok --agree-tos

Using nginx, here's how I configured it:

upstream rails_app {
  server                localhost:3000;
}

server {

  listen 443 ssl;
  server_name           example.com;

  ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

  location / {
    proxy_pass          http://rails_app;
    proxy_read_timeout  90;
  }

}

server {
  listen                80;
  server_name           example.com;

  # Necessary for Let's Encrypt Domain Name ownership validation
  location /.well-known/acme-challenge/ {
    root /vagrant/www/current/public/.well-known/acme-challenge/;
  }
  return 301 https://$host$request_uri;
}

The location /.well-known/acme-challenge/ block makes the folder publicly accessible, hence validation is made automatically.