Manual plugin "--shell-command" option

[brianmhunt]

Right now the --manual plugin requires user interaction, but there are some use cases where it needn't.

For example, when using --manual with Google App Engine, one can automate that entire process, but it's quite convoluted.

The process could be vastly simplified if, instead of using manual input, the --manual plugin took a shell command, that I would imagine could work as follows:

1. User calls letsencrypt --manual --shell-command "./upload-script --server=example.com" -d example.com -d www.example.com

2. Let's Encrypt calls upload-script, and pipes the challenges to the standard input, something like:

   example.com challenge-filename-1 challenge-1 
   www.example.com challenge-filename-2 challenge-2
   

   It would be trivial to generalize this with another argument like --shell-pipe-format with options such as csv, tsv, json, etc.

3. Once the shell command completes Let's Encrypt runs the challenge.

---

Example

In the case of Google App Engine, a simplistic upload-script might look like (and my bash is a little rusty, but I hope it conveys the point):

# Write the challenge files to a place where AppEngine will read/publish them
CHALLENGE_PATH = "app/acme/challenges"
while read line
do
  CHALLENGE=$line[2]
  CHALLENGE_FILE="$CHALLENGE_PATH$line[1]"
  echo CHALLENGE > CHALLENGE_FILE
while done < "${1:-/dev/stdin//:/ }"

# Publish to AppEngine
appcfg.py update app

Once the script completes (and presuming that acme/challenges publishes to ./well-known/...), Let's Encrypt should validate successfully against the App Engine.

---

I find this nicely abstracts the validation process.

Compare this to the fragile monstrosities I have created to accomplish the same that carefully read and interpret the output of Let's Encrypt.

(In retrospect my time would have been better applied writing a patch for or alternative to --manual – sorry I missed that opportunity; I hope to return to it as time permits, but I wanted to take the time share this to see if there is interest in such a patch and in case someone else might be able to pick it up)

[bmw]

I really like the idea of doing something like this. There has been talk of doing something similar with webroot to allow for people to include scripts/shell commands to send challenges to their server with rsync/sftp, but I think it makes more sense for this functionality to be included in the manual plugin.

[pde]

Hmmm, I should probably have closed #2782 as a dupe of this, but I will do it the other way round. @brianmhunt if you still like your proposed design in comparison to the others I added over there, can you please paste your proposal as a comment on #2782?

[jsha]

We have a couple of extensibility mechanisms already built in: plugins and --pre-hook / --post-hook. Adding this introduces a third extensibility mechanism. Since each of these mechanisms will be integrated into people's automated systems, we'll have to maintain them more or less forever, so we should be cautious about adding new ones.

My main question: Why isn't the existing plugin mechanism good enough to accomplish the goal? I think the current way of writing a plugin is probably too cumbersome for the purpose, but that probably means we should simplify the process of writing small plugins. For instance, it would be great if I could put a file in /etc/letsencrypt/plugins/rsync.py with contents:

def pre(domain, filename):
  subprocess.check_output("rsync %s %s:/var/www/.well-known/acme-challenges" % (filename, domain)

def post(domain, filename):
  subprocess.check_output("ssh %s rm /var/www/.well-known/acme-challenges/%s" % (domain, filename)

The syntax is still a little clunkier than shell scripts, but if we can make plugin writing this easy, then we wind up with only a single mechanism to maintain for the long term, and we also make general-purpose integration with Certbot easier.

If we go with shell scripts as an alternate integration mechanism, I think users will be very confused about the difference between /etc/letsencrypt/script.d/ and --pre-hook and --post-hook. We should include script types that reproduce the functionality of --pre-hook and --post-hook, and deprecate those flag, in order to reduce confusion about which method to use.

[bmw]

I think this is definitely worth considering. If we can simplify the process of writing plugins, not only can sysadmins easily create their own for their custom setups, but it would encourage more 3rd party plugin development.

[bmw]

Also just noticed the correct issue for this is #2782. I'll post a comment over there linking to this discussion.