Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a "script" plugin and/or fix manual for scriptability #2782

augnustin opened this issue Apr 7, 2016 · 20 comments

Implement a "script" plugin and/or fix manual for scriptability #2782

augnustin opened this issue Apr 7, 2016 · 20 comments


Copy link

I guess I'm not the first one to raise the issue, but I haven't found anything yet by searching.

I'm trying to automate VM deployment with a letsencrypt certificate. So I have tried to remove user interaction as much as possible and reached:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m -d -t --manual-public-ip-logging-ok --agree-tos

But if I add the -n option, I get:

PluginError('Running manual mode non-interactively is not supported',)

I'm guessing the point is that manual mode does require user interaction hence cannot be fully done in non-interactive mode. At least during the file-validation step.

Which is why I'd like to discuss splitting the command in two steps:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m -d -t --manual-public-ip-logging-ok --agree-tos --generate-file /home/user

which would create the file /home/user/.well-known/acme-challenge/buCgmexKDQypJfWE4-XmUfP1cpILzsBR4UWRE_o8iuc with the expected content and then running

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m -d -t --manual-public-ip-logging-ok --agree-tos --validate

would finish the process.

Would this make sense? Did I miss something?

Another option would be to build allow a script to be executed between the file generation step and the verification step:

sudo /opt/letsencrypt/letsencrypt-auto certonly --manual -m -d -t --manual-public-ip-logging-ok --agree-tos --deploy-verification ./
Copy link

bmw commented Apr 7, 2016

This certainly does make sense. #2610 also suggests the script approach, but there are benefits to having two different client invocations as well. We certainly want to implement something like this in the future.

@bmw bmw added this to the 1.0.0 milestone Apr 7, 2016
Copy link

pde commented Apr 7, 2016

@joohoi is actually working on this, in part to make support for DNS challenges practical. I don't think he's decided on a a design yet, though some of the options under discussion include:

  • Run a "Step 1" command that gives you a challenge, and then a "Step 2" command when you've deployed it. Challenge: how to serialise and save state so that the client knows where to pick up from.
  • Call the client with two hook functions, one of which is to be run when a challenge is ready, the other when it has succeeded. Challenge: does this mean that the user has to write three separate scripts, one for each hook and one to call the client with the hooks?

Another option that @bmw and I talked about recently is:

  • Implement hooks, but encourage the user write a fancy script that the client can itself invoke, something like this:

DeployChallenge() {
   # do stuff

CleanupChallenge() {
   # undo stuff

if [ -z "$WITHIN_LE_CLIENT" ] ; then
   letsencrypt certonly --script --script-challenge HTTP01 --challenge-hook ". $0 ; DeployChallenge"  --cleanup-hook ". $0 ; CleanupChallenge"
   # deploy cert

That's not terrible, but it has the problems that the client invocation is a bit long and weird. And would even need extra stuff ($0 -> \"$0\") to work if the script had a space in its path. Which leads to:

  • Do the same thing as above, but bake in the bash function names and add some syntactic sugar. So the script would look like:

DeployHTTP01Challenge() {
   # do stuff

CleanupHTTP01Challenge() {
   # undo stuff

if [ -z "$WITHIN_LE_CLIENT" ] ; then
   letsencrypt certonly --script --script-challenge HTTP01 --script-hooks "$0"
   # deploy cert

@pde pde changed the title Manual + Non Interactive mode Implement a "script" plugin and/or fix manual for scriptability Apr 7, 2016
Copy link

bmw commented Apr 8, 2016

The original post in #2610 should definitely be read for additional context as it was @brianmhunt and that post that gave us the idea for something like this in the first place.

pde and I have been thinking about this issue a lot over the past couple of days and I wanted to provide my thoughts.

For starters, I don't think doing this should involve two client invocations. That is first suggestion that pde posted. Doing this feels clunky, would require some major changes in the client, and I can't think of anything you could do in this approach that you couldn't do with passing commands/scripts to the client.

Next, between the options of passing one or two scripts to the client, I think I like the option of two scripts better. If we put everything in one large script, we have to have a LE defined way of separating out the deploy and cleanup code (such as the bash functions with specific names). This seems more complicated and less intuitive than just having separate deploy/cleanup commands/scripts.

Furthermore, as suggested by pde, having the client take two commands still provides a way to write a single script for the manual plugin that can be easily shared with others. The large script can define bash functions, create temporary files, etc.

Lastly, I'd like to suggest that we don't provide multiple ways to do this, unless there is a real benefit. We could provide something like --manual-deploy-command, --manual-cleanup-command, and --manual-combined-command, but by doing this we further complicate our command line and I'm unaware of anything you could do with one approach that you couldn't do with the other.

If anyone else has thoughts on this, I'd love to hear them. If this is done right, I think it could be a huge benefit to our users with more complex server setups.

Copy link

I'm glad to read that I'm not the only one having this need. 😄

@brianmhunt I'll have a look at your blog post to find an intermediary solution before this issue gets completed.

I'm not fully aware of how the client works, but as far as I'm concerned, having a script as an option of a CLI doesn't seem like a common pattern. It means the user has to create an external file to set its automated instructions. If I were to tutorialize the two options, steps would be:

  1. Deploy a file to the correct file
  2. Create a script with the above steps and make it executable
  3. Run letsencrypt certonly --deploy-script ./

This sounds more complicated than having a tutorial that says:

  1. Run letsencrypt certonly ...
  2. Go set your file in the server (which could be done by a script or manually depending on user prefrence)
  3. Run letsencrypt validate

Being a Rubyist, the first pattern felt like a decent solution, but from a shell perspective, I think I like the second one better.

But if it is not possible, and if you consider that anyways, if we are there, it is that we want to fully automate then both shall work.

Copy link

Thanks everyone for the conversation and giving life to this issue! :) I'm cool with any proposal, so long as it gives end-to-end automation.

As @pde suggested, I'm putting the gist of my issue from #2610 below – not because I favour my method to the others (I don't have a solid enough grasp of the others to really think of all the edge cases), but just for ease of reference.


Right now the --manual plugin requires user interaction, but there are some use cases where it needn't.

For example, when using --manual with Google App Engine, one can automate that entire process, but it's quite convoluted.

The process could be vastly simplified if, instead of using manual input, the --manual plugin took a shell command, that I would imagine could work as follows:

  1. User calls letsencrypt --manual --shell-command "./upload-script" -d -d
  2. Let's Encrypt calls upload-script, and pipes the challenges to the standard input, something like: challenge-filename-1 challenge-1 challenge-filename-2 challenge-2

It would be trivial to generalize this with another argument like --shell-pipe-format with options such as csv, tsv, json, etc.

  1. Once the shell command completes Let's Encrypt runs the challenge.


In the case of Google App Engine, a simplistic upload-script might look like (and my bash is a little rusty, but I hope it conveys the point):

# Write the challenge files to a place where AppEngine will read/publish them
CHALLENGE_PATH = "app/acme/challenges"
while read line
while done < "${1:-/dev/stdin//:/ }"

# Publish to AppEngine update app

Once the script completes (and presuming that acme/challenges publishes to ./well-known/...), Let's Encrypt should validate successfully against the App Engine.

I find this nicely abstracts the validation process.

Compare this to the fragile monstrosities I have created to accomplish the same that carefully read and interpret the output of Let's Encrypt.

Copy link

joohoi commented Apr 9, 2016

Thanks for a good discussion everyone!

@pde @bmw I think it could be beneficial for the users, if we'd implement the hooking method that lukas2511/ uses. It's basically using one bash script with predefined function names for each needed scripted action. I think it's well thought, and would allow users to write interoperable scripts, and use which ever client they want. We'd provide the path to script on command line.

Example hook script here:

We could easily provide example scripts for the most common operations as well. That would make the learning curve for users pretty much nonexistent.

Copy link

kuba commented Apr 10, 2016

One link:

Copy link

@kuba While letsencrypt-external will solve the issue for some scenarios, it comes with the constraint about servers of "as long as [the server validating] shares the same external IP address [as the server that will user the certificate]." That constraint can never be satisfied with providers such as Google AppEngine.

Copy link

@kuba I tried to install Letsencrypt-external but installation failed.

Here's the issue: marcan/certbot-external#1

Copy link

I finally made it through using the --webroot option:

sudo /opt/letsencrypt/letsencrypt-auto certonly --webroot -w /vagrant/www/current/public -d -m -t --manual-public-ip-logging-ok --agree-tos

Using nginx, here's how I configured it:

upstream rails_app {
  server                localhost:3000;

server {

  listen 443 ssl;
  server_name ;

  ssl_certificate /etc/letsencrypt/live/;
  ssl_certificate_key /etc/letsencrypt/live/;

  location / {
    proxy_pass          http://rails_app;
    proxy_read_timeout  90;


server {
  listen                80;
  server_name ;

  # Necessary for Let's Encrypt Domain Name ownership validation
  location /.well-known/acme-challenge/ {
    root /vagrant/www/current/public/.well-known/acme-challenge/;
  return 301 https://$host$request_uri;

The location /.well-known/acme-challenge/ block makes the folder publicly accessible, hence validation is made automatically.

Copy link

silbe commented Apr 29, 2016

FWIW, #2053 is also related. I'm currently using a small plug-in to invoke scripts that deploy the challenge to the actual HTTP servers. This is similar to the "hooks" approach.

The two invocations approach proposed by @aug-riedinger is interesting because it allows the steps to be completely decoupled. There is no need for the letsencrypt client to stay running until the challenges have been deployed to all servers.

Copy link

pde commented May 13, 2016

@marcan also appears to have been working on something similar:

(edit: didn't see that Kuba had already mentioned that)

Copy link

joohoi commented May 13, 2016

@pde letsencrypt-external is currently tied to tls-sni-01, and providing just an external way to deal with the validation for it. As I see this task, we should take this to one step lower, ie: providing an option to use every challenge currently available, and working from there.

It also currently lacks all tests, so the question goes to @marcan : Would you like to work on your plugin to implement tests, and make it work with all the challenges available, the aim would be inclusion to the default installation.

Copy link

pde commented May 13, 2016

We ended up with fancy script hooks implemented for renewal (certbot --help renew), so the facilities in should be extensible to assist this plugin too.

Copy link

marcan commented May 13, 2016

I'm interested in improving the plugin and adding support for the other challenges, though I don't know how much time I'll be able to invest in that for the next couple of months; I have lots of things coming up. But one of them is a talk about Let's Encrypt so I may be able to use that as an excuse to spend some time on it :).

Copy link

joohoi commented May 15, 2016

Taken @marcan time constraints, need to support all challenge types and the fact that we already have a hook implementation in place to reuse (ie. most, if not all of the code would need to be rewritten anyway), I think I'll start from the scratch.

Copy link

bmw commented Sep 28, 2016

I think we should consider what @jsha wrote here.

Copy link

ph4r05 commented Oct 26, 2016

Just in case anybody is interested. I modified plugin so it produces JSON-only stdout, e.g., challenges to solve. After challenge is solved, caller sends '\n' to stdin so certbot moves forward. It can be used with pipes when caller solves the challenges. It works with DNS, HTTP and TLS-SNI challenges.

Checkout the readme:

I use it for DNS-01 domain validation in our project.

EDIT01: the plugin is also slightly based on letsencrypt-external
EDIT02: the plugin now supports also handler mode as @marcan implemented in letsencrypt-external
EDIT03: I implemented TLS-SNI to the plugin, so I can create a PR into the Certbot directly if you deem code OK.
EDIT04: DNS hooks are compatible with Dehydrated DNS Hook scripts - former Installer is supported too.

Copy link

joohoi commented Jan 21, 2017

This was shipped with Certbot 0.10.0, functionality added to manual plugin to enable scripting the authentication and cleanup.

  Authenticate through manual configuration or custom shell scripts. When
  using shell scripts, an authenticator script must be provided. The
  environment variables available to this script are $CERTBOT_DOMAIN which
  contains the domain being authenticated, $CERTBOT_VALIDATION which is the
  validation string, and $CERTBOT_TOKEN which is the filename of the
  resource requested when performing an HTTP-01 challenge. An additional
  cleanup script can also be provided and can use the additional variable
  $CERTBOT_AUTH_OUTPUT which contains the stdout output from the auth

  --manual-auth-hook MANUAL_AUTH_HOOK
                        Path or command to execute for the authentication
                        script (default: None)
  --manual-cleanup-hook MANUAL_CLEANUP_HOOK
                        Path or command to execute for the cleanup script
                        (default: None)
                        Automatically allows public IP logging (default: Ask)

Copy link

angelnu commented Apr 15, 2017

I was able to use the --manual-cleanup-hook option to renew using the DNS method and GoDaddy.

I downloaded the goodaddy hook and put together the following command:
env GD_SECRET=<secret> GD_KEY=<key> certbot --manual --preferred-challenges dns --manual-public-ip-logging-ok --manual-auth-hook 'le-godaddy-dns/ deploy_challenge $CERTBOT_DOMAIN bla $CERTBOT_VALIDATION' renew --no-self-upgrade --quiet

I run the above with a cronjob every week

bmw added a commit that referenced this issue Dec 3, 2019
The old plugin at says it's obsolete and points people to The new plugin is also an installer.

I also removed the reference to #2782 about us adding similar functionality since that's been done for a long time. We could reference our manual plugin instead, but I think that devalues their plugin a bit which I don't think is necessary or correct as it has different features.
Tomoyuki-GH added a commit to Tomoyuki-GH/certbot that referenced this issue Apr 3, 2020
* Use dummy values for ancestor (#7462)

* [Apache v2] AugeasBlockNode find_comments() implementation (#7457)

* find_comments implementation and AugeasCommentNode creation

* Use dummy value for ancestor

* Add NotImplementedError when calling find_comments with exact parameter

* Remove parameter 'exact' from find_comments interface

* Fix comment

* Remove references to TLS-SNI-01 outside of ACME (#7479)

This is a big part of #7214. It removes all references to TLS-SNI-01 outside of acme (and pytest.ini). Those changes will come in a subsequent PR. I thought this one was getting big enough.

* Remove references to TLS-SNI-01 in Apache plugin

* Remove references to TLS-SNI-01 from certbot-nginx

* Remove references to TLS-SNI from Certbot.

* Remove TLS-SNI reference from docs

* add certbot changelog

* Clarify test behavior

* Fix invalid escape sequence \. (#7486)

Signed-off-by: Mickaël Schoentgen <>

* Don't use dev version of 3.8. (#7485)

Now that Python 3.8 is out, we don't need to use the development version.

* Don't use acme.test_util outside of acme. (#7484)

`certbot-compatibility-test` is using code in `acme` that I proposed making private and not trivially importable in

To fix it, I switched to using Certbot's test utilities which I proposed keeping public to help with writing tests for plugins. When doing this I had to change the name of the key because `rsa1024_key.pem` doesn't exist in Certbot.

I also deleted the keys in `certbot-compatibility-test`'s testdata because because they are unused.

* Use distro library for all OS version detection (#7467)

This pull request ensures that we use distro package in all the distribution version detection. It also replaces the custom systemd /etc/os-release parsing and adds a few version fingerprints to Apache override selection.

Fixes: #7405

* Revert "Try to use platform.linux_distribution() before distro equivalent (#7403)"

This reverts commit ca3077d0347aae12163a43bf74a0c8321284367e.

* Use distro for all os detection code

* Address review comments

* Add changelog entry

* Added tests

* Fix tests to return a consistent os name

* Do not crash on non-linux systems

* Minor fixes to distro compatibility checks

* Make the tests OS independent

* Update certbot/

Co-Authored-By: Brad Warren <>

* Skip linux specific tests on other platforms

* Test fixes

* Better test state handling

* Lower the coverage target for Windows tests

* Remove changelog entry about unpackaged scripts. (#7490)

We don't package so I don't think we need to mention changes to it in our changelog which is primarily read by users and packagers.

* Deprecate more code related to TLS-SNI-01 (#7483)

I tried to finish up #7214 by removing the code in acme but we can't really do that until #7478 is resolved which we cannot do until we release 0.40.0.

Since we have to wait, this PR adds deprecation warnings for code that uses the TLS-SNI-01 code or was only used by the long deprecated TLS-SNI-01 code.

I'd like this PR to land before our next release.

* Deprecate more code related to TLS-SNI-01.

* Assert about warning message.

* Describe distributed Certbot components. (#7493)

* Clarify when the changelog should be modified (#7491)

* Dropped deprecated flags from commands (#7482)

This pull request addresses #7451 by removing the deprecated flags.

* Dropped deprecated flags from commands

* Updated changelog for dropped flags and deleted outdated tests

* removed init-script part of apache test

* Use fresh authorizations in dry runs (#7442)

* acme: re-populate uri in deactivate_authorization

* Use fresh authorizations in dry runs

--dry-run now deactivates 'valid' authorizations if it encounters them
when creating a new order.

Resolves #5116.

* remove unused code

* typo in local-oldest-requirements

* better error handling

* certbot-ci: AUTHREUSE to 100 + unskip dry-run test

* improve test coverage for error cases

* restore newline to local-oldest-requirements.txt

* Build Windows installers with pinned dependencies (#7498)

* Consume constraints file

* Independent pywin32 dependency definition in and

* Don't use --agree-dev-preview in tests. (#7501)

* Update changelog for 0.40.0 release

* Release 0.40.0

* Add contents to for next version

* Bump version to 0.41.0

* Add back Python 3.4 support (#7510)

* Revert "Deprecation warnings for Python 3.4 (#7378)"

This reverts commit 6fcdfb0e5006be85500fad67a5a67b47befedb2a.

* Revert "Migrate certbot-auto users on CentOS 6 to Python 3.6 (#7268)"

This reverts commit e19b2e04c75b6df4e3f8a455700aa95fca79bcc3.

* add changelog entry

* keep mona in authors

* Add back Python 3.4 support (#7510) (#7511)

* Revert "Deprecation warnings for Python 3.4 (#7378)"

This reverts commit 6fcdfb0e5006be85500fad67a5a67b47befedb2a.

* Revert "Migrate certbot-auto users on CentOS 6 to Python 3.6 (#7268)"

This reverts commit e19b2e04c75b6df4e3f8a455700aa95fca79bcc3.

* add changelog entry

* keep mona in authors

(cherry picked from commit 9b848b1d65783000a13ef3f94ac5fe0e8c3879e7)

* Update changelog for 0.40.1 release

* Release 0.40.1

* Add contents to for next version

* Bump version to 1.0.0

* Pin all build dependencies for the Windows installer (#7504)

This PR uses pipstrap to bootstrap the venv used to build Windows installers. This effectively pin all build dependencies, since pynsist is already installed through script.

* Use pipstrap

* Pin also NSIS version

* [Apache v2] Implement set_parameters() (#7461)

* find_comments implementation and AugeasCommentNode creation

* set_parameters implementation

* Change parameters to a property

* Remove parameters property setter

* More pythonic iteration handling

* Match our Travis logic in Azure. (#7514)

In Travis, the full test suite doesn't run on PRs for point release branches, just on commits for them. I think this behavior makes sense because what we actually want to test before a point release is the exact commit we want to release after any squashing/merging has been done. This PR modifies Azure to match this behavior.

After this PR lands, I need to update the tests required to pass on GitHub.

* Deprecate config_changes (#7469)

Closes #7454

* Deprecate config_changes

* Error on config_changes

* Fix tests for

* Fix CHANGELOG entry

* Remove remnants of config_changes

* Fix CHANGELOG and add removed functions

* dns-rfc2136: use TCP to query SOA records (#7503)

* Use tcp query on dns-rfc2136 plugin

To improve network robust; fixes #7502.

* Update

* Fix dns-rfc2136 test cases

* Add UDP fallback to dns-rfc2136

* Remove tls sni common (#7527)

* fixes #7478

* add changelog entry

* remove get_systemd_os_info (#7526)

Fixes #7500.

* Make uncomplicated modules private (#7528)

* Create _internal package for Certbot's non-public modules

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* update apache and nginx oldest requirements

* Keep the lock file as certbot.lock

* nginx oldest tests still need to rely on newer certbot

* python doesn't have good dependency resolution, so specify the transitive dependency

* update required minimum versions in nginx

* Move to _internal (#7531)

Part of #5775. Methodology similar to #7528, but slightly more manual.

* Move items in certbot/display to _internal (#7532)

* Move display/ to _internal/

* Move display/ to _internal/

* Move display/ to _internal/

* Create in _internal/display

* Move to _internal (#7530)

* Move to _internal

* missed a few certbot.effs in tests

* remove sublime autocompletion

* fix messy scripting

* [Apache v2] Adding nodes 1/3 : add_child_block() (#7497)

* Implement add_child_block()

* Add comments and example

* Check augas path inconsistencies in initialization

* Remove TLS-SNI objects in ACME (#7535)

* fixes #7214

* update changelog

* remove unused import

* Move items in certbot/plugins to _internal (#7533)

* Create and initialize _internal/plugins

* Move plugins/ to _internal/

* Move plugins/ to _internal/

* Move plugins/ to _internal/

* Move plugins/ to _internal/

* Move plugins/ to _internal/

* Move plugins/ to _internal/

* add missed internalization

* shorten line

* Update outdated init comment

* Move to _internal (#7534)

* Don't call core constants from nginx plugin

* Move to _internal/

* Move ENHANCEMENTS from now-internal constants to public plugins.enhancements

* Update display.enhancements.ask from its 2015 comment

* fix docstring

* Remove python2 and certbot-auto references in how to set up a Certbot build environment. (#7549)

Fixes #7548.

This PR udpdates installation instructions to get rid of python2 and certbot-auto in the how-to explaining the Certbot development environment setup.

Instead, Python 3 is used, and appropriate instructions for APT and RPM based distributions are provided.

* Implement add_child_directive (#7517)

* [Apache v2] Implement save() and unsaved_files() (#7520)

* Implement save() and unsaved_files()

* Linter fix

* [Apache v2] Implement delete_child() (#7521)

* Implement delete_child

* Fix linter

* [Windows] Fix certbot renew task failure under NT AUTHORITY\SYSTEM account (#7536)

Turned out that the scheduled task that runs `certbot renew` twice a day, is failing. Without any kind of log of course, otherwise it would not be fun.

It can be revealed by opening a powershell under the `NT AUTHORITY\SYSTEM` account, under which the scheduled task is run. Under theses circumstances, the bug is revealed: Certbot breaks when trying to invoke `certbot.compat.filesystem._get_current_user()`. Indeed the logic there implied to call `win32api.GetUserNameEx(win32api.NameSamCompatible)` and this function does not return always a useful value.

For normal account, it will be typically `DOMAIN_OR_MACHINE_NAME\YOUR_USER_NAME` (e.g. `My Machine\Adrien Ferrand`). But for the account `NT AUTHORITY\SYSTEM`, it will return `MACHINE_NAME\DOMAIN$`, which is a nonsense and makes fail the resolution of the actual SID of the account at the end of `_get_current_user()`.

This PR fixes this behavior by using an explicit construction of the account name that works both for normal users and `SYSTEM`.

* Use a different way to resolve current user account, that works both for normal users and SYSTEM.

* Add a comment to run Certbot under NT AUTHORITY\SYSTEM

* [Windows] Avoid letsencrypt.log permissions error during scheduled certbot renew task (#7537)

While coding for #7536, I ran into another issue. It appears that Certbot logs generated during the scheduled task execution have wrong permissions that make them almost unusable: they do not have an owner, and their ACL contains nonsense values (non existant accounts name).

The class `logging.handler.RotatingFileHandler` is responsible for these logs, and become mad when it is in a Python process run under a scheduled task owned by `SYSTEM`. This is precisely our case here.

This PR avoids (but not fix) the issue, by changing the owner of the scheduled task from `SYSTEM` to the `Administrators` group, that appears to work fine.

* Use Administrators group instead of SYSTEM to run the certbot renew task

* Move to _internal (#7542)

Part of #5775. Methodology similar to #7528. Also refactors NGINX test util to use certbot.tests.util.ConfigTestCase.

* refactor nginx tests to no longer rely on certbot.configuration internals

* Move to _internal

* Deprecate certbot register --update-registration (#7556)

Closes #7452.

* Fix shebang in rebuild_deps (#7557)

When you try to run this script, it crashes with:
standard_init_linux.go:211: exec user process caused "exec format error"
This is caused by the script being written to have the contents:
set -e
This fixes the problem by removing the slash and moving the shebang to the first line of the string.

* Internalize modules called by internal plugins (#7543)

* Move to _internal

* Move to _internal

* Update pinned dependencies (#7558)

Fixes #7184.

I updated #7358 to track the issue of unpinning all of these dependencies.

* pin back configargparse

* Pin back zope packages.

* update deps

* Add changelog entry.

* run

* fixes #7553 (#7560)

* [Apache v2] Initial ApacheParser skeleton (#7559)

* Fix metadata & primary references in Augeas tests.

When performing actions only on one of the trees in DualNodeParser, the two
trees get out-of-sync. Similarly, we can't expect that the metadata between
the two trees will remain the same.

Did a pass over the tests to re-wire metadata and primary usage.

* Add ApacheParser skeleton.

Fix plumbing in configurator & dualparser to initialize ApacheParser
alongside AugeasParser.

* Silence coverage reports for now

* Implement add_child_comment (#7518)

* Remove unused apache docs (#7575)

Part of #5775. We don't use these docs anywhere, so delete them.

- `certbot-apache/`
- `certbot-apache/docs/` folder
- docs include in ``
- docs dependencies in ``

* Remove unused certbot-compatibility-test docs (#7577)

Part of #5775. We don't use these docs anywhere, so delete them.

- `certbot-compatibility-test/`
- `certbot-compatibility-test/docs/` folder
- docs include in ``
- docs dependencies in ``

* Remove DNS plugin API docs. (#7578)

Replace DNS plugins' API documentation with a note that plugins adhere to certbot's plugin interface.

* remove unused route53 tools (#7586)

* Remove unused nginx docs (#7576)

Part of #5775. We don't use these docs anywhere, so delete them.

- `certbot-nginx/`
- `certbot-nginx/docs/` folder
- docs include in ``
- docs dependencies in ``

* Remove unused nginx docs

* Add changelog entry about the removal

* Make the contents of the apache plugin private (#7579)

Part of #5775.

├── _internal
│   ├──
│   ├── augeas_lens
│   │   ├── httpd.aug
│   │   └── README
│   ├── centos-options-ssl-apache.conf
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   ├── options-ssl-apache.conf
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   ├──
│   └──
└── tests
    ├── ...

* Create _internal folder for certbot_apache

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move augeas_lens to _internal

* Move options-ssl-apache.conf files to _internal

* move augeas_lens in MANIFEST

* Clean up some stray references to certbot_apache that could use _internal

* Correct imports and lint

* Make the contents of the DNS plugins private (#7580)

Part of #5775.

modify_item () {
    mkdir certbot-dns-$1/certbot_dns_$1/_internal
    git grep -l "from certbot_dns_$1 import dns_$1" | xargs sed -i "s/from certbot_dns_$1 import dns_$1/from certbot_dns_$1._internal import dns_$1/g"
    git grep -l "certbot_dns_$1\.dns_$1" | xargs sed -i "s/certbot_dns_$1\.dns_$1/certbot_dns_$1._internal.dns_$1/g"
    git checkout -- certbot-dns-$1/certbot_dns_$1/
    echo '"""Internal implementation of \`~certbot_dns_$1.dns_$1\` plugin."""' > certbot-dns-$1/certbot_dns_$1/_internal/
    mv certbot-dns-$1/certbot_dns_$1/dns_$ certbot-dns-$1/certbot_dns_$1/_internal
    git checkout --
    git status
    git add -A
    git commit -m "Move certbot-dns-$1 to _internal structure"

Structure now looks like this:
├── certbot_dns_cloudflare
│   ├──
│   ├──
│   └── _internal
│       ├──
│       └──

* Move certbot-dns-cloudflare to _internal structure

* Move certbot-dns-cloudxns to _internal structure

* Move certbot-dns-digitalocean to _internal structure

* Move certbot-dns-dnsimple to _internal structure

* Move certbot-dns-dnsmadeeasy to _internal structure

* Move certbot-dns-gehirn to _internal structure

* Move certbot-dns-google to _internal structure

* Move certbot-dns-linode to _internal structure

* Move certbot-dns-luadns to _internal structure

* Move certbot-dns-nsone to _internal structure

* Move certbot-dns-ovh to _internal structure

* Move certbot-dns-rfc2136 to _internal structure

* Move certbot-dns-sakuracloud to _internal structure

* Init file comments need to be comments

* Move certbot-dns-route53 to _internal structure

* Fix comment in route53 init

* Refactor certbot/ and certbot/tests/ to use the same structure as the other packages (#7544)

Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.

New directory structure summary:
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│   ├──
│   ├── certbot
│   │   ├──
│   │   ├──
│   │   ├── _internal
│   │   │   ├──
│   │   │   ├──
│   │   │   ├── ...
│   │   ├── ...
│   ├── tests
│   │   ├──
│   │   ├── display
│   │   │   ├──
│   │   │   ├── ...
│   │   ├── ... # note no at this level
│   ├── ...
├── acme
│   ├── ...
├── certbot-apache
│   ├── ...
├── ...

* refactor certbot/ and certbot/tests/ to use the same structure as the other packages

* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"

* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"

* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"

* Remove replacement of certbot into . in

* copy license back out to main folder

* remove and from certbot/ because these files are not under certbot/

* Move README back into main folder, and make the version inside certbot/ a symlink

* symlink certbot READMEs the other way around

* move testdata into the public api certbot zone

* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests

* certbot version has been bumped down a directory level

* make certbot tests directory not a package and import sibling as module

* Remove unused script cruft

* change . to certbot in test_sdists

* remove outdated comment referencing a command that doesn't work

* Install instructions should reference an existing file

* update file paths in Dockerfile

* some package named in tox.ini were manually specified, change those to certbot

* new directory format doesn't work easily with pyargs according to

* remove other instance of pyargs

* fix up some references in by searching for ' . ' and manual check

* another stray . in tox.ini

* fix paths in tools/

* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper

* Create public shim around main and make that the entry point

* without pyargs, tests cannot be run from an empty directory

* Remove cruft for running certbot directly from main

* Have main shim take real arg

* add docs/api file for main, and fix up main comment

* Update certbot/docs/install.rst

Co-Authored-By: Brad Warren <>

* Fix comments in readthedocs requirements files to refer to current package

* Update .[docs] reference in contributing.rst

* Move plugins tests to certbot tests directory

* add certbot tests to so packagers can run python test

* move examples directory inside certbot/

* Move CHANGELOG into certbot, and create a top-level symlink

* Remove unused sys and logging from main shim

* nginx http01 test no longer relies on certbot plugins common test

* Make the contents of the nginx plugin private (#7589)

Part of #5775.

* Create _internal folder certbot-nginx

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Move to _internal

* Update location and references for tls_configs

* exclude parser_obj from coverage

* Update (#7596)

* Update

* Remove line breaks

Github seems to be keeping the line breaks rather than ignoring them, making it be formatted weirdly, so remove them.

* Fix refactor (#7597)

Clean up some places missed by #7544.

Found this when running test farm tests. They were working as of 5d90544, and I will truly shocked if subsequent changes (all to the windows installer) made them stop working.

* Release script needs to target new CHANGELOG location

* Clean up various other CHANGELOG path references

* Update windows paths for new certbot location

* Add certbot to packages list for windows installer

* Implement redirect by default (#7595)

* Change redirect default to yes so that it happens automatically in noninteractive mode

* Update changelog

* Refactor tests out of packaged module for dns plugins (#7599)

* Refactor tests out of module for certbot-dns-cloudflare

* Refactor tests out of module for certbot-dns-cloudxns

* Refactor tests out of module for certbot-dns-digitalocean

* Refactor tests out of module for certbot-dns-dnsimple

* Refactor tests out of module for certbot-dns-dnsmadeeasy

* Refactor tests out of module for certbot-dns-gehirn

* Refactor tests out of module for certbot-dns-google

* Refactor tests out of module for certbot-dns-linode

* Refactor tests out of module for certbot-dns-luadns

* Refactor tests out of module for certbot-dns-nsone

* Refactor tests out of module for certbot-dns-ovh

* Refactor tests out of module for certbot-dns-rfc2136

* Refactor tests out of module for certbot-dns-sakuracloud

* Refactor tests out of module for certbot-dns-route53

* Move certbot-dns-google testdata/ under tests/

* Use pytest for dns plugins

* Exclude pycache and .py[cod]

* Refactor tests out of packaged module for acme plugin (#7600)

* Move acme tests to tests/ directory outside of acme module

* Fix call to messages_test in client_test

* Move and testdata/ into tests/

* Update manifest to package tests

* Exclude pycache and .py[cod]

* Exclude pycache and .py[cod] from certbot package (#7608)

* Refactor tests out of packaged module for nginx plugin (#7606)

* Refactor tests out of packaged module for nginx plugin

* Exclude pycache and .py[cod]

* Refactor tests out of packaged module for apache plugin (#7607)

Part of #7593.

* Refactor tests out of packaged module for apache plugin

* Exclude pycache and .py[cod]

* Change tests path in tox.ini

* Defines the RenewableCert API (#7603)

This is my proposed fix for #7540. I would ideally like this to be included in our 1.0 release.

I came up with this design by adding all attributes used either in our own plugins, 3rd party plugins listed at, or our public API code.

Despite me thinking that zope is unneeded nowadays, I initially tried to use it to define this interface since we have it and it gives us a way to define expected attributes, but it doesn't work because zope interface objects also have a method called `names` which conflict with the API.

I talked about this with Adrien out of band and did some of my own research and there are some minor benefits with this new approach of using properties:

1. It's more conventional.
2. If you also change the implementation to inherit from the class, Python will error if all properties aren't defined.
3. The PEP 526 style type annotations with mypy seem to (currently) only be used to validate code using the class, not the class implementation itself. You can add a type annotation saying the class needs to have this attribute, never define it, and mypy won't complain.

With this new approach, I had to fix `names` because pylint was complaining that the arguments differed, however, we never used the optional parameter to `names` outside of tests so I just deleted the code altogether.

* fixes #7540

* move to properties

* [Apache v2] Implement find_ancestors (#7561)

* Implement find_ancestors

* Create the node properly and add assertions

* Update certbot-apache/certbot_apache/

Co-Authored-By: ohemorange <>

* Remove comment

* Upgrade to pywin32>=227 (#7615)

Current version of pywin32 used in certbot (225) does not have wheels available for Python 3.8. Installing certbot for development in this case requires to build from source. On Windows, this implies a Visual Studio C++ environment up and ready, which is absolutely not fun.

Let's upgrade to pywin32 227, that provides these wheels for all Python versions from 3.5 up to current dev status of 3.9.

* [Apache v2] Move the apachectl parsing to apache_util (#7569)

* Move the Apache CLI parsing to apache_util

* Fix test mocks

* Address review comments

* Fix the parsernode metadata dictionary

* acme/ comment refers to "PyOpenSSL" not "mock" (#7619)

* Update changelog for 1.0.0 release

* Release 1.0.0

* Add contents to certbot/ for next version

* Bump version to 1.1.0

* document main (#7610)

I deleted the exceptions because I think it's not feasible to document the possible exceptions raised by all of Certbot.

* update external plugin (#7604)

The old plugin at says it's obsolete and points people to The new plugin is also an installer.

I also removed the reference to #2782 about us adding similar functionality since that's been done for a long time. We could reference our manual plugin instead, but I think that devalues their plugin a bit which I don't think is necessary or correct as it has different features.

* Add full API documentation (#7614)

A lot of Certbot's files don't have API documentation which is fixed by this PR. To do this, from the top level certbot directory I ran:
sphinx-apidoc -Me -o docs/api certbot
I then merged the resulting `modules.rst` file with `docs/api.rst`.

* fix bad links in docs (#7623)

This PR fixes the failures at

Once this PR lands, I'll update certbot/website#508 to include this commit.

* Don't list DNS plugins as alpha quality. (#7624)

They should be considered production quality like our other packaged code.

* Don't list adding type annotations as a PR req. (#7627)

* Reorganize imports (#7616)

* Isort execution

* Fix pylint, adapt coverage

* New isort

* Fix magic_typing lint

* Second round

* Fix pylint

* Third round. Store isort configuration

* Fix latest mistakes

* Other fixes

* Add newline

* Fix lint errors

* [Apache v2] Implement parsed_files (#7562)

* Implement parsed_files

* Add parsed_files stub to ApacheParserNodes and fix assertions

* Update certbot-apache/certbot_apache/

Co-Authored-By: ohemorange <>

* Add more descriptive comments

* Update certbot-apache/certbot_apache/

Co-Authored-By: ohemorange <>

* Update certbot-apache/certbot_apache/

Co-Authored-By: ohemorange <>

* Update certbot-apache/certbot_apache/

Co-Authored-By: ohemorange <>

* Lint certbot code on Python 3, and update Pylint to the latest version (#7551)

Part of #7550

This PR makes appropriate corrections to run pylint on Python 3.

Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.

Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.

Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.

Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.

That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.

I also made PyLint and MyPy checks run correctly on Windows.

* Start configuration

* Reconfigure travis

* Suspend a check specific to python 3. Start fixing code.

* Repair call_args

* Fix return + elif lints

* Reconfigure development to run mainly on python3

* Remove incompatible Python 3.4 jobs

* Suspend pylint in some assertions

* Remove pylint in dev

* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path

* Various return + else lint errors

* Find a set of deps that is working with current mypy version

* Update local oldest requirements

* Remove all current pylint errors

* Rebuild letsencrypt-auto

* Update mypy to fix pylint with new astroid version, and fix mypy issues

* Explain type: ignore

* Reconfigure tox, fix none path

* Simplify pinning

* Remove useless directive

* Remove debugging code

* Remove continue

* Update requirements

* Disable unsubscriptable-object check

* Disable one check, enabling two more

* Plug certbot dev version for oldest requirements

* Remove useless disable directives

* Remove useless no-member disable

* Remove no-else-* checks. Use elif in symetric branches.

* Add back assertion

* Add new line

* Remove unused pylint disable

* Remove other pylint disable

* Execute Windows installer integration tests on several Windows versions (#7641)

This PRs extends the installer tests on Azure Pipeline, in order to run the integration tests on a certbot instance installed with the Windows installer for several Windows versions, corresponding to the scope of supported versions on Certbot:
* Windows Server 2012 R2
* Windows Server 2016
* Windows Server 2019

One can see the result on:

* Try specific installer-build step

* Install Python manually

* Add tests on windows 2019

* discourage dns plugins (#7639)

* Remove warning about dev preview (#7640)

* Add docker-compose as a requirement of certbot-ci (#7120)

Fixes #7110 

This PR declares docker-compose as a requirement for certbot-ci. This way, a recent version of docker-compose is installed in the standard virtual environment set up by `tools/` and `tools/`, and so is available to pytest integration tests from `tox` or in the virtual environment enabled.

* Add docker-compose as a dev dependency and declares it in certbot-ci requirements

* Update docker-compose 1.25.0

* Remove other 3.8-dev references. (#7646)

* Implement get_virtual_hosts() for ParserNode interfaces (#7564)

* How to uninstall certbot-auto (#7648)

* Include header files for compilation. (#7650)

* Remove POST-as-GET fallback to GET (#6994)

* Update (#7659)

* Fix gating to ensure that no parsernode functionality is run unless explicitly requested (#7654)

* Modifications needed for merging to master

* [Apache v2] Add apacheconfig as a dependency (#7643)

* Add apacheconfig as a dependency.

* Change apacheconfig to a dev dependency

* Bump apacheconfig dep to 0.3.1

* Implement a sunset mechanism in certbot-auto for systems not supported anymore (#7587)

* Sunset mechanism

* Simplify code

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <>

* Update template

* Deprecate for all RHEL/CentOS 6 32bits flavors

* Add a wrapper to uname to do tests on fake 32 bits versions

* Replace all occurences

* Add some tests about sunset mechanism

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Brad Warren <>

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Brad Warren <>

* Various corrections

* Recreate script

* Update comment position

* Test also install only

* Fix docker

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Brad Warren <>

* What error command is doing here ?

* Fix permissions

* Rebuild script

* Add changelog

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <>

* Update changelog

* Trigger CI

* Handle old venv path

* Modify test

* Fix test error detection from subpaths

* Edit echo

* Use set -e

* Update letsencrypt-auto-source/letsencrypt-auto.template

Co-Authored-By: Brad Warren <>

* Corrections

Co-authored-by: Brad Warren <>

* improve help about supply selecting in delete command (#7673)

for #6625

* Do not document private members (#7675)

It looks like we're currently documenting functions that are marked private (prefixed with an underscore) such as I do not think we should do this because the functionality is private, should not be used, and including it in our docs just adds visual noise.

This PR stops us from documenting private code and fixes up `tools/` so we don't document it in future modules.

* Do not document private code.

* Don't document private members in the future.

* Fix certbot-auto regarding python 3.4 -> python 3.6 migration for CentOS 6 users (#7519)

* Revert "Add back Python 3.4 support (#7510)"

This reverts commit 9b848b1d65783000a13ef3f94ac5fe0e8c3879e7.

* Fix certbot-auto

* Use a more consistent way to enable rh-python36

* Avoid to call CompareVersions unecessarily

* Control rh-python36 exit code

* Fix travis config

* Remove vscode config

* Ignore vscode

* Fix merge conflicts regarding #7587 (#70)

* Add changelog entry

* Finish sentence

* Update certbot/

Co-Authored-By: Joona Hoikkala <>

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Joona Hoikkala <>

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Joona Hoikkala <>

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Joona Hoikkala <>

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Joona Hoikkala <>

* Update letsencrypt-auto-source/tests/

Co-Authored-By: Joona Hoikkala <>

* Update comments

* Improve warning message

* Update changelog

Co-authored-by: Joona Hoikkala <>

* Update changelog for 1.1.0 release

* Release 1.1.0

* Add contents to certbot/ for next version

* Bump version to 1.2.0

* Add missing directory field (#7687)

Fixes #7683.

* Add missing directory field to error message

* Added change to

* Do not list the name twice. (#7689)

* Cleanup disabled warnings list in pytest.ini. (#7690)

* Fix minimum certbot version in plugins (#7684)

Fixes the problem found at

* Don't run some tests multiple times. (#7685)

* Include added/deleted TXT record name in RFC 2136 debug log (#7696)

* Spelling and grammar fixes (#7695)

* Downgrade NSIS and upgrade Python (#7702)

* Add --allow-downgrade to chocolatey command.

* Upgrade tests to use Python 3.8.1.

* Drop Travis tests for Python 3.4 (#7394)

* Minor release script improvements (#7697)

* Do not use git diff.

* Add a warning on exit.

* Don't run Python 3.5 tests twice. (#7704)

* unpin macos (#7705)

* fixes #1948 -- MD5 on FIPS systems (#7708)

* use MD5 in non-security mode to get around FIPS issue

* update CHANGELOG

* add myself to AUTHORS

* ignore hashlib params

* Update documentation files to remove claiming support for Python 3.4 (#7395)

* Fix imports for Python 3.9 (#7707)

* Fix imports for Python 3.9

* Update

* No longer ignore deprecation warning

* Update changelog

* Remove outdated comment

* Disabling no-name-in-module not needed as linting is on Python 3

* Remove ECDHE-RSA-AES128-SHA from NGINX ciphers list (#7719)

As mentioned in, it's time to remove this ciphersuite now that Windows 2008 R2 and Windows 7 are EOLed.

* Remove ECDHE-RSA-AES128-SHA from NGINX ciphers list to celebrate Windows 2008 R2 deprecation

* Update changelog

* Drop Python 3.4 support (#7721)

Fixes #7393.

* Remove Python 3.4 classifiers

* Remove unneeded typing dependency

* Exclude Python 3.4 in python_requires

* Remove Python 3.4 deprecation warning

* update changelog

* Disable old SSL versions and ciphersuites to follow Mozilla recommendations in Apache (#7712)

Part of #7204.

Makes the smaller changes described at to disable many old ciphersuites and TLS versions < 1.2. Does not add checks for OpenSSL version or modify session tickets.

Since Apache uses TLS protocol blacklisting instead of whitelisting (as in NGINX), we additionally may not need to determine if the server supports TLS1.3 and turn it on or off based on Apache version.

* Update SSL versions and ciphersuites based on Mozilla intermediate recommendations for apache

* Update constants with hashes of new config files

* Update changelog

* Unpin Python 3.4 dependencies (#7709)

* Unpin dependencies pinned back for py3.4 support.

* update pinned packages

* run

* Update boto3 and deps to work with requests

* Update dns-lexicon version. (#7723)

* dns-cloudflare: Implement limited-scope API Tokens (#7583)

A while ago Cloudflare added support for limited-scope API Tokens in place of using a global API key, but support for them in cloudflare/python-cloudflare took a while to get through.

In summary, this PR:
- Implements token functionality through the INI file parameter `dns_cloudflare_api_token` (in addition to the traditional `dns_cloudflare_email` and `dns_cloudflare_api_key`). This needed a more advanced parameter validator than the built in `required_variables` mechanism.
- Updates the docs to reflect the new option, needed token permissions, and version details of the `cloudflare` module

* Update python-cloudflare version

* Add Cloudflare API Token support to certbot-dns-cloudflare

* Add token-specific errors to certbot-dns-cloudflare

* Tidy up certbot-dns-cloudflare

* Implement Cloudflare API Tokens in testing for certbot-dns-cloudflare(needs work)

* Further tidying of certbot-dns-cloudflare

* Update CHANGELOG with Cloudflare API Tokens implementation

* Improve testing of certbot-dns-cloudflare

* Improve certbot-dns-cloudflare test formatting

* Further improve testing for certbot-dns-cloudflare

* Change needed permissions for token

* Add documentation regarding python-cloudflare version

* Fix changelog, references to python-cloudflare and docs

* Fix behaviour when domain does not match cloudflare root domain. Improve error handling.

* Improve testing

* Improve hints and error handling

* Add backwards compatibility docs (#7611)

Fixes #7463.

* Add backwards compatibility docs.

* Exclude certbot-auto

* Remove SSLCompression off line from all config options (#7726)

Based on discussion at

* Remove SSLCompression off line from all config options

* Update changelog

* Add space between words.

* Update instructions about how to build docs (#7605)

* Turn off Travis notifications in test branches. (#7733)

When I want to manually run the full test suite to test something, I've been manually deleting our notification setup from `.travis.yml` to avoid spamming IRC with my personal test failures.

This PR sets this behavior up to happen automatically by turning off IRC notifications in test branches. You can see this working by noticing the IRC notification section in the bottom of the config for this PR at and the fact that it is absent from a `test-` branch based on this one at

* Parse `$hostname` in `server_name`

* Add CHANGELOG entry

* Forgot to remove a `breakpoint()` statement

* Use unrestrictive umask for challenge directory

* Update changelog

* Update changelog for 1.2.0 release

* Release 1.2.0

* Add contents to certbot/ for next version

* Bump version to 1.3.0

* Wrap makedirs() within exception handelrs

* Missing import

* Windows installer integration tests (#7724)

As discussed in #7539, we need proper tests of the Windows installer itself in order to variety that all the logic contained in a production-grade runtime of Certbot on Windows is correctly setup by each version of the installer, and so for a variety of Windows OSes. 

This PR handles this requirement. The new `windows_installer_integration_tests` module in `certbot-ci` will:
* run the given Windows installer
* check that Certbot is properly installed and working
* check that the scheduled renew task is set up
* check that the scheduled task actually launch the Certbot renew logic

The Windows nightly tests are updated accordingly, in order to have the tests run on Windows Server 2012R2, 2016 and 2019.

These tests will evolve as we add more logic on the installer. 

* Configure an integration test testing the windows installer

* Write the test module

* Configurable installer path, prepare azure pipelines

* Fix option

* Update

* Add confirmation for this destructive test

* Use regex to validate certbot --version output

* Explicit dependency on a log output

* Use an exception to ask confirmation

* Use --allow-persistent-changes

* Set recreate = true in tox.ini. (#7746)

Fixes #7745.

* Add triggers for only a single CI system (#7748)

* Configure travis-test to only run on Travis.

* Configure azure-test to only run on Azure.

* Add docs and comments to keep it up-to-date.

* restore CHANGELOG in root directory

* Add test for $hostname parsing

* Remove todo::

* Fixing existing tests

* Don't display todo comments in docs (#7753)

Currently if you go to, there is a todo comment displayed at the top of the page. These todos were written for developers, not users, so I do not think they should be shown from our documentation.

This PR makes the quick and easy fix of configuring Sphinx not to show these todo items. I created #7752 to track removing all of these todos from our docstrings and disabling the Sphinx todo extension.

* Set todo_include_todos=False in sphinx-quickstart

* Remove todos from existing docs.

* Remove text that certbot.tests.utils isn't public (#7754)

* Remove link to letsencrypt readthedocs (#7757)

After a brief discussion in Mattermost, I shut down Turns out we were linking to it in our README here so let's remove the broken link.

I didn't update the link to point to one of the readthedocs projects we still have because are main Certbot docs are self-hosted rather than being on readthedocs.

* Really remove old docs link from README (#7758)

* Move to public api (#7744)

We should move to public API, as an upcoming OCSP prefetching functionality in Apache plugin relies on it, and as the plugins are note released in lockstep with the Certbot core, we need to be careful when changing those APIs.

* Move to public api

* Fix type annotations, move to pointing to an interface and fix linting

* Add certbot.ocsp to documentation table of contents

* Modify tests to reflect the changes in

* Add changelog entry

* Fix notAfter mock for tests

* Print script output in case of a failure. (#7759)

These tests failed at but do not include any output from the script about what went wrong because the string created from `subprocess.CalledProcessError` does not include value of output.

This PR fixes that by printing these values which `pytest` will include in the output if the test fails.

* Fix unpinned tests (#7760)

Our nightly tests failed last night due to a new release of `virtualenv` and `pip`'s lack of dependency resolution: It looks like we were not the only ones affected by this problem:

This fixes the problem by using `-I` to skip the logic where `pip` decides a dependency is already satisfied and has it reinstall/update the packages passed to `pip` and all of their dependencies.

You can see our nightly tests passing with this change at

* Remove duplicate pyparsing pin

* update pyparsing comment

* Remove _internal from docstring.

* Remove useless pylint error suppression directives (#7657)

As pylint is evolving, it improves its accuracy, and several pylint error suppression (`# pylint: disable=ERROR) added in certbot codebase months or years ago are not needed anymore to make it happy.

There is a (disabled by default) pylint error to detect the useless suppressions (pylint-ception: `useless-suppression`). It is not working perfectly (it has also false-positives ...) but it is a good start to clean the codebase.

This PR removes several of these useless suppressions as detected by the current pylint version we use.

* Remove useless suppress

* Remove useless lines

* more robustly stop patches (#7763)

* Remove letshelp-certbot (#7761)

* remove references to letshelp

* remove letshelp files

* Remove line continuation

Co-authored-by: ohemorange <>

* Fix spurious pylint errors. (#7780)

This fixes (part of) the problem identified in

When I tested our pylint setup on Python 3.5.9, 3.6.9, or 3.6.10, tests failed with:
************* Module acme.challenges
acme/acme/ E1101: Instance of 'UnrecognizedChallenge' has no 'jobj' member (no-member)
************* Module acme.jws
acme/acme/ E1101: Class 'Signature' has no '_orig_slots' member (no-member)
These errors did not occur for me on Python 3.6.7 or Python 3.7+.

You also cannot run our lint setup on Python 2.7 because our pinned version of pylint's dependency `asteroid` does not support Python 2. Because of this, `pylint` is not installed in the virtual environment created by `tools/` and our [`lint` environment in tox specifies that Python 3 should be used](

I tried updating pylint and its dependencies to fix the problem, but they still occur so I think adding back these disable checks on these lines again is the best fix for now.

* Correct AutoHSTS docs (#7767)

domains is a list of strings, not a single string.

* Correct AutoHSTS docs.

* Fix Apache enable_autohsts docs.

* add pgp key docs (#7765)

Fixes #7613.

* Move our macOS tests to Azure Pipelines (#7793)

[Our macOS tests are failing]( again this time due to the problem described at

I tried adding `update: true` to the Homebrew config as described in that thread, but [it didn't work]( I also tried updating the macOS image we use which [didn't work](

Since we continue to have problems with macOS on Travis, let try moving the tests to Azure Pipelines.

* test macos

* Remove Travis macOS setup

* add displayName

* Refactor, splitting in it smaller submodules (#6803)

* Refactor into a package with submodules

* Added unit tests for helpful module in cli.

* Fixed linter errors

* Fixed pylint issues

* Updated

* Fixed test failing and mypy error. Appeared a new pylint error (seems to be in conflict with mypy)

mypy require zope.interface to be imported but when imported it is not used and pylint throws an error.

* Fixed pylint errors

* Apply changes to cli since last merge from master (efc8d49806b14a31d88cfc0f1b6daca1dd373d8d)

* Fix lint

* Remaining lint errors

Co-authored-by: Adrien Ferrand <>

* Add test

* Use UTF-8 encoding for nginx plugin

* Use `io` module instead of `codecs`


* Added test for valid/invalid unicode characters

* Fix lint problems with long lines

* Relpace deprecated `logger.warn()` with `logger.warning()`

* Remove `unicode_support/` path in test case

* Add test case for `_parse_ssl_options()`

* Add logging test for `_parse_files()`

* Trivial code clean-up

* Add my name to


* Add this change to

* Add `TestCase.assertLogs()` backport for Python 2.7

* Add simple comments

* acme: ignore params in content-type check (#7342)

* acme: ignore params in content-type check

Fixes the warning in #7339

* Suppress coverage complaint in test


* Repair symlink

Co-authored-by: Adrien Ferrand <>

* Fix issue #7165 in _create_challenge_dirs(), attempt to fix pylint errors (#7568)

* fix issue #7165 by checking if directory exists before trying to create it, fix possible pylint issues in

* fix get_chall_pref definition

* Update

* Update

Co-authored-by: Adrien Ferrand <>

* remove _internal docs (#7801)

* Fixed typo & some trivial documentation change

* Update comment in testdata file

* Update parser test to better assert logging output

* Split advanced pipeline (#7813)

I want to do what I did in to our Azure Pipelines setup, but unfortunately this isn't currently possible. The only filters available for service hooks for the "build completed" trigger are the pipeline and build status. See 
![Screen Shot 2020-02-26 at 3 04 56 PM](

To accomplish this, I propose splitting the "advanced" pipeline into two cases. One is for builds on protected branches where we want to be notified if they fail while the other is just used to manually run tests on certain branches.

* update letstest reqs (#7809)

I don't fully understand why, but since I updated my macbook to macOS Catalina, the test script currently fails to run for me with the versions of our dependencies we have pinned. Updating the dependencies solves the problem though and you can see Travis also successfully running tests with these new dependencies at

* Remove unused notify code. (#7805)

This code is unused and hasn't been modified since 2015 except for various times our files have been renamed. Let's remove it.

* Change how _USE_DISTRO is set for mypy (#7804)

If you run `mypy --platform darwin certbot/certbot/` you'll get:
certbot/certbot/ error: Name 'distro' is not defined
certbot/certbot/ error: Name 'distro' is not defined
certbot/certbot/ error: Name 'distro' is not defined
This is because mypy's logic for handling platform specific code is pretty simple and can't figure out what we're doing with `_USE_DISTRO` here. See for more info.

Setting `_USE_DISTRO` to the result of `sys.platform.startswith('linux')` solves the problem without changing the overall behavior of our code here though.

This fixes part of, but there's more work to be done on Windows.

* Fix tests on macOS Catalina (#7794)

This PR fixes the failures that can be seen at

You can see this code running on macOS Catalina at

* Remove references to deprecated flags in Certbot. (#7509)

Related to, this removes some references to deprecated options in Certbot.

The only references I didn't remove were:

* In `certbot/tests/testdata/sample-renewal*` which contains a lot of old values and I think there's even some value in keeping them so we know if we make a change that suddenly causes old renewal configuration files to error.
* In the Apache and Nginx plugins and I created to resolve that issue.

* Remove codecov (#7811)

After getting a +1 from everyone on the team, this PR removes the use of `codecov` from the Certbot repo because we keep having problems with it.

Two noteworthy things about this PR are:

1. I left the text at because I think it's useful to document how to set up a secret variable in general.
2. I'm not sure what the text "Option -e makes sure we fail fast and don't submit to codecov." in `` refers to but it seems incorrect since `-e` isn't accepted or used by the script so I just deleted the line.

As part of this, I said I'd open an issue to track setting up coveralls (which seems to be the only real alternative to codecov) which is at

With my change, failure output looks something like:
$ tox -e py27-cover
Name                                                         Stmts   Miss  Cover   Missing
certbot/certbot/                                      1      0   100%
certbot/certbot/_internal/                            0      0   100%
certbot/certbot/_internal/                           191      4    98%   62-63, 206, 337
certbot/tests/                                  530      0   100%
certbot/tests/                                     374     29    92%   211-213, 480-484, 489-499, 504-511, 545-547, 552-554
TOTAL                                                        14451    647    96%
Command '['/path/to/certbot/dir/.tox/py27-cover/bin/python', '-m', 'coverage', 'report', '--fail-under', '100', '--include', 'certbot/*', '--show-missing']' returned non-zero exit status 2
Test coverage on certbot did not meet threshold of 100%.
ERROR: InvocationError for command /Users/bmw/Development/certbot/certbot/.tox/py27-cover/bin/python (exited with code 1)
_________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________
ERROR:   py27-cover: commands failed
I printed the exception just so we're not throwing away information.

I think it's also possible we fail for a reason other than the threshold not meeting the percentage, but I've personally never seen this, `coverage report` output is not being captured so hopefully that would inform devs if something else is going on, and saying something like "Test coverage probably did not..." seems like overkill to me personally.

* remove codecov

* remove unused variable group

* remove codecov.yml

* Improve failure output.

* Don't run advanced tests on PRs. (#7820)

When I wrote, I didn't understand the default behavior for pull requests if you don't specify `pr` in the yaml file. According to

> If no pr triggers appear in your YAML file, pull request builds are automatically enabled for all branches...

This is not the behavior we want. This PR fixes the problem by disabling builds on PRs.

You should be able to see this working because the advanced tests should not run on this PR but they did run on

* Document safe and simple usage by services without root privileges (#7821)

Certificates are public information by design: they are provided by
web servers without any prior authentication required.  In a public
key cryptographic system, only the private key is secret information.

The private key file is already created as accessible only to the root
user with mode 0600, and these file permissions are set before any key
content is written to the file.  There is no window within which an
attacker with access to the containing directory would be able to read
the private key content.

Older versions of Certbot (prior to 0.29.0) would create private key
files with mode 0644 and rely solely on the containing directory
permissions to restrict access.  We therefore cannot (yet) set the
relevant default directory permissions to 0755, since it is possible
that a user could install Certbot, obtain a certificate, then
downgrade to a pre-0.29.0 version of Certbot, then obtain another
certificate.  This chain of events would leave the second
certificate's private key file exposed.

As a compromise solution, document the fact that it is safe for the
common case of non-downgrading users to change the permissions of
/etc/letsencrypt/{live,archive} to 0755, and explain how to use chgrp
and chmod to make the private key file readable by a non-root service

This provides guidance on the simplest way to solve the common problem
of making keys and certificates usable by services that run without
root privileges, with no requirement to create a custom (and hence
error-prone) executable hook.

Remove the existing custom executable hook example, so that the
documentation contains only the simplest and safest way to solve this
very common problem.

Signed-off-by: Michael Brown <>

* Check OCSP as part of determining if the certificate is due for renewal (#7829)

Fixes #1028.

Doing this now because of

The new `ocsp_revoked_by_paths` function  is taken from with the optional argument removed for now because it is unused.

This function was added in this PR because `` uses `self.latest_common_version()` to determine which certificate should be looked at for determining renewal status at

I think this is unnecessary and you can just look at the currently linked certificate, but I don't think we should be changing the logic that code has always had now.

* Check OCSP status as part of determining to renew

* add integration tests

* add ocsp_revoked_by_paths

* Update changelog for 1.3.0 release

* Release 1.3.0

* Add contents to certbot/ for next version

* Bump version to 1.4.0

* Fix issues with Azure Pipelines (#7838)

This PR fixes two issues.

First, it fixes #7814 by removing our tests on Windows Server 2012. I also added the sentence "Certbot supports Windows Server 2016 and Windows Server 2019." to

Second, it fixes the test failures which can be seen at by no longer manually installing our own version of Python and instead using the one provided by Azure.

These small changes are in the same PR because I wanted to fix test failures ASAP and `UsePythonVersion` is not available on Windows 2012. See

You can see tests passing with this change at

* stop testing on win2012

* switch to UsePythonVersion

* Add changes to the correct changelog entry (#7833)…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

10 participants