Make Certbot runnable on Windows

[adferrand]

So I wanted to build a python library for Linux and Windows using Certbot, when I discovered that it was compatible only with UNIX-like systems.

@ahaw021 already did a study and local actions to make Certbot runnable on Windows (see this topic) last year, but it did not get to upstream.

This PR is based on his work, and includes other compatibility fixes with the latest certbot version.

So far, here is what I tested, on Windows 10 with Python 3.7:

• run certbot help from command line
• run certbot certonly using standalone plugin
• run certbot certonly using webroot plugin
• run certbot certonly using manual plugin + authenticator/cleanup hook scripts

I will update this PR with more tests and relevant improvements, in particular functionalities mentionned as working by @ahaw021. Do not hesitate to submit use cases to make the compatibility more complete.

Currently, this is the very beginning, with really basic functionalities. In particular, all the logic of certbot-auto should be ported to Windows, certainly as a Batch script.

Available for any discussion, I know this is a long-term work before this PR could be merged.

[adferrand]

I just added a prototype of certbot-auto for windows, coded as a batch script. It is a wrapper that:

• ensures that python executable is installed
• ensures that python is of version 3
• install python 3 if needed
• create/update a virtual env
• activate this venv
• update certbot with pip
• execute certbot with the arguments given to certbot-auto
• disable the venv

In theory, a user could clone the certbot repo, and run something like certbot-auto certonly -d example.com with nothing more than the cmd, everything else will be installed from the script before delegating to certbot, similarly to certbot-auto for UNIX-like systems. It is really a prototype, as I could not decently translate all the business intelligence accumulated on the bash script, as it handles very versatile system configurations in the Linux world.

For now, this script installs certbot from the git repo sources as editable through pip for testing purpose.

[ahaw021]

hi mate

https://github.com/ahaw021/CERTBOT-WINDOWS-BUILD/blob/master/WIN-CERTBOT-BUILDER.ps1

this is something i built a while ago to build certbot on windows environments

i haven't touched it in a while but more than happy to review :D

[adferrand]

Hello @ahaw021, yes I analyzed your script together with the posts you did to integrate I order to make this PR.

My approach here is to modify the certbot source code to make it runnable both on Linux and on Windows, in a manner that one can call pip install certbot with the Python Windows distribution, and has certbot working correctly.

In your script there is two parts, modifications on code required to avoid exceptions on windows, and the runtime operations needed at the os and python level. For the modifications on the code, I gathered them on a new library in certbot, compat.py, to abstract the code execution divergence between Linux and Windows from the business logic itself. For the runtime, I built a certbot-auto Batch script, equivalently to the certbot-auto Shell script for Linux and its use (configure and launch certbot from a unique script).

[ahaw021]

yup you are right - i forgot i cheated on removing some of the issues :D

good to see you have everything in hand -- let me know if you would like me to test

[adferrand]

Definitively I will need your help to test this ^^

First, I have only a Windows 10, but the target for production is very likely to be Windows Server editions.

Secondly, I would like to build a plugin for IIS. I know that Microsoft provides a IIS instance for developers on Windows 10, but making certbot production-grade ready for a full fledged IIS running on Windows Server 2012R2 is another story :)

[adferrand]

As we are on testing, about continuous integration. So far all tests are passing on Linux (like through Travis CI) as one can see on the PR. It was one of the goal, to keep complete compatibility in Linux.

On Windows, launching the unit tests on the core library gives a lot of errors. For what I can see, it is not because of the functionalities themselves, but because of the mocking mechanism which is completely failing. And of course integration tests on Linux apache, nging and so on are irrelevant.

So I think that for Windows, we would need a completely new stack for integration tests.

[adferrand]

OK, here is what I think for a CI stack on Windows. We can create a new tox-win.ini file to configure specifically tox for Windows, because all shell scripts that configure integration tests on Linux do not apply.

Then, the equivalent of Travis CI for Windows is AppVeyor. It allows, in a very similar way, to run builds on a Windows environment (typically 2012 or 2016 Server). It is configured using a yml file located at the root of the project: appveyor.yml. The syntax is very similar to .travis.yml. AppVeyor can be interfaced with GitHub, showing build statuses on PR, and is free for open-source projects.

About the tests themselves, approximately 25% of them are failing for various reasons. It will take a lot of time to correct them. In the mean time, I propose a decorator, certbot.tests.util.broken_on_windows, which skip all tests method decorated with it current plateform is Windows, and if the env variable SKIP_TESTS_BROKEN_ON_WINDOWS is not set to false (by default, true is taken).

I did a start on my forked repo on the branch windows-ci.

If it is ok, I will merge windows-ci on the current PR or a new PR. When it is accepted, certbot maintainers will only need to create an account on AppVeyor linked to certbot GitHub repo. Then CI build for Windows will be executed for each PR for master and on commit on branches currently watched by Travis CI.

[adferrand]

So, about the python compatibility matrix. I will not make compatible certbot with python 2.7. It will require a least python 3.4.

On 2.x, I have 40 tests that are failing just because of the lack of support for windows file manipulation, in particular the symbolic links. Adding the support manually seems to be quite impossible, and some proposed patches never reached upstream, as basically python 2.x does not receive developments anymore.

And as we can consider certbot on windows as a "new" software, we can follow most of the advices in the community about dropping support for python 2.x.

So here it is. For Linux, where Debian still uses a well supported python 2.x version by default, it seems reasonable to ensure the compatibility. But on Windows, python 2.x is quite a mess, and users should be required to run python 3.4 to run correctly certbot.

I am preparing the CI for Windows accordingly.

[adferrand]

OK @bmw, for me, the PR is now complete.

Do not hesitate to ask me if some work still need to be done.

Regards,
Adrien Ferrand

[bmw]

So, about the python compatibility matrix. I will not make compatible certbot with python 2.7. It will require a least python 3.4.

That's totally fine. We probably will be deprecating Python 2.7 support on UNIX in the near future as well.

I have a few very small changes I'd like to see, but otherwise, this LGTM!

EDIT: I forgot to mention that I opened issues to track the issues we're leaving unresolved for now. They were:

#6354
#6355
#6356

[bmw]

A few very minor comments but we're getting close here!