Skip to content

Latest commit

 

History

History
491 lines (347 loc) · 17.6 KB

contributing.rst

File metadata and controls

491 lines (347 loc) · 17.6 KB
Raw

Developer Guide

Table of Contents

Getting Started

Certbot has the same system requirements <system_requirements> when set up for development. While the section below will help you install Certbot and its dependencies, Certbot needs to be run on a UNIX-like OS so if you're using Windows, you'll need to set up a (virtual) machine running an OS such as Linux and continue with these instructions on that UNIX-like OS.

Running a local copy of the client

Running the client in developer mode from your local tree is a little different than running Certbot as a user. To get set up, clone our git repository by running:

git clone https://github.com/certbot/certbot

If you're on macOS, we recommend you skip the rest of this section and instead run Certbot in Docker. You can find instructions for how to do this here <docker-dev>. If you're running on Linux, you can run the following commands to install dependencies and set up a virtual environment where you can run Certbot.

cd certbot
./certbot-auto --debug --os-packages-only
tools/venv.sh

If you have Python3 available and want to use it, run the venv3.sh script.

tools/venv3.sh

Note

You may need to repeat this when Certbot's dependencies change or when a new plugin is introduced.

You can now run the copy of Certbot from git either by executing venv/bin/certbot, or by activating the virtual environment. You can do the latter by running:

source venv/bin/activate
# or
source venv3/bin/activate

After running this command, certbot and development tools like ipdb, ipython, pytest, and tox are available in the shell where you ran the command. These tools are installed in the virtual environment and are kept separate from your global Python installation. This works by setting environment variables so the right executables are found and Python can pull in the versions of various packages needed by Certbot. More information can be found in the virtualenv docs.

Find issues to work on

You can find the open issues in the github issue tracker. Comparatively easy ones are marked good first issue. If you're starting work on something, post a comment to let others know and seek feedback on your plan where appropriate.

Once you've got a working branch, you can open a pull request. All changes in your pull request must have thorough unit test coverage, pass our tests, and be compliant with the coding style <coding-style>.

Testing

When you are working in a file foo.py, there should also be a file foo_test.py either in the same directory as foo.py or in the tests subdirectory (if there isn't, make one). While you are working on your code and tests, run python foo_test.py to run the relevant tests.

For debugging, we recommend putting import ipdb; ipdb.set_trace() statements inside the source code.

Once you are done with your code changes, and the tests in foo_test.py pass, run all of the unittests for Certbot with tox -e py27 (this uses Python 2.7).

Once all the unittests pass, check for sufficient test coverage using tox -e cover, and then check for code style with tox -e lint (all files) or pylint --rcfile=.pylintrc path/to/file.py (single file at a time).

Once all of the above is successful, you may run the full test suite using tox --skip-missing-interpreters. We recommend running the commands above first, because running all tests like this is very slow, and the large amount of output can make it hard to find specific failures when they happen.

Warning

The full test suite may attempt to modify your system's Apache config if your user has sudo permissions, so it should not be run on a production Apache server.

Integration testing with the Boulder CA

Generally it is sufficient to open a pull request and let Github and Travis run integration tests for you, however, if you want to run them locally you need Docker and docker-compose installed and working. Fetch and start Boulder, Let's Encrypt's ACME CA software, by using:

./tests/boulder-fetch.sh

If you have problems with Docker, you may want to try removing all containers and volumes and making sure you have at least 1GB of memory.

Set up a certbot_test alias that enables easily running against the local Boulder:

export SERVER=http://localhost:4000/directory
source tests/integration/_common.sh

Run the integration tests using:

./tests/boulder-integration.sh

Code components and layout

acme

contains all protocol specific code

certbot

main client code

certbot-apache and certbot-nginx

client code to configure specific web servers

certbot.egg-info

configuration for packaging Certbot

Plugin-architecture

Certbot has a plugin architecture to facilitate support for different webservers, other TLS servers, and operating systems. The interfaces available for plugins to implement are defined in interfaces.py and plugins/common.py.

The main two plugin interfaces are ~certbot.interfaces.IAuthenticator, which implements various ways of proving domain control to a certificate authority, and ~certbot.interfaces.IInstaller, which configures a server to use a certificate once it is issued. Some plugins, like the built-in Apache and Nginx plugins, implement both interfaces and perform both tasks. Others, like the built-in Standalone authenticator, implement just one interface.

There are also ~certbot.interfaces.IDisplay plugins, which can change how prompts are displayed to a user.

Authenticators

Authenticators are plugins that prove control of a domain name by solving a challenge provided by the ACME server. ACME currently defines three types of challenges: HTTP, TLS-SNI, and DNS, represented by classes in acme.challenges. An authenticator plugin should implement support for at least one challenge type.

An Authenticator indicates which challenges it supports by implementing get_chall_pref(domain) to return a sorted list of challenge types in preference order.

An Authenticator must also implement perform(achalls), which "performs" a list of challenges by, for instance, provisioning a file on an HTTP server, or setting a TXT record in DNS. Once all challenges have succeeded or failed, Certbot will call the plugin's cleanup(achalls) method to remove any files or DNS records that were needed only during authentication.

Installer

Installers plugins exist to actually setup the certificate in a server, possibly tweak the security configuration to make it more correct and secure (Fix some mixed content problems, turn on HSTS, redirect to HTTPS, etc). Installer plugins tell the main client about their abilities to do the latter via the ~.IInstaller.supported_enhancements call. We currently have two Installers in the tree, the ~.ApacheConfigurator. and the ~.NginxConfigurator. External projects have made some progress toward support for IIS, Icecast and Plesk.

Installers and Authenticators will oftentimes be the same class/object (because for instance both tasks can be performed by a webserver like nginx) though this is not always the case (the standalone plugin is an authenticator that listens on port 443, but it cannot install certs; a postfix plugin would be an installer but not an authenticator).

Installers and Authenticators are kept separate because it should be possible to use the ~.StandaloneAuthenticator (it sets up its own Python server to perform challenges) with a program that cannot solve challenges itself (Such as MTA installers).

Installer Development

There are a few existing classes that may be beneficial while developing a new ~certbot.interfaces.IInstaller. Installers aimed to reconfigure UNIX servers may use Augeas for configuration parsing and can inherit from ~.AugeasConfigurator class to handle much of the interface. Installers that are unable to use Augeas may still find the ~.Reverter class helpful in handling configuration checkpoints and rollback.

Writing your own plugin

Certbot client supports dynamic discovery of plugins through the setuptools entry points using the certbot.plugins group. This way you can, for example, create a custom implementation of ~certbot.interfaces.IAuthenticator or the ~certbot.interfaces.IInstaller without having to merge it with the core upstream source code. An example is provided in examples/plugins/ directory.

While developing, you can install your plugin into a Certbot development virtualenv like this:

. venv/bin/activate
. tests/integration/_common.sh
pip install -e examples/plugins/
certbot_test plugins

Your plugin should show up in the output of the last command. If not, it was not installed properly.

Once you've finished your plugin and published it, you can have your users install it system-wide with pip install. Note that this will only work for users who have Certbot installed from OS packages or via pip. Users who run certbot-auto are currently unable to use third-party plugins. It's technically possible to install third-party plugins into the virtualenv used by certbot-auto, but they will be wiped away when certbot-auto upgrades.

Warning

Please be aware though that as this client is still in a developer-preview stage, the API may undergo a few changes. If you believe the plugin will be beneficial to the community, please consider submitting a pull request to the repo and we will update it with any necessary API changes.

Coding style

Please:

  1. Be consistent with the rest of the code.
  2. Read PEP 8 - Style Guide for Python Code.

  3. Follow the Google Python Style Guide, with the exception that we use Sphinx-style documentation:

    def foo(arg):
    """Short description.

    :param int arg: Some number.

    :returns: Argument
    :rtype: int

    """
    return arg
  4. Remember to use pylint.

Mypy type annotations

Certbot uses the mypy static type checker. Python 3 natively supports official type annotations, which can then be tested for consistency using mypy. Python 2 doesn’t, but type annotations can be added in comments. Mypy does some type checks even without type annotations; we can find bugs in Certbot even without a fully annotated codebase.

Certbot supports both Python 2 and 3, so we’re using Python 2-style annotations.

Zulip wrote a great guide to using mypy. It’s useful, but you don’t have to read the whole thing to start contributing to Certbot.

To run mypy on Certbot, use tox -e mypy on a machine that has Python 3 installed.

Note that instead of just importing typing, due to packaging issues, in Certbot we import from acme.magic_typing and have to add some comments for pylint like this:

from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module

Also note that OpenSSL, which we rely on, has type definitions for crypto but not SSL. We use both. Those imports should look like this:

from OpenSSL import crypto
from OpenSSL import SSL # type: ignore # https://github.com/python/typeshed/issues/2052

Submitting a pull request

Steps:

  1. Write your code!
  2. Make sure your environment is set up properly and that you're in your virtualenv. You can do this by running ./tools/venv.sh. (this is a very important step)
  3. Run tox -e lint to check for pylint errors. Fix any errors.
  4. Run tox --skip-missing-interpreters to run the entire test suite including coverage. The --skip-missing-interpreters argument ignores missing versions of Python needed for running the tests. Fix any errors.
  5. Submit the PR.
  6. Did your tests pass on Travis? If they didn't, fix any errors.

Asking for help

If you have any questions while working on a Certbot issue, don't hesitate to ask for help! You can do this in the #letsencrypt-dev IRC channel on Freenode. If you don't already have an IRC client set up, we recommend you join using Riot.

Updating certbot-auto and letsencrypt-auto

Updating the scripts

Developers should not modify the certbot-auto and letsencrypt-auto files in the root directory of the repository. Rather, modify the letsencrypt-auto.template and associated platform-specific shell scripts in the letsencrypt-auto-source and letsencrypt-auto-source/pieces/bootstrappers directory, respectively.

Building letsencrypt-auto-source/letsencrypt-auto

Once changes to any of the aforementioned files have been made, the letsencrypt-auto-source/letsencrypt-auto script should be updated. In lieu of manually updating this script, run the build script, which lives at letsencrypt-auto-source/build.py:

python letsencrypt-auto-source/build.py

Running build.py will update the letsencrypt-auto-source/letsencrypt-auto script. Note that the certbot-auto and letsencrypt-auto scripts in the root directory of the repository will remain unchanged after this script is run. Your changes will be propagated to these files during the next release of Certbot.

Opening a PR

When opening a PR, ensure that the following files are committed:

  1. letsencrypt-auto-source/letsencrypt-auto.template and letsencrypt-auto-source/pieces/bootstrappers/*
  2. letsencrypt-auto-source/letsencrypt-auto (generated by build.py)

It might also be a good idea to double check that no changes were inadvertently made to the certbot-auto or letsencrypt-auto scripts in the root of the repository. These scripts will be updated by the core developers during the next release.

Updating the documentation

In order to generate the Sphinx documentation, run the following commands:

make -C docs clean html man

This should generate documentation in the docs/_build/html directory.

Note

If you skipped the "Getting Started" instructions above, run pip install -e ".[docs]" to install Certbot's docs extras modules.

Running the client with Docker

You can use Docker Compose to quickly set up an environment for running and testing Certbot. To install Docker Compose, follow the instructions at https://docs.docker.com/compose/install/.

Note

Linux users can simply run pip install docker-compose to get Docker Compose after installing Docker Engine and activating your shell as described in the Getting Started <getting_started> section.

Now you can develop on your host machine, but run Certbot and test your changes in Docker. When using docker-compose make sure you are inside your clone of the Certbot repository. As an example, you can run the following command to check for linting errors:

docker-compose run --rm --service-ports development bash -c 'tox -e lint'

You can also leave a terminal open running a shell in the Docker container and modify Certbot code in another window. The Certbot repo on your host machine is mounted inside of the container so any changes you make immediately take effect. To do this, run:

docker-compose run --rm --service-ports development bash

Now running the check for linting errors described above is as easy as:

tox -e lint

Notes on OS dependencies

OS-level dependencies can be installed like so:

./certbot-auto --debug --os-packages-only

In general...

  • sudo is required as a suggested way of running privileged process
  • Python 2.7 or 3.4+ is required
  • Augeas is required for the Python bindings
  • virtualenv is used for managing other Python library dependencies

FreeBSD

FreeBSD by default uses tcsh. In order to activate virtualenv (see above), you will need a compatible shell, e.g. pkg install bash && bash.