An ACME-based CA, written in Go.
Go JavaScript Python Shell Other
Latest commit 170e37c Jan 20, 2017 @rolandshoemaker rolandshoemaker committed with cpu Add a special error message if we are trying to talk TLS to a HTTP-on…
…ly server (#2511)

If the VA fails to validate a TLS-SNI-01 challenge because it is trying to talk TLS to a HTTP-only server return a special error message that is slightly more informative.
Permalink
Failed to load latest commit information.
Godeps Update cfssl, CT, and OCSP dependencies (#2170) Jan 13, 2017
akamai Fix akamai cache purger bugs (#2443) Dec 21, 2016
bdns Adds unique VA DNS validation error for empty TXTs. (#2401) Dec 8, 2016
ca Add counter for signatures (#2510) Jan 20, 2017
cdr Spelling fixes Nov 30, 2016
cmd Purge POST'd OCSP responses as well as GETs (#2449) Jan 18, 2017
core Only resubmit missing SCTs. (#2342) Dec 5, 2016
csr Improve error messages. (#2256) Oct 18, 2016
data Adds warning about unsub link (#2404) Dec 8, 2016
docs Update OCSP load testing doc. (#2486) Jan 18, 2017
features Switch to Google's v4 safebrowsing library. (#2446) Dec 27, 2016
goodkey Simplify KeyPolicy code (#2092) Jul 30, 2016
grpc Allow probs.ProblemDetails to be passed across gRPC layer (#2506) Jan 19, 2017
log Spelling (#2500) Jan 16, 2017
mail Fix auth line in email test (#2499) Jan 17, 2017
metrics Copy all statsd stats to Prometheus. (#2474) Jan 10, 2017
mocks Only resubmit missing SCTs. (#2342) Dec 5, 2016
nonce Remove direct usages of go-statsd-client in favor of using metrics.Sc… Sep 7, 2016
policy Handles trailing '.' in domain with specific err (#2384) Dec 2, 2016
prefixdb Renames prefixedDatabase pkg to prefixdb Nov 30, 2016
probs Adds RejectedIdentifierError on Blacklisted error. (#1944) Jun 29, 2016
publisher Update cfssl, CT, and OCSP dependencies (#2170) Jan 13, 2017
ra Fix SA wrappers for maps. (#2498) Jan 17, 2017
ratelimit Add certificatesPerName rate limit to integration test (#1940) Jun 17, 2016
reloader Spelling (#2500) Jan 16, 2017
revocation Fixes gofmt -s diffs Nov 30, 2016
rpc Spelling (#2500) Jan 16, 2017
sa Add gRPC server to SA (#2374) Dec 3, 2016
test Temporarily switch to SIGKILL for startservers shutdown. (#2512) Jan 20, 2017
va Add a special error message if we are trying to talk TLS to a HTTP-on… Jan 20, 2017
vendor Update cfssl, CT, and OCSP dependencies (#2170) Jan 13, 2017
wfe Spelling (#2500) Jan 16, 2017
.dockerignore Roll forward "Run Travis tests in Docker (#1830)" (#1838) May 24, 2016
.gitignore Docker improvements. Apr 4, 2016
.travis.yml Switch to Golang 1.7.3 in travis (#2305) Nov 7, 2016
CODE_OF_CONDUCT.md Add Code of Conduct (#1961) Jun 22, 2016
CONTRIBUTING.md Spelling (#2500) Jan 16, 2017
DESIGN.md Spelling (#2500) Jan 16, 2017
Dockerfile Remove /go/bin from PATH in Docker images (#2412) Dec 9, 2016
LICENSE.txt Remove all stray copyright headers and appends the initial line to LI… May 31, 2016
Makefile Switch back to go 1.5 in Travis. (#2261) Oct 20, 2016
README.md Update cfssl, CT, and OCSP dependencies (#2170) Jan 13, 2017
docker-compose.yml Add Prometheus to docker-compose. Jan 7, 2017
docker-rebuild.sh Add tiny docker-compose rebuild script (#2039) Jul 13, 2016
start.py Make restarting boulder in docker nicer. (#2492) Jan 13, 2017
test.sh Don't echo command from run_and_expect_silence. (#2405) Dec 8, 2016

README.md

Boulder - An ACME CA

This is an implementation of an ACME-based CA. The ACME protocol allows the CA to automatically verify that an applicant for a certificate actually controls an identifier, and allows domain holders to issue and revoke certificates for their domains.

Build Status Coverage Status

Quickstart

Boulder has a Dockerfile to make it easy to install and set up all its dependencies. This is how the maintainers work on Boulder, and is our main recommended way to run it.

Make sure you have a local copy of Boulder in your $GOPATH:

export GOPATH=~/gopath
git clone https://github.com/letsencrypt/boulder/ $GOPATH/src/github.com/letsencrypt/boulder

To start Boulder in a Docker container, run:

docker-compose up

To run tests:

docker-compose run boulder ./test.sh

To run a specific unittest:

docker-compose run boulder go test ./ra

The configuration in docker-compose.yml mounts your $GOPATH on top of its own $GOPATH. So you can edit code on your host and it will be immediately reflected inside Docker images run with docker-compose.

If docker-compose fails with an error message like "Cannot start service boulder: oci runtime error: no such file or directory" or "Cannot create container for service boulder" you should double check that your $GOPATH exists and doesn't contain any characters other than letters, numbers, - and _.

By default, Boulder uses a fake DNS resolver that resolves all hostnames to 127.0.0.1. This is suitable for running integration tests inside the Docker container. If you want Boulder to be able to communicate with a client running on your host instead, you should find your host's Docker IP with:

ifconfig docker0 | grep "inet addr:" | cut -d: -f2 | awk '{ print $1}'

And edit docker-compose.yml to change the FAKE_DNS environment variable to match.

Alternatively, you can override the docker-compose.yml default with an environmental variable using -e (replace 172.17.0.1 with the host IPv4 address found in the command above)

docker-compose run -e FAKE_DNS=172.17.0.1 --service-ports boulder ./start.py

Boulder's default VA configuration (test/config/va.json) is configured to connect to port 5002 to validate HTTP-01 challenges and port 5001 to validate TLS-SNI-01 challenges. If you want to solve challenges with a client running on your host you should make sure it uses these ports to respond to validation requests, or update the VA configuration's portConfig to use ports 80 and 443 to match how the VA operates in production and staging environments. If you use a host-based firewall (e.g. ufw or iptables) make sure you allow connections from the Docker instance to your host on the required ports.

If a base image changes (i.e. letsencrypt/boulder-tools) you will need to rebuild images for both the boulder and bhsm containers and re-create them. The quickest way to do this is with this command:

./docker-rebuild.sh

Slow start

If you can't use the Docker setup, here are instructions for setting up a Boulder development environment without it.

We recommend setting git's fsckObjects setting for better integrity guarantees when getting updates.

Boulder requires an installation of RabbitMQ, libtool-ltdl, goose, and MariaDB 10.1 to work correctly. On Ubuntu and CentOS, you may have to install RabbitMQ from https://rabbitmq.com/download.html to get a recent version. If you want to save some trouble installing MariaDB and RabbitMQ you can run them using Docker:

docker-compose up -d bmysql brabbitmq bhsm

Also, Boulder requires Go 1.5. As of September 2015 this version is not yet available in OS repositories, so you will have to install from https://golang.org/dl/. Add ${GOPATH}/bin to your path.

Ubuntu:

sudo apt-get install libltdl3-dev mariadb-server rabbitmq-server

CentOS:

sudo yum install libtool-ltdl-devel MariaDB-server MariaDB-client rabbitmq-server

Arch Linux:

sudo pacman -S libtool mariadb rabbitmq --needed

OS X:

brew install libtool mariadb rabbitmq

or

sudo port install libtool mariadb-server rabbitmq-server

(On OS X, using port, you will have to add CGO_CFLAGS="-I/opt/local/include" CGO_LDFLAGS="-L/opt/local/lib" to your environment or go invocations.)

Edit /etc/hosts to add this line:

127.0.0.1 boulder boulder-rabbitmq boulder-mysql

Resolve Go-dependencies, set up a database and RabbitMQ:

./test/setup.sh

Note: setup.sh calls create_db.sh, which uses the root MariaDB user with the default password, so if you have disabled that account or changed the password you may have to adjust the file or recreate the commands.

Install SoftHSM to store the CA private key in a way that can be accessed using PKCS#11. Then run ./test/make-softhsm.sh and follow its instructions.

Start all boulder components with test configs (Ctrl-C kills all):

./start.py

Run tests:

./test.sh

Working with a client:

Check out the Certbot client from https://github.com/certbot/certbot and follow the setup instructions there. Once you've got the client set up, you'll probably want to run it against your local Boulder. There are a number of command line flags that are necessary to run the client against a local Boulder, and without root access. The simplest way to run the client locally is to source a file that provides an alias for certbot (certbot_test) that has all those flags:

source ~/certbot/tests/integration/_common.sh
certbot_test certonly -a standalone -d example.com

Your local Boulder instance uses a fake DNS server that returns 127.0.0.1 for any query, so you can use any value for the -d flag. You can also override that value by setting the environment variable FAKE_DNS=1.2.3.4

Component Model

The CA is divided into the following main components:

  1. Web Front End
  2. Registration Authority
  3. Validation Authority
  4. Certificate Authority
  5. Storage Authority
  6. OCSP Updater
  7. OCSP Responder

This component model lets us separate the function of the CA by security context. The Web Front End and Validation Authority need access to the Internet, which puts them at greater risk of compromise. The Registration Authority can live without Internet connectivity, but still needs to talk to the Web Front End and Validation Authority. The Certificate Authority need only receive instructions from the Registration Authority. All components talk to the SA for storage, so lines indicating SA RPCs are not shown here.


                             +--------- OCSP Updater
                             |               |
                             v               |
                            CA               |
                             ^               |
                             |               v
       Subscriber -> WFE --> RA --> SA --> MariaDB
                             |               ^
Subscriber server <- VA <----+               |
                                             |
          Browser ------------------>  OCSP Responder

Internally, the logic of the system is based around four types of objects: registrations, authorizations, challenges, and certificates, mapping directly to the resources of the same name in ACME.

Requests from ACME clients result in new objects and changes to objects. The Storage Authority maintains persistent copies of the current set of objects.

Objects are also passed from one component to another on change events. For example, when a client provides a successful response to a validation challenge, it results in a change to the corresponding validation object. The Validation Authority forwards the new validation object to the Storage Authority for storage, and to the Registration Authority for any updates to a related Authorization object.

Boulder uses AMQP as a message bus. For components that you want to be remote, it is necessary to instantiate a "client" and "server" for that component. The client implements the component's Go interface, while the server has the actual logic for the component. More details in amqp-rpc.go.

The full details of how the various ACME operations happen in Boulder are laid out in DESIGN.md

Dependencies

All Go dependencies are vendored under the vendor directory, to make dependency management easier.

Local development also requires a RabbitMQ installation and MariaDB 10 installation (see above). MariaDB should be run on port 3306 for the default integration tests.

To update the Go dependencies:

# Fetch godep
go get -u github.com/tools/godep
# Check out the currently vendorized version of each dependency.
godep restore
# Update to the latest version of a dependency. Alternately you can cd to the
# directory under GOPATH and check out a specific revision. Here's an example
# using cfssl:
go get -u github.com/cloudflare/cfssl/...
# Update the Godep config to the appropriate version.
godep update github.com/cloudflare/cfssl/...
# Save the dependencies
godep save ./...
git add Godeps vendor
git commit

NOTE: If you get "godep: no packages can be updated," there's a good chance you're trying to update a single package that belongs to a repo with other packages. For instance, godep update golang.org/x/crypto/ocsp will produce this error, because it's part of the golang.org/x/crypto repo, from which we also import the pkcs12 package. Godep requires that all packages from the same repo be on the same version, so it can't update just one. The error message is not particularly helpful. See https://github.com/tools/godep/issues/164 for the issue dedicated to fixing it.

NOTE: Updating cfssl in particular is tricky, because cfssl vendors github.com/google/certificate-transparency/... and golang.org/x/crypto/ocsp/..., which we also vendor. In practice this means you need to check out those two dependencies to the same version cfssl uses (available in vendor/manifest in the cfssl repo). If you fail to do this, you will get conflicting types between our vendored version and the cfssl vendored version.

godep update golang.org/x/crypto/...  github.com/cloudflare/cfssl/... github.com/google/certificate-transparency/...
godep save ./...