Auditing for TLS certificates.
C++ Go Python Java Shell Protocol Buffer Other
Latest commit b61c0fa Dec 7, 2016 @eranmes eranmes Python: Add ECDSA signing code.
Add a module for producing ECDSA signatures that are suitable for use in
CT log implementations (both 6962 and 6962-bis):

* Produce deterministic signatures.
* Signature bytes are DER-encoded.
* Signature is wrapped in a DigitallySigned protobuf for easy TLS
  serialization.

This is necessary for a reference implementation of a 6962-bis log.

The dependency on ecdsa is also re-introduced.
Permalink
Failed to load latest commit information.
build travis: On OSX add user's pip dir to PATH (#1315) Oct 13, 2016
cloud Add etcd reinit script Oct 11, 2016
cpp Roll BoringSSL forward and remove temporary ifdefs. (#1166) Dec 14, 2016
docs docs: tweaks and clarifications Jun 29, 2016
go go: fix some golint/go vet warnings (#1357) Dec 12, 2016
java Java: Implement verification of SCTs for logs with RSA keys (#1349) Dec 6, 2016
m4 Add support to configure.ac for checking thread local support Aug 14, 2015
proto Remove the Java package from the main ct.proto file. May 23, 2016
python Python: Add ECDSA signing code. Dec 20, 2016
test Java: Implement verification of SCTs for logs with RSA keys (#1349) Dec 6, 2016
.clang-format Clang format Nov 16, 2015
.dockerignore Add initial cut of configs to run up SuperDuper on GKE. Jan 29, 2015
.gitignore Add a missing entry to the .gitignore file. Jun 29, 2016
.travis.yml travis: Run go tests with -race Nov 8, 2016
AUTHORS Java: Implement verification of SCTs for logs with RSA keys (#1349) Dec 6, 2016
CONTRIBUTING.md Add a CONTRIBUTING.md file. May 21, 2014
CONTRIBUTORS Java: Implement verification of SCTs for logs with RSA keys (#1349) Dec 6, 2016
DEPS Roll BoringSSL forward and remove temporary ifdefs. (#1166) Dec 14, 2016
Dockerfile Base Docker images off Ubuntu 14.04 Jun 29, 2016
Dockerfile-bintrans-xjson Base Docker images off Ubuntu 14.04 Jun 29, 2016
Dockerfile-ct-mirror Base Docker images off Ubuntu 14.04 Jun 29, 2016
LICENSE Rename LICENCE to LICENSE. Jul 10, 2014
Makefile.am Remove the unused objecthash bits from C++ code. Aug 24, 2016
README-MacOS.md docs: rename & tweak Mac OS X instructions Jun 29, 2016
README.md Fix the second build-quick-start anchor link Aug 19, 2016
autogen.sh Use autoconf/automake. Apr 21, 2015
build.gclient Fixes typos and formatting (#1148) Apr 11, 2016
build.properties Travis uses gclient. Oct 12, 2015
build.xml Make sure Travis notices failing Java tests. May 23, 2016
configure.ac Add a check for objecthash.h and make xjson-server depend on it. Apr 20, 2016
gtest.patch gtest patch for FreeBSD. Oct 23, 2014

README.md

certificate-transparency: Auditing for TLS certificates

Build Status

Introduction

This repository holds open-source code for functionality related to certificate transparency (CT). The main areas covered are:

  • An open-source, distributed, implementation of a CT Log server, also including:
    • An implementation of a read-only "mirror" server that mimics a remote Log.
    • Ancillary tools needed for managing and maintaining the Log.
  • A collection of client tools and libraries for interacting with a CT Log, in various programming languages.
  • An experimental implementation of a DNS server that returns CT proofs in the form of DNS records.
  • An experimental implementation of a general Log that allows arbitrary data (not just TLS certificates) to be logged.

The supported platforms are:

  • Linux: tested on Ubuntu 14.04; other variants (Fedora 22, CentOS 7) may require tweaking of compiler options.
  • OS X: version 10.10
  • FreeBSD: version 10.*

Build Quick Start

First, ensure that the build machine has all of the required build dependencies. Then use gclient to retrieve and build the other software needed by the Log, and then use (GNU) make to build and test the CT code:

export CXX=clang++ CC=clang
mkdir ct  # or whatever directory you prefer
cd ct
gclient config --name="certificate-transparency" https://github.com/google/certificate-transparency.git
gclient sync  # retrieve and build dependencies
# substitute gmake or gnumake below if that's what your platform calls it:
make -C certificate-transparency check  # build the CT software & self-test

Code Layout

The source code is generally arranged according to implementation language, in the cpp, go, java and python subdirectories. The key subdirectories are:

  • For the main distributed CT Log itself:
    • cpp/log: Main distributed CT Log implementation.
    • cpp/merkletree: Merkle tree implementation.
    • cpp/server: Top-level code for server implementations.
    • cpp/monitoring: Code to export operation statistics from CT Log.
  • The CT mirror Log implementation also uses:
    • cpp/fetcher: Code to fetch entries from another Log
  • Client code for accessing a CT Log instance:
    • cpp/client: CT Log client code in C++
    • go/client: CT Log client code in Go
    • python/ct: CT Log client code in Python
    • java/src/org/certificatetransparency/ctlog: CT Log client code in Java
  • Other tools:
    • go/fixchain: Tool to fix up certificate chains
    • go/gossip: Code to allow gossip-based synchronization of cert info
    • go/scanner: CT Log scanner tool
    • go/merkletree: Merkle tree implementation in Go.

Building the Code

The CT software in this repository relies on a number of other open-source projects, and we recommend that:

  • The CT software should be built using local copies of these dependencies rather than installed packages, to prevent version incompatibilities.
  • The dependent libraries should be statically linked into the CT binaries, rather than relying on dynamically linked libraries that may be different in the deployed environment.

The supported build system uses the gclient tool from the Chromium project to handle these requirements and to ensure a reliable, reproducible build. Older build instructions for using Ubuntu or Fedora packages and for manually building dependencies from source are no longer supported.

Within a main top-level directory, gclient handles the process of:

  • generating subdirectories for each dependency
  • generating a subdirectory for for the CT Log code itself
  • building all of the dependencies
  • installing the built dependencies into an install/ subdirectory
  • configuring the CT build to reference the built dependencies.

Under the covers, this gclient build process is controlled by:

  • The master DEPS file, which configures the locations and versions of the source code needed for the dependencies, and which hooks onto ...
  • The makefiles in the build/ subdirectory, which govern the build process for each dependency, ensuring that:
    • Static libraries are built.
    • Built code is installed into the local install/ directory, where it is available for the build of the CT code itself.

Build Dependencies

The following tools are needed to build the CT software and its dependencies.

  • depot_tools
  • autoconf/automake etc.
  • libtool
  • shtool
  • clang++ (>=3.4)
  • cmake (>=v3.1.2)
  • git
  • GNU make
  • Tcl
  • pkg-config
  • Python 2.7

The exact packages required to install these tools depends on the platform. For a Debian-based system, the relevant packages are: autoconf automake libtool shtool cmake clang git make tcl pkg-config python2.7

Software Dependencies

The following collections of additional software are used by the main CT Log codebase.

  • Google utility libraries:
    • gflags: command-line flag handling
    • glog: logging infrastructure, which also requires libunwind.
    • Google Mock: C++ test framework
    • Google Test: C++ mocking framework
    • Protocol Buffers: language-neutral data serialization library
    • tcmalloc: efficient malloc replacement optimized for multi-threaded use
  • Other utility libraries:
    • libevent: event-processing library
    • libevhtp: HTTP server plug-in/replacement for libevent
    • json-c: JSON processing library
    • libunwind: library for generating stack traces
  • Cryptographic library: one of the following, selected via the SSL build variable.
  • Data storage functionality: one of the following, defaulting (and highly recommended to stick with) LevelDB.
    • LevelDB: fast key-value store, which uses:
    • SQLite: file-based SQL library

The extra (experimental) CT projects in this repo involve additional dependencies:

  • The experimental CT DNS server uses:
    • ldnbs: DNS library, including DNSSEC function (which relies on OpenSSL for crypto functionality)
  • The experimental general Log uses:
    • objecthash: tools for hashing objects in a language/encoding-agnostic manner
    • ICU: Unicode libraries (needed to normalize international text in objects)

Build Troubleshooting

Compiler Warnings/Errors

The CT C++ codebase is built with the Clang -Werror flag so that the codebase stays warning-free. However, this can cause build errors when newer/different versions of the C++ compiler are used, as any newly created warnings are treated as errors. To fix this, add the appropriate -Wno-error=<warning-name> option to CXXFLAGS.

For example, on errors involving unused variables try using:

CXXFLAGS="-O2 -Wno-error=unused-variable" gclient sync

If an error about an unused typedef in a glog header file occurs, try this:

CXXFLAGS="-O2 -Wno-error=unused-variable -Wno-error=unused-local-typedefs" gclient sync

When changing CXXFLAGS it's safer to remove the existing build directories in case not all dependencies are properly accounted for and rebuilt. If problems persist, check that the Makefile in certificate-transparency contains the options that were passed in CXXFLAGS.

Working on a Branch

If you're trying to clone from a branch on the CT repository then you'll need to substitute the following command for the gclient config command above, replacing branch as appropriate

gclient config --name="certificate-transparency" https://github.com/google/certificate-transparency.git@branch

Using BoringSSL

The BoringSSL fork of OpenSSL can be used in place of OpenSSL (but note that the experimental CT DNS server does not support this configuration). To enable this, after the first step (gclient config ...) in the gclient build process, modify the top-level .gclient to add:

      "custom_vars": { "ssl_impl": "boringssl" } },

Then continue the build process with the gclient sync step.

Testing the Code

Unit Tests

The unit tests for the CT code can be run with the make check target of certificate-transparency/Makefile.

Testing and Logging Options

Note that several tests write files on disk. The default directory for storing temporary testdata is /tmp. You can change this by setting TMPDIR=<tmpdir> for make.

End-to-end tests also create temporary certificate and server files in test/tmp. All these files are cleaned up after a successful test run.

For logging options, see the glog documentation.

By default, unit tests log to stderr, and log only messages with a FATAL level (i.e., those that result in abnormal program termination). You can override the defaults with command-line flags.

Deploying a Log

The build process described so far generates a set of executables; however, other components and configuration is needed to set up a running CT Log. In particular, as shown in the following diagram:

  • A set of web servers that act as HTTPS terminators and load balancers is needed in front of the CT Log instances.
  • A cluster of etcd instances is needed to provide replication and synchronization services for the CT Log instances.

Configuring and setting up a distributed production Log is covered in a separate document.

Operating a Log

Running a successful, trusted, certificate transparency Log involves more than just deploying a set of binaries. Information and advice on operating a running CT Log is covered in a separate document