Auditing for TLS certificates, Go code.
Go Shell Other
Clone or download
daviddrysdale jsonclient: increase backoff test leeway (#294)
There has been an occasional flake like:
  backoff_test.go:114: backoff.set(30m0s)=59m59.999970667s; want 1h0m0s
Latest commit 7b28828 Jul 18, 2018
Permalink
Failed to load latest commit information.
asn1 asn1: add extra test case for optional RawValue Feb 15, 2018
client Switch ct-go travis build to 1.10 (#270) Jun 8, 2018
ctutil sctcheck: more diagnostic output May 24, 2018
dnsclient *client: add BaseURI() method for convenience (#221) Apr 26, 2018
fixchain fixchain: create valid add-chain response (#260) May 30, 2018
gossip gossip: fix logging format specifier Jun 18, 2018
ingestor/ranges Include estimated % complete and time left. (#132) Jan 24, 2018
jsonclient jsonclient: increase backoff test leeway (#294) Jul 18, 2018
logid Make the String method take a LogID instead of a *LogID. Oct 3, 2017
loglist loglist: prefix match odd number of hex digits (#252) May 30, 2018
preload Add flags to configure preloader http Transport (#275) Jun 18, 2018
scanner scanner: add ScanLog() which returns tree size May 24, 2018
scripts scripts: retry gometalinter when it times out (#293) Jul 18, 2018
testdata integration: run gosmin in parallel with hammer May 24, 2018
third_party/prometheus trillian: initial Prometheus consoles Jun 20, 2017
tls Wean CTFE off trillian "tcrypto" for Go "crypto" (#194) Mar 21, 2018
trillian Use %s formatting for most errors in handlers.go (#292) Jul 18, 2018
vendor/github.com Upgrade vendored etcd to v3.3.7 Jun 20, 2018
x509 x509: make Windows policy parameter type version-specific (#286) Jul 5, 2018
x509util certcheck: add option to use system roots Jul 6, 2018
.gitignore gossip: add goshawk for checking minimal gossip May 24, 2018
.travis.yml Switch ct-go travis build to latest protoc (#269) Jun 11, 2018
AUTHORS client: make backoff to per-client not per-POST (#75) Oct 30, 2017
CHANGELOG.md Add changelog entry for v1.2.0. (#287) Jul 5, 2018
CONTRIBUTING.md Pristine copy of CT Go code May 4, 2017
CONTRIBUTORS client: make backoff to per-client not per-POST (#75) Oct 30, 2017
LICENSE Squashed 'vendor/github.com/coreos/etcd/' content from commit 44a017e… Sep 11, 2017
README.md Rewrite README to be more up-to-date/accurate (#145) Feb 7, 2018
cloudbuild_tag.yaml Add config for creating Docker images for tagged releases automatically Jun 25, 2018
gometalinter.json Switch ct-go travis build to 1.10 (#270) Jun 8, 2018
serialization.go ct: add LeafHashForLeaf entrypoint Mar 21, 2018
serialization_test.go Mark t.Helper() functions in tests Oct 11, 2017
signatures.go Remove spurious flag in ct package May 24, 2018
signatures_test.go Remove spurious flag in ct package May 24, 2018
types.go Add CT API endpoint strings (#254) May 29, 2018
types_test.go Factor out conversion from GetSTHResponse to STH (#248) May 23, 2018

README.md

Certificate Transparency: Go Code

Build Status Go Report Card GoDoc

This repository holds Go code related to Certificate Transparency (CT). The repository requires Go version 1.9.

Repository Structure

The main parts of the repository are:

  • Encoding libraries:
    • asn1/ and x509/ are forks of the upstream Go encoding/asn1 and crypto/x509 libraries. We maintain separate forks of these packages because CT is intended to act as an observatory of certificates across the ecosystem; as such, we need to be able to process somewhat-malformed certificates that the stricter upstream code would (correctly) reject. Our x509 fork also includes code for working with the pre-certificates defined in RFC 6962.
    • tls holds a library for processing TLS-encoded data as described in RFC 5246.
    • x509util provides additional utilities for dealing with x509.Certificates.
  • CT client libraries:
    • The top-level ct package (in .) holds types and utilities for working with CT data structures defined in RFC 6962.
    • client/ and jsonclient/ hold libraries that allow access to CT Logs via entrypoints described in section 4 of RFC 6962.
    • scanner/ holds a library for scanning the entire contents of an existing CT Log.
  • Command line tools:
    • ./client/ctclient allows interaction with a CT Log
    • ./scanner/scanlog allows an existing CT Log to be scanned for certificates of interest; please be polite when running this tool against a Log.
    • ./x509util/certcheck allows display and verification of certificates
    • ./x509util/crlcheck allows display and verification of certificate revocation lists (CRLs).
  • CT Personality for Trillian:
    • trillian/ holds code that allows a Certificate Transparency Log to be run using a Trillian Log as its back-end -- see below.

Trillian CT Personality

The trillian/ subdirectory holds code and scripts for running a CT Log based on the Trillian general transparency Log.

The main code for the CT personality is held in trillian/ctfe; this code responds to HTTP requests on the CT API paths and translates them to the equivalent gRPC API requests to the Trillian Log.

This obviously relies on the gRPC API definitions at github.com/google/trillian; the code also uses common libraries from the Trillian project for:

  • exposing monitoring and statistics via an interface and corresponding Prometheus implementation (github.com/google/trillian/monitoring/...)
  • dealing with cryptographic keys (github.com/google/trillian/crypto/...).

The trillian/integration/ directory holds scripts and tests for running the whole system locally. In particular:

  • trillian/integration/ct_integration_test.sh brings up local processes running a Trillian Log server, signer and a CT personality, and exercises the complete set of RFC 6962 API entrypoints.
  • trillian/integration/ct_hammer_test.sh brings up a complete system and runs a continuous randomized test of the CT entrypoints.

These scripts require a local database instance to be configured as described in the Trillian instructions.

Working on the Code

Developers who want to make changes to the codebase need some additional dependencies and tools, described in the following sections. The Travis configuration for the codebase is also useful reference for the required tools and scripts, as it may be more up-to-date than this document.

Rebuilding Generated Code

Some of the CT Go code is autogenerated from other files:

  • Protocol buffer message definitions are converted to .pb.go implementations.
  • A mock implementation of the Trillian gRPC API (in trillian/mockclient) is created with GoMock.

Re-generating mock or protobuffer files is only needed if you're changing the original files; if you do, you'll need to install the prerequisites:

and run the following:

go generate -x ./...  # hunts for //go:generate comments and runs them

Updating Vendor Code

The codebase includes a couple of external projects under the vendor/ subdirectory, to ensure that builds use a fixed version (typically because the upstream repository does not guarantee back-compatibility between the tip master branch and the current stable release). See instructions in the Trillian repo for how to update vendored subtrees.

Running Codebase Checks

The scripts/presubmit.sh script runs various tools and tests over the codebase.

# Install gometalinter and all linters
go get -u github.com/alecthomas/gometalinter
gometalinter --install

# Run code generation, build, test and linters
./scripts/presubmit.sh

# Run build, test and linters but skip code generation
./scripts/presubmit.sh  --no-generate

# Or just run the linters alone:
gometalinter --config=gometalinter.json ./...