Submit issued certificates to CT logs #454

rolandshoemaker · 2015-07-10T22:19:37Z

This PR adds a new service, Publisher, which exists to submit issued certificates to various Certificate Transparency logs. Once submitted the Publisher will also parse and store the returned SCT (Signed Certificate Timestamp) receipts that are used to prove inclusion in a specific log in the SA database. A SA migration adds the new SCT receipt table.

The Publisher only exposes one method, SubmitToCT, which is called in a goroutine by ca.IssueCertificate as to not block any other issuance operations. This method will iterate through all of the configured logs attempting to submit the certificate, and any required intermediate certificates, to them. If a submission to a log fails it will be retried the pre-configured number of times and will either use a back-off set in a Retry-After header or a pre-configured back-off between submission attempts.

This changeset is the first of a number of changes ending with serving SCT receipts in OCSP responses and purposefully leaves out the following pieces for follow-up PRs.

A fake CT server for integration testing
A external tool to search the database for certificates lacking a full set of SCT receipts
A method to construct X.509 v3 extensions containing receipts for the OCSP responder
Returned SCT signature verification (beyond just checking that the signature is of the correct type so we aren't just serving arbitrary binary blobs to clients)

Resolves #95.

rolandshoemaker · 2015-07-10T22:24:43Z

Note I want to make some changes to how the chain (for submission) is generated, this will really depend on how individual CAs issued leafs chain back up to their root. Basically every element of the chain should be included in the submission until the end of the chain or an element in the target logs root pool is encountered. This will require a bit of hackery, open to any suggestions.

jcjones · 2015-07-11T01:21:58Z

rpc/rpc-wrappers.go

+
+// SubmitToCT sends a request to submit a certifcate to CT logs
+func (pub PublisherAuthorityClient) SubmitToCT(cert *x509.Certificate) (err error) {
+	_, err = pub.rpc.DispatchSync(MethodSubmitToCT, cert.Raw)


Does this really need to be designed to be synchronous?

Not for the current implementation, in the future we will probably want either another external tool (or to integrate the functionality into one of the existing ones) that can scan the database for certificates that haven't been submitted to all configured logs and attempt to re-submit them. This tool would probably want it to be synchronous so that it can properly retrieve any error messages that are passed back.

Actually, I think it should be asynchronous, and assume a durable queue. No responses. The publisher's job is then to chew through that queue at whatever rate it can.
We'll want the publisher to only ack a message off the queue though if it is successful or has an exception -- not if it crashes, etc.

jcjones · 2015-07-11T01:34:55Z

I'd like to be able to trigger this code -- somehow -- on already-signed certificates. Maybe not on day 1, but be able to easily make a tool handle that can, for a given cert, publish it. Are we going to need to produce something that can, for a given cert, decide if it's been published yet and publish it? Or maybe we can mark the RPC queue as durable for that?

I appreciate that this code uses a goroutine to avoid affecting issuance.

(And maybe we should use the goroutine trick with OCSP actually -- I opened #458 to consider)

rolandshoemaker · 2015-07-11T01:43:07Z

I'm currently working on storing the SCTs returned from logs in the certificateStatus table, which will be needed for generating the x509v3 extensions we will need to append to OCSP responses. The lack of a SCT would then be an indicator that the certificate hasn't been published and needs to be.

I do like the durable queue idea although since we will need to store the SCT for future operations we could run into a situation where the tool we run to check for unsubmitted certs would have to know which certs don't have an SCT due to failure (incl. RPC failure) and which certs still have SubmitToCT calls that haven't been ACK'd in the RPC queue in order not to double submit.

rolandshoemaker · 2015-07-14T02:23:40Z

So I've made the submission chain construction better, it would be nice to have an external tool that can take the full chain (from issuer cert to its root) and a collected root pool from configured CT logs (available via an endpoint) and create the shortest valid submission chain for the SubmitToCT. This should be relatively easy to implement and I'll get to it shortly.

I've also added a new SQL table, sctReceipts, instead of appending the receipts to an existing table and the required SQL methods to Add SCT receipts, get a single receipt, and get multiple receipts that will be needed by the OCSP-Updater to form the x509v3 extensions (in an other PR).

I've come around to @jcjones way of thinking on sync vs async and durability of the queue. I still think we will need an external tool to scan the certificate database and check that SCT receipts for all configured CT logs are present (and submitted if not) but the overlap between this and certificates waiting un ACK'd in the RPC queue should be small enough that it doesn't matter (especially since this will probably only needed to be run if there are network issues, troubles for log server, etc, ).

I'll rework the RPC stuff as suggested and add a tool to scan the database for certificates that don't have SCT receipts for all configured logs. Hopefully then we will be done on the feature front...!

rolandshoemaker · 2015-07-14T07:15:49Z

I've added the CT chain bundle generation tool, boulder-gen-ct-bundle, which, based on the configured CT logs and a provided chain of certificates, will download the root pools for those logs and check the shortest shared valid chain that can be submitted after issued certificates. This chain is then written out in a format that the Publisher config object uses.

When using a chain with a root signed by something in the root pool the output looks something like this (using the LE website certificate as an example, with truncated output)

$ boulder-gen-ct-bundle -cert leaf.pem -cert inter.pem -cert root.pem -config -out ct-bundle.der
2015/07/13 23:48:01 [DEBUG] Parsed OID [2 23 140 1 2 1]
2015/07/13 23:48:01 [DEBUG] Parsed OID [1 2 3 4]
# ct.googleapis.com/pilot
    Downloading roots from https://ct.googleapis.com/pilot/ct/v1/get-roots
    Downloaded 389 certificates
    Testing chain validity with downloaded log root pool
        letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1 -> constructed root pool: Valid!
        letsencrypt.org -> TrustID Server CA A52 -> constructed root pool: Invalid!

    Bundle size for ct.googleapis.com/pilot: 3
# ct.googleapis.com/aviator
    Downloading roots from https://ct.googleapis.com/aviator/ct/v1/get-roots
    Downloaded 389 certificates
    Testing chain validity with downloaded log root pool
        letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1 -> constructed root pool: Valid!
        letsencrypt.org -> TrustID Server CA A52 -> constructed root pool: Invalid!

    Bundle size for ct.googleapis.com/aviator: 3
# ct.googleapis.com/rocketeer
    Downloading roots from https://ct.googleapis.com/rocketeer/ct/v1/get-roots
    Downloaded 389 certificates
    Testing chain validity with downloaded log root pool
        letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1 -> constructed root pool: Valid!
        letsencrypt.org -> TrustID Server CA A52 -> constructed root pool: Invalid!

    Bundle size for ct.googleapis.com/rocketeer: 3
# ct1.digicert-ct.com/log
    Downloading roots from https://ct1.digicert-ct.com/log/ct/v1/get-roots
    Downloaded 51 certificates
    Testing chain validity with downloaded log root pool
        letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1 -> constructed root pool: Invalid!

    !! Couldn't construct any valid chains, this may mean you haven't   !!
    !! provided the full chain or that this CT log doesn't contain a    !!
    !! root certificates that chain those provided. In the case of the  !!
    !! latter you should remove this log from your configuration since  !!
    !! your submissions will fail and be discarded.                     !!
# ct.izenpe.com
    Downloading roots from https://ct.izenpe.com/ct/v1/get-roots
    Downloaded 38 certificates
    Testing chain validity with downloaded log root pool
        letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1 -> constructed root pool: Invalid!

    !! Couldn't construct any valid chains, this may mean you haven't   !!
    !! provided the full chain or that this CT log doesn't contain a    !!
    !! root certificates that chain those provided. In the case of the  !!
    !! latter you should remove this log from your configuration since  !!
    !! your submissions will fail and be discarded.                     !!
# log.certly.io
    Downloading roots from https://log.certly.io/ct/v1/get-roots
    Downloaded 182 certificates
    Testing chain validity with downloaded log root pool
        letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1 -> constructed root pool: Valid!
        letsencrypt.org -> TrustID Server CA A52 -> constructed root pool: Invalid!

    Bundle size for log.certly.io: 3

# Shared bundle size: 3 certificates, letsencrypt.org -> TrustID Server CA A52 -> IdenTrust Commercial Root CA 1
# CT submission bundle has been written to ct-bundle.der

The failures for ct1.digicert-ct.com/log and ct.izenpe.com are due to the fact that these logs root pools are quite small and only seem to contain their own roots or those for closely associated services indicating that we probably shouldn't submit to them. We may also want to ignore log.certly.io and only submit to the Google logs, Certly seems relatively small (their log currently only contains ~16000 entries) and we may just flood them with traffic.

Using the actual LE issuance certs this currently will fail for all logs (presumably because it isn't cross signed yet) like this

$ boulder-gen-ct-bundle -cert le-x2.pem -cert le-root.pem  -out ct-bundle.der
# ct.googleapis.com/pilot
    Downloading roots from https://ct.googleapis.com/pilot/ct/v1/get-roots
    Downloaded 389 certificates
    Testing chain validity with downloaded log root pool
        Let's Encrypt Authority X2 -> ISRG Root X1 -> constructed root pool: Invalid!

    !! Couldn't construct any valid chains, this may mean you haven't   !!
    !! provided the full chain or that this CT log doesn't contain a    !!
    !! root certificates that chain those provided. In the case of the  !!
    !! latter you should remove this log from your configuration since  !!
    !! your submissions will fail and be discarded.                     !!

!! Couldn't find any valid chains for configured logs, this may     !!
!! mean you haven't provided the full chain or that this CT log     !!
!! doesn't contain a root certificates that chain those provided.   !!
!! The bundle will still be written out but you should only use     !!
!! this if you really know what you are doing!                      !!

# Shared bundle size: 2 certificates, Let's Encrypt Authority X2 -> ISRG Root X1
# CT submission bundle has been written to ct-bundle.der

jsha · 2015-08-07T23:02:36Z

core/objects.go

+	LogID []byte `db:"logID",json:"rpcLogID"`
+	// Timestamp (in ms since unix epoc) at which the SCT was issued
+	Timestamp uint64 `db:"timestamp",json:"rpcTimestamp"`
+	// For future extensions to the protocol


Is this used today? If not, remove it I think.

This will be required to assemble the SCT we want to put in OCSP responses in the future (but yes not now).

rolandshoemaker · 2015-08-11T23:19:15Z

Things that still need to be done

Re-write use of RPC to assume durable queues and only ack messages on submission and storage

Things deferred for another changeset

Add tool that scans the database for certificates that haven't been submitted to all logs and submit them

jcjones · 2015-08-27T03:40:20Z

test/boulder-config.json

+  },
+
+  "publisher": {
+    "dbDriver": "sqlite3",


This is a rebase error, I guess

jsha · 2015-08-28T23:08:51Z

This branch has merge conflicts now, in addition to the above comments about pulling out the durable queue stuff.

jsha · 2015-09-16T19:09:38Z

publisher/publisher.go

+}
+
+func postJSON(client *http.Client, uri string, data []byte, respObj interface{}) (*http.Response, error) {
+	if !strings.HasPrefix(uri, "http://") && !strings.HasPrefix(uri, "https://") {


I really don't like this auto-adding https: prefix. Can you remove it and just require that uri be a valid absolute URL?

Whoops, thought I had done this in the last changeset.

jsha · 2015-09-16T23:26:45Z

ca/certificate-authority_test.go

@@ -246,6 +256,8 @@ func TestIssueCertificate(t *testing.T) {
 	defer ctx.cleanUp()
 	ca, err := NewCertificateAuthorityImpl(ctx.caDB, ctx.caConfig, ctx.fc, caCertFile)
 	test.AssertNotError(t, err, "Failed to create CA")
+	pub, _ := publisher.NewPublisherImpl(publisher.CTConfig{})


Why do we need real publishers rather than mocks here?

jmhodges · 2015-09-17T22:25:13Z

sa/_db/migrations/20150916173200_SCTReceipts.sql

+  `sctVersion` tinyint(1) NOT NULL,
+  `logID` varchar(255) NOT NULL,
+  `timestamp` bigint(20) NOT NULL,
+  `extensions` mediumblob,


blob for both of these

letsencryptbot · 2015-09-17T23:32:25Z

go vet ./... core/objects.go:565: struct field tag `db:"logID` not compatible with reflect.StructTag.Get: bad syntax for struct tag value exit status 1 Status ID=69db7718fe6fecab8c9b85e908bc5c631fa585a0 State=failure Context=test/vet URL=https://travis-ci.org/letsencrypt/boulder/builds/80931755 Description= {"state":"failure","context":"test/vet","target_url":"https://travis-ci.org/letsencrypt/boulder/builds/80931755"} [!] FAILURE: go vet ./...

letsencryptbot · 2015-09-17T23:32:37Z

go vet ./... core/objects.go:565: struct field tag `db:"logID` not compatible with reflect.StructTag.Get: bad syntax for struct tag value exit status 1 Status ID=69db7718fe6fecab8c9b85e908bc5c631fa585a0 State=failure Context=test/vet URL=https://travis-ci.org/letsencrypt/boulder/builds/80931755 Description= {"state":"failure","context":"test/vet","target_url":"https://travis-ci.org/letsencrypt/boulder/builds/80931755"} [!] FAILURE: go vet ./...

letsencryptbot · 2015-09-17T23:33:50Z

go vet ./... core/objects.go:565: struct field tag `db:"logID` not compatible with reflect.StructTag.Get: bad syntax for struct tag value exit status 1 Status ID=69db7718fe6fecab8c9b85e908bc5c631fa585a0 State=failure Context=test/vet URL=https://travis-ci.org/letsencrypt/boulder/builds/80931755 Description= {"state":"failure","context":"test/vet","target_url":"https://travis-ci.org/letsencrypt/boulder/builds/80931755"} [!] FAILURE: go vet ./...

jmhodges · 2015-09-17T23:38:13Z

LGTM

I am okay with this. The next thing to do is to implement backfilling per my email just now about our certificate numbering scheme.

Adds a new service, Publisher, which exists to submit issued certificates to various Certificate Transparency logs. Once submitted the Publisher will also parse and store the returned SCT (Signed Certificate Timestamp) receipts that are used to prove inclusion in a specific log in the SA database. A SA migration adds the new SCT receipt table. The Publisher only exposes one method, SubmitToCT, which is called in a goroutine by ca.IssueCertificate as to not block any other issuance operations. This method will iterate through all of the configured logs attempting to submit the certificate, and any required intermediate certificates, to them. If a submission to a log fails it will be retried the pre-configured number of times and will either use a back-off set in a Retry-After header or a pre-configured back-off between submission attempts. This changeset is the first of a number of changes ending with serving SCT receipts in OCSP responses and purposefully leaves out the following pieces for follow-up PRs. * A fake CT server for integration testing * A external tool to search the database for certificates lacking a full set of SCT receipts * A method to construct X.509 v3 extensions containing receipts for the OCSP responder * Returned SCT signature verification (beyond just checking that the signature is of the correct type so we aren't just serving arbitrary binary blobs to clients) Resolves #95.

Submit issued certificates to CT logs

jcjones reviewed Jul 11, 2015
View reviewed changes

jcjones mentioned this pull request Jul 11, 2015

Consider using Goroutine for post-issuance OCSP response generation #458

Closed

rolandshoemaker added the blocked label Jul 22, 2015

rolandshoemaker added needs-revision and removed blocked labels Aug 7, 2015

jsha reviewed Aug 7, 2015
View reviewed changes

rolandshoemaker force-pushed the ct-submission branch from 97e43f3 to 0165646 Compare August 26, 2015 20:56

rolandshoemaker added r? and removed needs-revision labels Aug 26, 2015

jcjones reviewed Aug 27, 2015
View reviewed changes

test/boulder-config.json

},

"publisher": {

"dbDriver": "sqlite3",

Copy link

Contributor

jcjones Aug 27, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a rebase error, I guess

jsha added needs-revision and removed r? labels Aug 28, 2015

jsha reviewed Sep 16, 2015
View reviewed changes

jsha mentioned this pull request Sep 16, 2015

Move MockSA from WFE to mocks. #807

Merged

jsha reviewed Sep 16, 2015
View reviewed changes

jsha added the r=jsha label Sep 17, 2015

jmhodges reviewed Sep 17, 2015
View reviewed changes

rolandshoemaker force-pushed the ct-submission branch from 69db771 to b53f8bb Compare September 17, 2015 23:34

jmhodges added the r=jmhodges label Sep 17, 2015

rolandshoemaker force-pushed the ct-submission branch from b53f8bb to ff6eca7 Compare September 18, 2015 01:16

jsha removed the r? label Sep 18, 2015

jsha added a commit that referenced this pull request Sep 18, 2015

Merge pull request #454 from letsencrypt/ct-submission

b26eaae

Submit issued certificates to CT logs

jsha merged commit b26eaae into master Sep 18, 2015

jsha mentioned this pull request Sep 18, 2015

Add migrate-up script to easily do migrations. #814

Merged

bifurcation deleted the ct-submission branch October 21, 2015 11:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Submit issued certificates to CT logs #454

Submit issued certificates to CT logs #454

rolandshoemaker commented Jul 10, 2015

rolandshoemaker commented Jul 10, 2015

jcjones Jul 11, 2015

rolandshoemaker Jul 11, 2015

jcjones Jul 11, 2015

jcjones commented Jul 11, 2015

rolandshoemaker commented Jul 11, 2015

rolandshoemaker commented Jul 14, 2015

rolandshoemaker commented Jul 14, 2015

jsha Aug 7, 2015

rolandshoemaker Aug 7, 2015

rolandshoemaker commented Aug 11, 2015

jcjones Aug 27, 2015

jsha commented Aug 28, 2015

jsha Sep 16, 2015

rolandshoemaker Sep 16, 2015

jsha Sep 16, 2015

jmhodges Sep 17, 2015

letsencryptbot commented Sep 17, 2015

letsencryptbot commented Sep 17, 2015

letsencryptbot commented Sep 17, 2015

jmhodges commented Sep 17, 2015

Submit issued certificates to CT logs #454

Submit issued certificates to CT logs #454

Conversation

rolandshoemaker commented Jul 10, 2015

rolandshoemaker commented Jul 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcjones commented Jul 11, 2015

rolandshoemaker commented Jul 11, 2015

rolandshoemaker commented Jul 14, 2015

rolandshoemaker commented Jul 14, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rolandshoemaker commented Aug 11, 2015

Choose a reason for hiding this comment

jsha commented Aug 28, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

letsencryptbot commented Sep 17, 2015

letsencryptbot commented Sep 17, 2015

letsencryptbot commented Sep 17, 2015

jmhodges commented Sep 17, 2015