Add outline of static-ct fsck tool#383
Conversation
4775644 to
95d018f
Compare
| ) | ||
|
|
||
| var ( | ||
| storageURL = flag.String("storage_url", "", "Base tlog-tiles URL") |
There was a problem hiding this comment.
nit, maybe rename to "monitoring_url" for consistency with the specs?
| } | ||
| v := verifierFromFlags() | ||
| if *origin == "" { | ||
| *origin = v.Name() |
There was a problem hiding this comment.
I think you can remove this check? verifierFromFlags will fail if origin is empty.
| n := 0 | ||
| work := make(chan []byte, 10) | ||
| eg := errgroup.Group{} | ||
| for range 10 { |
There was a problem hiding this comment.
Should this be configurable? Or maybe set to the same value (or half?) that N is at?
There was a problem hiding this comment.
The assumption was that there weren't going to be (relative to the number of tiles/entrybundles just fetched) very many issuers, so some small amount of parallelism was "good enough", making it N is fine too, though - done!
| return func(bundle []byte) ([][]byte, error) { | ||
| r := make([][]byte, 0, layout.EntryBundleWidth) | ||
| b := cryptobyte.String(bundle) | ||
| for i := 0; i < layout.EntryBundleWidth && !b.Empty(); i++ { |
There was a problem hiding this comment.
I think you should be able to use something like
ortesseract/internal/types/staticct/staticct.go
Line 191 in 024f1e9
Now, here's an easy counter argument: this is a check tool..... so maybe it should reimplement everything?
There was a problem hiding this comment.
That was my original hope, too :)
What I needed is a small subset of the functionality of both of those methods - I want (only):
a. the preimages of the entries to recreate the leaf hashes, and
b. the list of issuer fingerprints for each.
The reason I ended up doing it this way rather than using those Unmarshal funcs is that I'm trying to avoid parsing/copying the same bytes twice on the premise that this tool is going to process literally billions of entries (and terrabytes of data) in a single run.
I did briefly think about adding a new UnmarshalEntries func which would go directly from a serialised entry bundle to a []Entry, but I'd still end up having to effectively process all the bytes again in order to re-construct the entry preimage for the hash, so I abandoned that too.
This PR adds a first-cut outline of a correctness/self-integrity checking tool (
fsck) for C2SPstatic-ctcompliant logs.The tool verifies that:
issuer/(although these intermediates are not parsed, nor are the chains they form verified