Skip to content

Add outline of static-ct fsck tool#383

Merged
AlCutter merged 6 commits into
transparency-dev:mainfrom
AlCutter:static_ct_fsck
Jun 27, 2025
Merged

Add outline of static-ct fsck tool#383
AlCutter merged 6 commits into
transparency-dev:mainfrom
AlCutter:static_ct_fsck

Conversation

@AlCutter

@AlCutter AlCutter commented Jun 25, 2025

Copy link
Copy Markdown
Collaborator

This PR adds a first-cut outline of a correctness/self-integrity checking tool (fsck) for C2SP static-ct compliant logs.

The tool verifies that:

  • The checkpoint is authentic
  • The merkle root in the checkpoint can be reproduced using the entries in the log
  • The internal node tiles of the log are correct
  • All issuer fingerprints seen within the entry bundles correspond to a static resource under issuer/ (although these intermediates are not parsed, nor are the chains they form verified
$ go run ./cmd/fsck \
    --storage_url=https://storage.googleapis.com/static-ct-staging-arche2025h1-bucket/ \
    --origin=arche2025h1.staging.ct.transparency.dev \
    --public_key="MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEn7khjUQH1H3NJ/C8QmmBgzoNTptlH6hT5bgiQ6mQcYYg5KZoe4ZK4xCszXu4NH5NiLaDH0wHKsvg3RIQ+TTaag==" \
    -N 2000 \
    -v 1
I0625 17:13:04.778133  210195 main.go:191] Using verifier string: arche2025h1.staging.ct.transparency.dev+cd333cb1+BTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJ+5IY1EB9R9zSfwvEJpgYM6DU6bZR+oU+W4IkOpkHGGIOSmaHuGSuMQrM17uDR+TYi2gx9MByrL4N0SEPk02mo=
I0625 17:13:04.835982  210195 fsck.go:56] Fsck: checkpoint:
arche2025h1.staging.ct.transparency.dev
1425162604
caEyaP+hTqoxjjGhZ5onK1NoIVuR30be8Cfi/e/0Zdc=

— arche2025h1.staging.ct.transparency.dev zTM8sQAAAZeoFCdNBAMARzBFAiEAm/Cxakit7UM0q150E1e14q50wdDGVxxt3aCxSlrLzG8CICr9SsiQoSjFVqKtsqIXBifO4mP5qoSKF8InqnK/C+bL
I0625 17:13:04.836251  210195 fsck.go:61] Fsck: checking log of size 1425162604
I0625 17:13:04.838503  210195 stream.go:89] stream.EntryBundles: streaming [0, 1425162604)
I0625 18:19:34.424004  210195 stream.go:117] stream.EntryBundles: exiting
I0625 18:19:35.915457  210195 fsck.go:114] Successfully fsck'd log with size 1425162604 and root caEyaP+hTqoxjjGhZ5onK1NoIVuR30be8Cfi/e/0Zdc= (71a13268ffa14eaa318e31a1679a272b5368215b91df46def027e2fdeff465d7)

@AlCutter
AlCutter marked this pull request as ready for review June 26, 2025 10:30
@AlCutter AlCutter added the enhancement New feature or request label Jun 26, 2025
@AlCutter
AlCutter requested a review from roger2hk June 26, 2025 10:31
@AlCutter
AlCutter force-pushed the static_ct_fsck branch 2 times, most recently from 4775644 to 95d018f Compare June 26, 2025 12:14
@AlCutter
AlCutter requested a review from phbnf June 27, 2025 09:30
Comment thread cmd/fsck/main.go Outdated
)

var (
storageURL = flag.String("storage_url", "", "Base tlog-tiles URL")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, maybe rename to "monitoring_url" for consistency with the specs?

Comment thread cmd/fsck/main.go Outdated
}
v := verifierFromFlags()
if *origin == "" {
*origin = v.Name()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can remove this check? verifierFromFlags will fail if origin is empty.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot, ta!

Comment thread cmd/fsck/main.go Outdated
n := 0
work := make(chan []byte, 10)
eg := errgroup.Group{}
for range 10 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be configurable? Or maybe set to the same value (or half?) that N is at?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assumption was that there weren't going to be (relative to the number of tiles/entrybundles just fetched) very many issuers, so some small amount of parallelism was "good enough", making it N is fine too, though - done!

Comment thread cmd/fsck/main.go
return func(bundle []byte) ([][]byte, error) {
r := make([][]byte, 0, layout.EntryBundleWidth)
b := cryptobyte.String(bundle)
for i := 0; i < layout.EntryBundleWidth && !b.Empty(); i++ {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should be able to use something like

func (t *EntryBundle) UnmarshalText(raw []byte) error {
or
func (t *Entry) UnmarshalText(raw []byte) error {
It might not work out of the box, but that's working as intended: I'd like to understand what we need these parsing functions for exactly before having one to rule them all. If you can't find an easy way to use any of these methods, put the parsing method you need over there, and at least they will all be in the same file.

Now, here's an easy counter argument: this is a check tool..... so maybe it should reimplement everything?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was my original hope, too :)

What I needed is a small subset of the functionality of both of those methods - I want (only):
a. the preimages of the entries to recreate the leaf hashes, and
b. the list of issuer fingerprints for each.

The reason I ended up doing it this way rather than using those Unmarshal funcs is that I'm trying to avoid parsing/copying the same bytes twice on the premise that this tool is going to process literally billions of entries (and terrabytes of data) in a single run.

I did briefly think about adding a new UnmarshalEntries func which would go directly from a serialised entry bundle to a []Entry, but I'd still end up having to effectively process all the bytes again in order to re-construct the entry preimage for the hash, so I abandoned that too.

@AlCutter
AlCutter merged commit dbc8b1e into transparency-dev:main Jun 27, 2025
7 checks passed
@AlCutter
AlCutter deleted the static_ct_fsck branch June 27, 2025 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants