From 19ac2bf0442b8d9e1ffcbe594d5befb04c1c7031 Mon Sep 17 00:00:00 2001 From: Nils Lehnen <30603423+iderex@users.noreply.github.com> Date: Wed, 12 Aug 2026 03:26:47 +0200 Subject: [PATCH] Say which commit a measurement in an answer was produced at (#123) An answer quotes a command and its output and says nothing about which version of the code the command ran against. Record 0004 lets that code be removed entirely afterwards, and when it is, the answer keeps its numbers and loses the thing that produced them silently, because the record still reads as complete. Record 0016 takes the field of the three options the issue set out, with the reasons and what each rejected option would have cost. It supersedes 0008 for that one thing and answers to 0013, so the field is optional and an absence is never refused. Record 0008 gains the one line record 0000 allows a superseded record to take. Measurement-Commit carries the object name in full, which is what record 0004 already writes for the neighbouring case. The runner refuses a value that is not the shape of an object name. The abbreviation is the fixture because it is the mistake somebody actually makes: it is what git prints, it resolves on the machine it was copied on, it stops resolving on a repository that has grown into a collision, and it cannot be told from a typo. Two things are written at the check rather than left to a green run. Nothing here asks git whether the object exists, is a commit, or has anything to do with the command in the answer, because the runner reads a checkout and opens no connection. And the absence stays unrefusable, so the field makes the fact recordable and never guaranteed. The one record on the board is left exactly as it was. Record 0013 says a record already on the default branch is not edited, not migrated and not marked, and adding the field to it afterwards is the migration that rule refuses. The template names the field in its prose rather than carrying it in its header, for the reason Answer-Written is named the same way. Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com> --- docs/decisions/0008-the-experiment-record.md | 7 +- ...-an-answer-names-the-commit-it-measured.md | 113 ++++++++++++++++++ docs/experiment-template.md | 13 +- internal/check/check.go | 1 + internal/check/measurement.go | 98 +++++++++++++++ .../expected | 4 + .../expected-refusals | 1 + .../near-neighbour | 1 + .../tree/experiments/one/EXPERIMENT.md | 18 +++ .../record-with-a-measurement-commit/expected | 4 + .../expected-refusals | 0 .../tree/experiments/one/EXPERIMENT.md | 18 +++ 12 files changed, 273 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/0016-an-answer-names-the-commit-it-measured.md create mode 100644 internal/check/measurement.go create mode 100644 testdata/cases/record-with-a-measurement-commit-abbreviated/expected create mode 100644 testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals create mode 100644 testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour create mode 100644 testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md create mode 100644 testdata/cases/record-with-a-measurement-commit/expected create mode 100644 testdata/cases/record-with-a-measurement-commit/expected-refusals create mode 100644 testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md diff --git a/docs/decisions/0008-the-experiment-record.md b/docs/decisions/0008-the-experiment-record.md index 579b710..901f91d 100644 --- a/docs/decisions/0008-the-experiment-record.md +++ b/docs/decisions/0008-the-experiment-record.md @@ -1,8 +1,9 @@ # 0008. What an experiment record looks like -Superseded by 0014, which fixes what a slug may be, and by 0015, which adds the -field naming what an experiment needs beyond the runner. Everything below stands -as it was written. +Superseded by 0014, which fixes what a slug may be, by 0015, which adds the +field naming what an experiment needs beyond the runner, and by 0016, which adds +the field naming the commit a measurement was produced at. Everything below +stands as it was written. ## What was decided diff --git a/docs/decisions/0016-an-answer-names-the-commit-it-measured.md b/docs/decisions/0016-an-answer-names-the-commit-it-measured.md new file mode 100644 index 0000000..799f322 --- /dev/null +++ b/docs/decisions/0016-an-answer-names-the-commit-it-measured.md @@ -0,0 +1,113 @@ +# 0016. An answer names the commit its measurement was produced at + +## What was decided + +This record adds a field to the experiment record format record `0008` fixes, +and it supersedes that record for that one thing. Everything `0008` says stays +as it was written, and how the format grows at all is record `0013`, which this +answers to rather than restates. + +An answer that quotes a measurement carries `Measurement-Commit` in the header, +whose value is the object name of the commit the measurement was produced at, +written in full. + + Measurement-Commit: e5067b1d0f4a2c8b6e3a9d7c1b5f8a2e4d6c0b93 + +The field is optional, because record `0013` makes every field added after it +optional and an absence is never refused. What is refused is a value that is not +the shape of an object name, which is the whole of what a checkout can be asked. + +The full name rather than an abbreviation. Record `0004` already writes the +commit that removed an experiment's code with the full hash, for the reason an +abbreviation stops being unique as a repository grows, and a measurement is the +same shape one step earlier. A reader who cannot resolve the name they were +given is in the position this field exists to remove them from. + +Why a field rather than a sentence in the prose. An answer already names the +command, the platform, the architecture and the toolchain, and none of that is +read by anything. What is missing is the version of the code the command ran +against, and record `0004` lets that code be removed entirely afterwards. When +it is, the answer keeps its numbers and loses the thing that produced them, and +it does so silently, because the record still reads as complete. A field is +where the runner already looks. + +What this does not buy, written here because a green run will otherwise be read +as more than it is. The runner reads a checkout and opens no connection, so +nothing here asks git whether the object named is in this repository, or whether +it is a commit, or whether the command in the answer was ever run against it. A +record naming forty hexadecimal characters that resolve to nothing passes. +What the refusal converts is the case where the value could not be resolved by +anybody, which is the only case a checkout can separate from the rest. + +And the absence stays unrefusable. A record quoting a measurement and carrying +no `Measurement-Commit` is legal, so the field makes the fact recordable and +never guaranteed. That cost is record `0013`'s, paid deliberately, and the +things that make a field usual without making it required are the template, the +review, and this record. + +## What it applies to + +Every `EXPERIMENT.md` under `experiments/`, from the commit this record lands +on, and the check that reads the field. + +It applies to what is present. A record already on the default branch is not +edited, not migrated and not marked, which is record `0013` and is why the one +record on the board when this landed was left exactly as it was. + +It applies to the template, which names the field in its prose rather than +carrying it in its header, for the reason `Answer-Written` is named the same +way: a template that ships a field filled in teaches every new record to declare +something it has no value for yet. + +It does not apply to the decision records in `docs/decisions/`. Those carry the +four sections record `0000` fixes and change by superseding. + +It does not decide anything about a measurement produced somewhere other than +this repository. An answer measuring another project's code has the same +problem and a different answer, and nothing here should be read as covering it. + +## What else was considered + +A convention in the answer prose, in the words the record already uses, with +nothing reading it. + +Nothing at all, on the argument that a reader who wants the exact code has git +and the record's own history. + +A field carrying the date of the measurement rather than the commit. + +A field required of every record whose answer quotes a command. + +Naming the commit with an abbreviated hash. + +## What each rejected option would have cost + +A convention costs exactly what it is: nothing reads it. The failure this is +about is a record that still reads as complete after the thing that produced its +numbers has gone, and a convention is another sentence in the file that already +read as complete. It also drifts, because two authors writing the same +convention in their own words produce two shapes, and the first check anybody +writes over it has to be right about both. + +Nothing costs the reader the work, and it charges the reader least able to do +it. Somebody who was there can find the commit from the date and the log. +Somebody who was not is the person this whole record is for, and the answer they +get is that the numbers were true of some version of a directory that record +`0004` may have removed. + +A date costs the precision that makes the field worth having. Several commits +land on one day here, and the answer already carries `Answer-Written`, so a date +would duplicate a field that exists and still not say which code ran. + +Requiring the field costs the format its ability to grow, which is record +`0013`'s whole argument. It would also require the checker to decide what counts +as an answer quoting a measurement, which is a judgement about prose that no +reading of a tree makes: an answer holding a number and an answer holding a +number that was measured look the same to everything in this repository. + +An abbreviation costs uniqueness at the moment somebody needs it. A short name +resolves today and stops resolving when the repository grows into a collision, +and the reader who meets that is the one who came back years later, which is the +case the field exists for. It also cannot be told from a typo: seven characters +that resolve to nothing and seven characters somebody mistyped are the same +string. diff --git a/docs/experiment-template.md b/docs/experiment-template.md index 998664d..cfea49b 100644 --- a/docs/experiment-template.md +++ b/docs/experiment-template.md @@ -10,6 +10,14 @@ day the question below was written, as `YYYY-MM-DD`, and it does not move afterwards, because the listing sorts by it. Add `Answer-Written` in the same change that writes the answer. +Add `Measurement-Commit` in that same change where the answer quotes a +measurement, with the object name of the commit the measurement was produced at, +written in full. It is neither here nor in `Answer-Written`'s position because a +template that ships a field filled in teaches every new record to declare a +value it does not have yet. The code the measurement ran against may be removed +later, and without this the answer keeps its numbers and loses the thing that +produced them while still reading as complete. + `Needs-Hardware` is what this experiment needs beyond the runner, in words somebody deciding whether to reproduce it can act on, or `none`. It starts at `none` here because that is the right answer for almost every experiment. A test @@ -19,8 +27,9 @@ under `internal/hardware`, in a file whose name ends directory says the other is refused. The format is `docs/decisions/0008-the-experiment-record.md`, as added to by -`docs/decisions/0015-an-experiment-declares-the-harness-it-needs.md`. This file -is a convenience and those records are the authority. +`docs/decisions/0015-an-experiment-declares-the-harness-it-needs.md` and by +`docs/decisions/0016-an-answer-names-the-commit-it-measured.md`. This file is a +convenience and those records are the authority. ## Question diff --git a/internal/check/check.go b/internal/check/check.go index fd8ee30..29063df 100644 --- a/internal/check/check.go +++ b/internal/check/check.go @@ -460,6 +460,7 @@ func walkExperiments(root string, res *Result) error { res.Refusals = append(res.Refusals, refuseQuestion(record, data)...) res.Refusals = append(res.Refusals, refuseState(record, data)...) res.Refusals = append(res.Refusals, refuseHeaderDates(record, data)...) + res.Refusals = append(res.Refusals, refuseMeasurementCommit(record, data)...) res.Refusals = append(res.Refusals, refuseDates(record, data, res.Now)...) res.Refusals = append(res.Refusals, refusePromotion(record, data)...) // The only rule here that reads the directory as well as the record, diff --git a/internal/check/measurement.go b/internal/check/measurement.go new file mode 100644 index 0000000..5dba8ce --- /dev/null +++ b/internal/check/measurement.go @@ -0,0 +1,98 @@ +package check + +import "fmt" + +// FieldMeasurementCommit is the object name of the commit a measurement in the +// answer was produced at. Record 0016 adds it and record 0013 makes it +// optional, as it makes every field added after it. +// +// It is declared here rather than beside the four record 0008 fixes, and that +// is the shape headerDateFields already argues for at its own list: a field a +// later record adds arrives with the check that reads it, so a checker built +// before the field is unaware of it and a field with no check has nowhere to +// hide. +const FieldMeasurementCommit = "Measurement-Commit" + +// RecordMeasurementCommitIsNotACommit refuses a measurement commit that is not +// the shape of an object name. +// +// An answer quotes a command and its output and says nothing about which +// version of the code the command ran against. Record 0004 lets that code be +// removed afterwards, and when it is, the answer keeps its numbers and loses +// the thing that produced them without reading as any less complete. +// +// The abbreviation is the value worth the fixture, because it is the mistake +// somebody actually makes: the short name is what git prints in a log, it is +// what a hand copies, it resolves on the machine it was copied on, and it stops +// resolving on a repository that has grown into a collision. It also cannot be +// told from a typo, since a short name that resolves to nothing and a mistyped +// one are the same string. +// +// A declared field with nothing after the colon is refused here too, for the +// reason the date refusal gives: record 0013 makes absence legal and an empty +// declaration a different statement, and a commit field with no value claims +// there is a commit rather than claiming there is none. +const RecordMeasurementCommitIsNotACommit = "record-measurement-commit-is-not-a-commit" + +// The object name lengths this accepts, and the reason there are two. +// +// Every object in this repository is named in the first of them today. The +// second is what the same repository would name its objects in under the other +// hash git can be told to use, and refusing it would make this check a decision +// about the object format, which record 0016 does not take and which is not a +// question about a measurement. +const ( + shortObjectName = 40 + longObjectName = 64 +) + +// refuseMeasurementCommit holds a measurement commit to the shape of an object +// name. +// +// WHAT IT CANNOT SAY, and this is the whole of the residual. Whether the object +// is in this repository, whether it is a commit rather than a blob, and whether +// the command quoted in the answer was ever run against it. The runner reads a +// checkout and opens no connection, and asking git any of those would cost it +// the dependency surface record 0001 chose. A value of forty hexadecimal +// characters that resolve to nothing passes here, so a green run says the name +// is resolvable in principle rather than that it resolves. +// +// A record whose bytes do not parse as a record is not judged here, for the +// reason every other header rule gives: nothing can read a field out of a file +// that has no header. +func refuseMeasurementCommit(path string, data []byte) []Refusal { + record, err := ParseRecord(data) + if err != nil { + return nil + } + + named, present := record.Field(FieldMeasurementCommit) + if !present { + // Absence is never a refusal. Record 0013 fixes that, and most + // records carry no measurement at all. + return nil + } + if isObjectName(named) { + return nil + } + + return []Refusal{{ + Property: RecordMeasurementCommitIsNotACommit, + Subject: path, + Detail: fmt.Sprintf("its %s is %q, and record 0016 writes an object name in full, which is %d or %d characters of lowercase hexadecimal", + FieldMeasurementCommit, named, shortObjectName, longObjectName), + }} +} + +// isObjectName says whether a value is the shape git names an object in. +func isObjectName(value string) bool { + if len(value) != shortObjectName && len(value) != longObjectName { + return false + } + for _, r := range value { + if (r < '0' || r > '9') && (r < 'a' || r > 'f') { + return false + } + } + return true +} diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/expected b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected new file mode 100644 index 0000000..466fe70 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected @@ -0,0 +1,4 @@ +directories 1 +records 1 +experiments present +decisions absent diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals new file mode 100644 index 0000000..7da7ef0 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals @@ -0,0 +1 @@ +record-measurement-commit-is-not-a-commit diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour b/testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour new file mode 100644 index 0000000..69c87c6 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour @@ -0,0 +1 @@ +record-with-a-measurement-commit diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md b/testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md new file mode 100644 index 0000000..b19e620 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md @@ -0,0 +1,18 @@ +Slug: one +State: answered +Question-Written: 2026-08-01 +Answer-Written: 2026-08-02 +Measurement-Commit: e5067b1 + +## Question + +Does reading a tree of a thousand records cost more than a second? + +## Method + +Built the tree and timed the walk. + +## Answer + +No. The walk took eleven seconds, and the cost is in reading the records +rather than in walking the tree. diff --git a/testdata/cases/record-with-a-measurement-commit/expected b/testdata/cases/record-with-a-measurement-commit/expected new file mode 100644 index 0000000..466fe70 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit/expected @@ -0,0 +1,4 @@ +directories 1 +records 1 +experiments present +decisions absent diff --git a/testdata/cases/record-with-a-measurement-commit/expected-refusals b/testdata/cases/record-with-a-measurement-commit/expected-refusals new file mode 100644 index 0000000..e69de29 diff --git a/testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md b/testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md new file mode 100644 index 0000000..2d68aa5 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md @@ -0,0 +1,18 @@ +Slug: one +State: answered +Question-Written: 2026-08-01 +Answer-Written: 2026-08-02 +Measurement-Commit: e5067b1d0f4a2c8b6e3a9d7c1b5f8a2e4d6c0b93 + +## Question + +Does reading a tree of a thousand records cost more than a second? + +## Method + +Built the tree and timed the walk. + +## Answer + +No. The walk took eleven seconds, and the cost is in reading the records +rather than in walking the tree.