Skip to content

Make the configuration hash the same on every machine - #303

Merged
adrpo merged 1 commit into
masterfrom
fix-confighash-machine-independent
Aug 11, 2026
Merged

Make the configuration hash the same on every machine#303
adrpo merged 1 commit into
masterfrom
fix-confighash-machine-independent

Conversation

@adrpo

@adrpo adrpo commented Aug 11, 2026

Copy link
Copy Markdown
Member

Two machines running master at the same time both claimed the same 20 libraries and tested them twice, which is what #295 exists to prevent:

     host      | branch |  state  | count
---------------+--------+---------+-------
 ryzen-5950x-1 | master | running |    96
 ryzen-5950x-2 | master | running |    20

 libraries_claimed_by_both: 20

Not the compiler and not the library: both machines report OMCompiler v1.28.0-dev.294+gce7a5ed7bc and the same libversion (there had been no omc commit for seven hours). What differed was the confighash, which is part of the claim key (branch, libname, libversion, omcversion, confighash):

ClaRa | ryzen-5950x-1 | 1.9.0 (https://github.com/xr... | 1132175474
ClaRa | ryzen-5950x-2 | 1.9.0 (https://github.com/xr... |  247072667

Why

The hash is taken over the library's configuration, and by that point the configuration holds the reference files as an absolute path:

destinationReal = os.path.realpath(destination)
c["referenceFiles"] = destinationReal
...
confighash = strToHashInt(str(c) + hashReferenceFiles(conf["referenceFiles"]))

So two machines whose workspaces sit in different places — or whose paths resolve through different symbolic links — hash an identical configuration differently.

This was harmless for as long as it existed: each machine had its own sqlite3.db, so the hash only ever had to be stable on the machine that produced it. Sharing one database is what made it matter, and it breaks two things at once:

The fix

The path contributes nothing the hash needs. The contents of the reference files are hashed on the very next line by hashReferenceFiles(), and the same files in a different directory are the same test. So the path is replaced by a constant before hashing.

Checked

The same configuration under two different reference directories:

before   machineA 3181697168  machineB 2732211752  -> DIFFERENT (both claim the library)
after    machineA 1884809167  machineB 1884809167  -> SAME (machines agree)
after    reference file changed          -> DIFFERENT (correct)

so the hash stops depending on where the files are while still depending on what is in them. Plus a full configs/sanityCheck.json run with the CI assertion passing.

One deliberate consequence

Every confighash changes value once. The first run after this sees no matching rows and treats every library as untested, so it tests everything once. From then on the machines agree — and until then they duplicate work anyway, which is the bug.


Generated by Claude Code.

Two machines testing the same library, with the same compiler and the same
library version, claimed it both and tested it both. The claim in #295 is keyed
by (branch, libname, libversion, omcversion, confighash), and their confighashes
differed, so as far as the database was concerned they were testing different
things.

The hash is taken over the configuration of the library, and the configuration
by then holds the reference files as an absolute path:

  destinationReal = os.path.realpath(destination)
  c["referenceFiles"] = destinationReal

so two machines whose workspaces sit in different places, or whose paths pass
through different symbolic links, hash the same configuration differently. That
never mattered while each machine had its own sqlite3 file and the hash only
had to be stable on the machine that made it. Sharing one database made it
matter, and it defeats both the claim and the check for results we already have.

The path says nothing that the hash needs: the contents of the reference files
are hashed right after it, and the same files in another directory are the same
test. It is replaced by a constant.

Checked with the same configuration under two different reference directories:
the hashes were 3181697168 and 2732211752 before, are equal after, and still
differ when a reference file itself changes.

Every confighash changes value once, so the first run after this treats every
library as untested and tests it. From then on the machines agree.

---
Generated by Claude Code.
@adrpo
adrpo merged commit eb86871 into master Aug 11, 2026
7 checks passed
adrpo added a commit that referenced this pull request Aug 11, 2026
Two machines testing master both claimed five of the same libraries and tested
them twice. #303 fixed the confighash being different for no reason; these five
differ for a real one. They are the libraries whose reference files come from
MAP-LIB_ReferenceResults/v4.1.0, which a maintenance job updates from time to
time, and the two machines had fetched them either side of such an update. The
libraries pinned to v4.0.0 and v3.2.3, which do not move, agreed.

So the configuration really did differ, and the claim was keyed by it:

  PRIMARY KEY (branch, libname, libversion, omcversion, confighash)

which is finer than the question a claim answers. Whether this machine may test
this library of this branch has nothing to do with which reference files it has;
that belongs to the two questions that are keyed by the configuration and stay
as they are - whether results for an exact configuration already exist, and
whether two runs' rows can collide.

The claim is now one row per (branch, libname). The versions are kept as
columns, so a run still says who is testing what and since when.

An existing job_claim is narrowed on connect, keeping the freshest claim per
library so that a run in progress does not lose its claims and let a second
machine in.

Checked against a database seeded the way production looks: the live claim
survives the migration, the key becomes (branch, libname), a second machine
with a different confighash is refused and told who holds it, a free library is
granted and then refused to the first machine, and after release the other
machine may take it. Plus a sqlite3 run through the CI configuration and a
PostgreSQL run with two FMI simulators.

---
Generated by Claude Code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant