Building a golden log

The golden log is the boot log of a board you know is good. The comparison is only as useful as this log: a healthy board under test should give no deviations. Getting there takes a few minutes per board type.

1. Capture several healthy boards

Take three or more boards that pass all other tests, from different batches if you can, and capture their boot logs the same way (Capturing boot logs). Pick one as the golden log.

2. Compare healthy boards with each other

$ boot-log-diff good-1.log good-2.log
$ boot-log-diff good-1.log good-3.log

Everything reported here is noise for your product: values that differ between healthy boards. Typical examples:

Differs because of	Do
serial number, host name, MAC address in a host name	--mask 'epc8m-[0-9a-f]+'
random values (crng init, UUIDs that are not in the usual form)	--ignore 'random: crng'
eMMC from two vendors (mmcblk2: mmc2:0001 S0J58X vs 8GTF4R)	--mask 'mmc2:0001 \S+'
a calibration value printed at boot	--mask 'offset=-?\d+'
a line that only some healthy boards print	--ignore 'that line'

--mask keeps the line and replaces the matching part, so the line must still be there. --ignore drops the line completely. Prefer --mask: a missing line is often the fault you are looking for.

Lines that changed position are not a problem: drivers probing in parallel do not keep their order, and boot-log-diff counts such lines as moved, not as deviations.

3. Write the options down

Repeat until healthy boards compare with exit status 0. Then keep the options next to the golden log, for example in a small script:

#!/bin/sh
# boot-check.sh DUT.log [extra options]
exec boot-log-diff "$(dirname "$0")/epc8m-golden.log" "$@" \
    --mask 'epc8m-[0-9a-f]+' \
    --mask 'mmc2:0001 \S+' \
    --ignore 'random: crng' \
    --fact 'fpga=FPGA bitstream v(\S+)' \
    --rules "$(dirname "$0")/epc8m-faults.toml"

4. Add facts for what must be exact

Facts are values compared on their own and listed at the top. The built-in ones are model, u-boot, dram, kernel, memory and cpus. Add what your product must have exactly — an FPGA version, an eMMC size, a display resolution:

--fact 'fpga=FPGA bitstream v(\S+)'
--fact 'emmc=mmcblk\d: \S+ \S+ (\S+ GiB)'
--fact 'panel=panel-simple .*: (\d+x\d+)'

The first group of the regex is the value.

One golden log per …

• hardware variant — a board with 1 GiB RAM and one with 2 GiB need different golden logs, or a fact check per variant;
• software release — after a bootloader or kernel update, capture a new golden log from a good board and keep the old one for boards in the field.