Random test failures in nixos.tests.boot.biosUsb

[edolstra]

http://hydra.nixos.org/build/36146783:

machine# ISOLINUX 6.04   Copyright (C) 1994-2015 H. Peter Anvin et al
machine# CHS: Error 0c00 reading sector 4220 (4/2/63)
machine# EDD: Error 0c00 reading sector 4220
error: timed out waiting for the VM to connect

Note: error 0c00 = unsupported track or invalid media (http://www.ctyme.com/intr/rb-0606.htm#Table234)

http://hydra.nixos.org/build/36146731 reaches the syslinux menu, then hangs loading the initrd:

Loading /boot/bzImage... ok
error: timed out waiting for the VM to connect

http://hydra.nixos.org/build/35988622:

machine# ISOLINUX 6.04   Copyright (C) 1994-2015 H. Peter Anvin et al
error: timed out waiting for the VM to connect

http://hydra.nixos.org/build/36146446:

Loading /boot/bzImage... CHS: Error 0c00 reading sector 11339 (11/3/63)
machine# EDD: Error 0c00 reading sector 11339
error: timed out waiting for the VM to connect

Haven't been able to reproduce these locally, so it might be triggered by high load on the host.

This may be a QEMU USB emulation issue. It doesn't appear to affect the USB UEFI boot test though.

[domenkozar]

@bobvanderlinden

[bobvanderlinden]

I'm not sure what to do about this. I can imagine syslinux being a bit pickier about the read-times, so it might timeout earlier than Linux itself. Maybe setting io limits on the host system can reproduce this problem. I'll give that a try later. However, this doesn't help the build much. If an IO timeout in syslinux is actually the problem, patching syslinux might solve the problem, but that seems like something you'd want to avoid for the boot test.

Is there some way to do this boot test not parallel to other tests?

[domenkozar]

For now the test has been removed from release blockers: a78ecb0

[domenkozar]

@bobvanderlinden we could set requiredSystemFeatures = [ "big-parallel" ];. @edolstra opinions?

[domenkozar]

Just FYI, this is still happening on 16.09

[bobvanderlinden]

Would removing/disabling this test be the better option? I'm not sure what "big-parallel" does for this test nor whether it will help in this case?

[roberth]

This still occurs quite often. I've gathered logs in #104642. I don't watch hydra closely but in my experience this is the most frequent flaky "blocker".

[bobvanderlinden]

Should we just disable it? I'm not sure how to solve this other than not using the usb emulation of qemu that we're using right now.

[roberth]

Your earlier comment may be worth considering

If an IO timeout in syslinux is actually the problem, patching syslinux might solve the problem, but that seems like something you'd want to avoid for the boot test.

This means we don't test one failure mode, but we still test many other usb-specific failure modes.
I don't think this timeout is a problem in practice anyway, or we could just ship a syslinux with an increased timeout.

Are you familiar with syslinux?

[bobvanderlinden]

Are you familiar with syslinux?

Not really. Only a bit as a user when I initially worked on USB boot support of NixOS.

That said, I did do a bit of spelunking in the syslinux code and did find that it tries to do this a number of times (retry count):

https://github.com/geneC/syslinux/blob/5e426532210bb830d2d7426eb8d8c154d9dfcba6/com32/lib/syslinux/disk.c#L52

There is does call interrupt 13, which is used to read from disks.

The error that @edolstra mentioned in the description is elsewhere:

https://github.com/geneC/syslinux/blob/5e426532210bb830d2d7426eb8d8c154d9dfcba6/core/fs/diskio_bios.c#L139-L142

It seems that error is also triggered when interrupt 13 is tried too many times. It has its own int13 call, so it also handles its own retry-count. The retry-count of this function is here:

https://github.com/geneC/syslinux/blob/5e426532210bb830d2d7426eb8d8c154d9dfcba6/core/fs/diskio_bios.c#L6

There doesn't seem to be a way to change this using configuration, so we would have to patch the code to increase the timeout.

This is all because interrupt 13 (IO) on the build server inside qemu is failing 6 times... or so it seems from what I'm reading in the syslinux code. I'm not sure we should be fixing syslinux, as there might be many other low-level tests that might be failing this way. Maybe there is some configuration within qemu to increase disk timeout?

[lukegb]

This might be failing inside SeaBIOS; my hypothesis is that we're hitting SeaBIOS' timeout on USB I/O (which is ~5s)

Given that this seems to succeed when it's retried, presumably because of lower overhead on whatever machine's getting used for the build: can we do something to force the USB image to remain resident in the disk cache throughout the test, e.g. by using vmtouch?

[ncfavier]

@lukegb I tried implementing what you suggested:

        testScript =
          ''
            ${optionalString (extraConfig ? usb) ''
            import subprocess
            subprocess.run("${pkgs.vmtouch}/bin/vmtouch -lfvd -m 1G ${extraConfig.usb}", shell=True, check=True)
            ''}
            machine = create_machine(${machineConfig})
            ...

But that fails because vmtouch can't lock that much memory. Is there a way to raise the ulimits for a test?

[ncfavier]

Some new developments in #170803

[ncfavier]

Proposed fix: #172059