Puppeteer headless test have slightly different result depending on how nixos kvm test is triggered.

[robinp]

At this point I'm not expecting a solution as is quite specific, but if you have any ideas, would be happy to hear.

I'm bringing up a nixos-test using testing.makeTest from nixpkgs/nixos/lib/testing.nix, which runs chromium through a custom-packaged puppeteer to exercise a webpage in the same vm. It is all in headless mode, without X.

If I start up the test using nix-build -A theTest, there's a click expectation in the headless browser test that fails.

If I start up the test using

nix-build -A theTest.driver
tests=$(cat result/test-script) ./result/bin/nixos-test-driver

then the click expectation succeeds.

I'm somewhat puzzled - the two ways of triggering should (?) result in identical qemu options, but the result is consistent. Maybe there is some timing issue? But how can that be so consistent. Or some graphics setup issue? But the headless browser doesn't use graphics even (and the vms run in headless mode in both cases).

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[robinp]

In retrospect, this might have been badly written test on my side. Puppeteer is an async JS API, and flakes can surface oddly. Generic Puppeteer tips:

• Make sure you don't miss an await on some of the methods.
• Disable or otherwise wait for animations in transitions. An element is often already "visible" when its opacity is still zero, so Puppeteer would start filling input fields. But then after the true fade-in, events might move focus around, disrupting the inputs.
• Run in slowMo mode, take screenshots, or run in headed (headless=false) mode to spot what's odd.

[robinp]

An other difference to note: the nixos-test runs with network isolation, while starting the driver manually will have network access. Might make a difference.