Flaky issue with helios deploy CI job during "ensuring datasets are ready"

[sunshowers]

I ran into this issue yesterday with the helios deploy job on CI:

https://buildomat.eng.oxide.computer/wg/0/details/01GJZX8864CXJBMQWWE2WYQMGS/rAHG6cmThT21CWqMwk99DpsKv74Vjb9oH9IGloNf3iZoqUb2/01GJZX8KJ04C0Y89G2AQ10YV3E#S349

The most pertinent lines are:

349 | 2022-11-28T20:29:12.948Z | creating IP pool...
350 | 2022-11-28T20:29:12.979Z | ensuring datasets are ready...
351 | 2022-11-28T20:31:13.903Z | Error: timed out after 120.654670292s

Rerunning the job on buildomat succeeded, which suggests a flaky issue with setup.

[smklein]

FYI @iliana - this line comes from

omicron/end-to-end-tests/src/bin/bootstrap.rs

Line 53 in daf316d

eprintln!("ensuring datasets are ready...");

. No idea what's causing the timeout, but I'd be curious about more info from the test runner

[iliana]

Three datasets need to be in place before disks can be created; before that, Nexus needs to be told about the three zpools the three datasets belong to. The nexus log for the job (search for /zpools/) indicates only two zpools were created before the 120-second timeout.

Possibilities:

• Exponential backoff for adding zpools and datasets should be changed to something less exponential.
• There's been a regression in how long it takes Nexus to come up.
• Clulow's Lament.

[jclulow]

I suspect it would be good if we did not use exponential backoff by default in almost any case -- it feels like some relatively tightly bounded linear growth would be better for most things?

[davepacheco]

See also #1270.