test failure: `test_update_uninitialized`

[davepacheco]

On branch main commit c5c0834, I saw this test failure:

        FAIL [  13.029s] omicron-nexus::test_all integration_tests::updates::test_update_status
  stdout ───

    running 1 test
    test integration_tests::updates::test_update_status ... FAILED

    failures:

    failures:
        integration_tests::updates::test_update_status

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 476 filtered out; finished in 12.50s
    
  stderr ───
    log file: /dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.0.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.0.log"
    DB URL: postgresql://root@[::1]:56691/omicron?sslmode=disable
    DB address: [::1]:56691
    log file: /dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.2.log
    note: configured to log to "/dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.2.log"

    thread 'integration_tests::updates::test_update_status' panicked at /home/dap/omicron-clean/nexus/test-utils/src/resource_helpers.rs:108:13:
    failed to make "GET" request to /v1/system/update/status: expected status code 200 OK, found 500 Internal Server Error
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: temporary directory leaked: "/dangerzone/omicron_tmp/.tmp66h2vw"
        If you would like to access the database for debugging, run the following:

        # Run the database
        cargo xtask db-dev run --no-populate --store-dir "/dangerzone/omicron_tmp/.tmp66h2vw/data"
        # Access the database. Note the port may change if you run multiple databases.
        cockroach sql --host=localhost:32221 --insecure
    WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 27363) and a temporary directory leaked, /dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.1-clickhouse-wt3WqY)
    failed to clean up ClickHouse data dir:
    - /dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.1-clickhouse-wt3WqY: File exists (os error 17)
    WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
    WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpE9Wcr3
    WARN: dropped MgdInstance without cleaning it up first (there may still be a

The log file has this error:

$ grep -w 500 /dangerzone/omicron_tmp/test_all-3946459b6ff90deb-test_update_uninitialized.27298.0.log  | looker
23:22:09.806Z INFO 913233fe-92a8-4635-9572-183f495429c4 (dropshot_external): request completed
    error_message_external = Internal Server Error
    error_message_internal = No inventory collection found
    latency_us = 152758
    local_addr = 127.0.0.1:39547
    method = GET
    remote_addr = 127.0.0.1:45483
    req_id = dc4a1132-37d8-4b8e-8478-5b16a51e75ae
    response_code = 500
    uri = /v1/system/update/status

1. This should probably be a 503, not a 500.
2. The test probably needs to either wait for an inventory collection to be present (probably what I'd do) or else retry transient errors (which would include 503s, but not 500s).

[davepacheco]

This was on a somewhat beefy machine (32 hardware threads) so it's probably more likely to hit races like this.