Want shared script for standing up Omicron on Helios

[rzezeski]

Recently I ran into a build error when running the latest main (5519e936) with my own script to build, package, and deploy Omicron on a Helios host.

error[E0658]: use of unstable library feature 'asm': inline assembly is not stable enough for use and is subject to change
   --> /home/rpz/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff33033/dropshot/src/lib.rs:574:1                         
    |                                                                                                                                                     
574 |   #[usdt::provider(provider = "dropshot")]                             
    |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |                                                                        
   ::: /home/rpz/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff33033/dropshot/src/server.rs:387:13
    |                                                                                                                                                     
387 | /             probes::request_finish!(|| {
388 | |                 crate::ResponseInfo {
389 | |                     id: request_id.parse().unwrap(),                                                                                              
390 | |                     local_addr,                                      
...   |                                                                      
394 | |                 }                                                    
395 | |             });                                                                                                                                   
    | |______________- in this macro invocation
    |                                                                        
    = note: see issue #72016 <https://github.com/rust-lang/rust/issues/72016> for more information
    = help: add `#![feature(asm)]` to the crate attributes to enable                                                                                      
    = note: this error originates in the macro `probes::request_finish` (in Nightly builds, run with -Z macro-backtrace for more info)

This happened when the script was running the following step:

./target/release/omicron-package package

It turns out that others were not hitting this problem because they are using other methods to stand up Omicron on Helios, and for whatever reason their methods avoid this (recent) issue. That's fine, and I realize one size does not fit all when you are actively developing something. But it would also be nice to have a shared, committed script that those of us less familiar with Omicron can use to easily stand up Omicron on Helios.

I've been using this script, adapted from something @smklein shared with me a couple of weeks ago. I could just commit this as-is, but I though someone who is more intimate with Omicron might want to write something a little different. Anyways, feel free to use this or something else altogether. My main intent with this issue is just to get something committed that we can all share going forward.

#!/bin/bash
set -eu

#OMICRON_HOME=~/oxidecomputer/omicron
OMICRON_HOME=~/omi-integration-2/omicron
MY_ORG=all-lucky-sevens
MY_PROJ=remake

trap "kill 0" EXIT

pushd $OMICRON_HOME

# Build the "packaging" binary.
echo "=== BUILD PACKAGING BINARY START"
cargo build --release --package omicron-package --bin omicron-package
echo "=== BUILD PACKAGING BINARY END"

# Build all the packages describes in omicron/package-manifest.toml.
# This includes Sled Agent, Nexus, Oximeter, Propolis Server.
echo "=== BUILD PACKAGES START"
./target/release/omicron-package package
echo "=== BUILD PACKAGES END"

# Run the databases in the background.
echo "=== RUN DATABASES START"
cargo run --bin=omicron-dev -- db-run &> /dev/null &
cargo run --bin=omicron-dev -- ch-run &> /dev/null &
echo "=== RUN DATABASES END"

# Installs all the built services to your machine (within /opt/oxide),
# *starts* Sled Agent. Sled Agent unpacks/starts all other services.
echo "=== INSTALL SERVICES START"
pfexec ./target/release/omicron-package install
echo "=== INSTALL SERVICES END"

echo "Sled Agent and Nexus Online"
sleep 12
echo "=== CREATE ORGS + PROJECTS START"
./tools/oxapi_demo organization_create_demo $MY_ORG
./tools/oxapi_demo project_create_demo      $MY_ORG $MY_PROJ
echo "=== CREATE ORGS + PROJECTS END"

echo "RUNNING OMICRONG FROM: $OMICRON_HOME"
wait

popd

[davepacheco]

Sorry for the trouble, @rzezeski. We resolved a bunch of this in chat, but for the record, Ryan hit two separate issues:

1. The build problem mentioned above, which I believe was a regression introduced by Adds DTrace probes to slog messages with slog-dtrace #489. Fixes usdt feature dependency resolution bug #491 proposes a fix to catch this in CI and I also talked with @bnaecker about ways to avoid this specific problem altogether.
2. 500 errors ("internal server error") from Nexus when invoking oxapi_demo. These appear to have resulted from a race between database schema setup (which happens in the background when that script runs cargo run --bin=omicron-dev -- db-run &> /dev/null &) and attempting to use the schema in the calls to oxapi_demo. This race is only avoided by the sleep 12, which wasn't long enough.

The first problem feels like unlucky timing (and a gap in test coverage, which is also fixed in #491). The second one seems like a bug in the script.

---

I'm of two minds about a script like this. I almost never run omicron-package because I'm rarely testing anything that I can't test by running the underlying programs directly. So I generally follow the instructions in the "Build and run" section of the README almost exactly -- I even copy/paste out of there most of the time.

Eventually, we will want automation that builds and deploys the whole stack. I hope we do this soon. I don't think that'll look like a shell script, though. Initial rack setup will presumably be a more controlled sequence where we deploy components to various sleds (including Nexus, Sled Agent, CockroachDB, Clickhouse, etc.), make sure they're working, then populate the schema, then report being finished.

I'm torn about scripts like this. I understand the desire to automate manual processes, but I find this sort of half-automation much more frustrating than running commands by hand because of the time spent debugging things like the "sleep 12" race.

[rzezeski]

I'm of two minds about a script like this. I almost never run omicron-package because I'm rarely testing anything that I can't test by running the underlying programs directly. So I generally follow the instructions in the "Build and run" section of the README almost exactly -- I even copy/paste out of there most of the time.

That's fair enough, but FWIW those steps talk about sled-agent-sim, not a real sled agent run real VMs, etc. When I first approached Omicron I was running those steps. But then I wanted to test a "real" setup so that I could start working on OPTE integration. That's when I approached @smklein and he pointed me towards this script he had been using. As someone who knew very little about how everything in Omicron fits together this was a nice relief to just get everything in place easily.

Looking again at the README, perhaps I should have noticed the Deploying Omicron section. Though I'm not sure I would have figured it out without some hand holding. Basically, now that I'm somewhat more familiar with some of this stuff I kind of get it. But when first approach I wasn't really sure what steps to do to get a non-simulated setup going on a Helios host. For example, the Deploying Omicron section is kind of separated from the build & run steps. Perhaps if the deploying steps were inline as an alternative in Step 4 I would have realized what was going on. Honestly, I'm not really sure what, if anything, would have put me down a better path. I was just trying to get a non-simulated Omicron going with as little fuss as possible so that I can start integrating it with OPTE.

[rzezeski]

To be clear @davepacheco, I'm totally okay with just leaving things as they are. This issue alone could serve as a pointer for future newcomers until we have something more proper in place in the future for integration.

Though one thing that might help, at the cost of some redundancy, is having a no-frills list of deploy steps that only shows the exact commands to run, without having to read a lot of text and jump around in the readme between the "build and run" section and the "deploying omicron" section. Because right now with all the prose interlaced between steps it feels a bit easy to miss things unless you are already familiar with Omicron.

[davepacheco]

Sorry -- I didn't mean to imply you did anything unreasonable! This looks painful and not because of anything you did wrong. We can make this better. The Deploying Omicron section is useful background, but it's a little hard to find and it's not as clear as either the "Build and Run" section or the script that you and @smklein use. I think it'd be an improvement to take the steps in this script and add them to the README under something like "if you want to test deploying to a real system, do this" -- not as a script, but a sequence of steps you could run by hand. (edit: I think that's kind of what you just suggested, too).

Also FWIW: I believe you can follow the "build and run" steps, but run "sled-agent" instead of "sled-agent-sim" (possibly with different command-line arguments). I don't think you need to use omicron-package to do that. If that's right, we could update the "build and run" section to explain that, too.

I'm more wondering about whether to add a script like this one as a golden path for folks. To me, if folks are going to spend as long as you did debugging a problem in the script, we're better off fixing the README instead.

I took a swing at trying to make it less racey. Here's what I put together:

timeout_seconds=60
database_url="postgresql://root@127.0.0.1:32221/omicron?sslmode=disable"
export TMPDIR=/var/tmp/omicron_tmp
cargo run --bin=omicron-dev -- db-run --no-populate & > /dev/null 2>&1
okay=false
for (( i = 0; i < $timeout_seconds; i++ )) {
    echo "checking for cockroachdb ... "
    if cockroach sql --url "$database_url" \
        -e 'select NOW()' > /dev/null 2>&1; then
        okay=true
        break;
    fi  
    sleep 1
}

if [[ $okay != true ]]; then
    echo "cockroachdb did not start after $timeout_seconds seconds" >&2
    exit 1
fi

cargo run --bin=omicron-dev db-populate --database-url=$database_url

That separates the steps of starting the database from populating it, and it attempts to reliably wait up to 60 seconds for it to start. That solves this race. But in testing it, I found more problems: if cockroach isn't on your PATH, my script waits 60 seconds before telling you it didn't work (and doesn't say why). That's not easy to fix without changing db-run. I expect the original script just barrels on despite the error because it can't check the exit status and you don't see the error message because it redirects the output to /dev/null.

Like most bash scripts, I come away from this feeling like it's not worthwhile to make it reliable and fail cleanly, and also that without doing that, it'd be worse to point newcomers at it than to fix the README. I'm interested in others' takes though.

We could potentially also create omicron-dev run-the-world, which starts Cockroach, Clickhouse, Nexus, and either a simulated or real sled agent, using configs we decide are suitable for development. The "hard part" of this is really easy because db-run and ch-run are already backed by libraries that do all the work. The bigger challenge is probably that omicron-dev today comes from the test-utils package, which everything else depends on. For it to start Nexus and Sled Agent, it would need to depend on them instead. Maybe that's not so bad -- we'd just put omicron-dev in a separate crate that uses library interfaces from test-utils.

---

Summary of the ideas I mentioned above:

• Update Build and run section to talk about running a real sled agent
• Update Deploying Omicron section with a more complete set of steps
• Split omicron-dev tool into its own package and add a run-the-world command that launches everything in a way that makes sense for development

[bnaecker]

@rzezeski This has been largely addressed with the setup scripts and omicron-package, any objections to closing it?

[rzezeski]

I forgot I even wrote this. Yes, we have come a long way and I'd say standing up Omicron is 1000x easier today.