Skip to content

Conversation

@leftwo
Copy link
Contributor

@leftwo leftwo commented Sep 25, 2025

Cleanup DTrace printing of downstairs state.

Convert the DsState type into a string before we stuff it into the DTrace probe. We were treating it as a string already on the other side, so why not just send the string we want to begin with. This avoids having to do more complicated processing inside the D scripts. Updated all the DTraces scripts that were using the old method to now expect the new string that is returned for the default display of DsState type.

Updated the default display for DsState to provide a bit more info for states that were "Connecting".

Changed a few log messages from error to warn in the case where we can't send an update message to Nexus.

Fix for various bits in #1772

Here is some sample output from some of the DTrace scripts:

alan@etrium:crucible$ pfexec dtrace -Z -s tools/dtrace/up-info.d 
  PID     UUID  SESSION DS0 DS1 DS2   NEXT_JOB  DELTA CONN   ELR   ELC   ERR   ERN
12140 b088b7c5 2d9b1fec ACT ACT ACT       1004      0    3     0     0     0     0
12140 b088b7c5 2d9b1fec ACT ACT ACT       1004      0    3     0     0     0     0
12140 b088b7c5 2d9b1fec ACT ACT ACT       1004      0    3     0     0     0     0
12140 b088b7c5 2d9b1fec ACT ACT ACT       1004      0    3     0     0     0     0
12146 2d50ec4b 431c0edf ACT ACT ACT       1004      0    3     0     0     0     0
12146 2d50ec4b 431c0edf ACT ACT ACT       1004      0    3     0     0     0     0
12146 2d50ec4b 431c0edf ACT ACT ACT       1004      0    3     0     0     0     0
12146 2d50ec4b 431c0edf ACT ACT ACT       1004      0    3     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684  WQ  WQ NEW       1000      0    2     0     0     0     0
12154 5f6d602e d8035684 REC REC REC       1000      0    3     0     0     2    18
12154 5f6d602e d8035684 ACT ACT ACT       1002      2    3     0     0    20     0
12154 5f6d602e d8035684 ACT ACT ACT       1002      0    3     0     0    20     0
12154 5f6d602e d8035684 ACT ACT ACT       1002      0    3     0     0    20     0
  PID     UUID  SESSION DS0 DS1 DS2   NEXT_JOB  DELTA CONN   ELR   ELC   ERR   ERN
12154 5f6d602e d8035684 ACT ACT ACT       1002      0    3     0     0    20     0
12179 0817aca3 2a0fd3d4 OFL ACT ACT       1401      0    3     0     0     0     0
12179 0817aca3 2a0fd3d4 OFL ACT ACT       1401      0    3     0     0     0     0
12179 0817aca3 2a0fd3d4 OFL ACT ACT       1401      0    3     0     0     0     0

I let the dtrace scripts run while running cargo test and I could see many of the states showing up.
Here is a bunch of noise from that:

alan@etrium:crucible$ pfexec dtrace -Z -s tools/dtrace/sled_upstairs_info.d
  PID     UUID  SESSION DS0 DS1 DS2   UPW   DSW  NEXT_JOB   WRITE_BO    IP0   IP1   IP2     D0    D1    D2     S0    S1    S2    ER0   ER1   ER2    EC0   EC1   EC2
12494 7dc3f943 1bb2b697 FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12491 861995a1 e6a231de OFL OFL OFL     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12497 5bd17c94 fe8e0586 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12492 34940348 dd0721be FLT ACT ACT     0   497      1497          0      0     0     0      0   497   497    497     0     0      0     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12503 da39d261 17b1d5c7 OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12498 09e744d8 e1e347c0 FLT ACT ACT     0  1200      2200          0      0     0     0      0  1200  1200   1200     0     0      0     0     0      0     0     0
12502 4137d113 1912fce1 OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12965 5fed87cb 30bb4c55 ACT ACT ACT     1    40      1184    1048576      1     1     1     39    39    39      0     0     0      0     0     0      0     0     0
12494 7dc3f943 1bb2b697 FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12497 5bd17c94 fe8e0586 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12492 34940348 dd0721be FLT ACT ACT     0   497      1497          0      0     0     0      0   497   497    497     0     0      0     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12498 09e744d8 e1e347c0 FLT ACT ACT     0  1200      2200          0      0     0     0      0  1200  1200   1200     0     0      0     0     0      0     0     0
12965 5fed87cb 30bb4c55 ACT ACT ACT     1    43      1187    1048576      1     1     1     42    42    42      0     0     0      0     0     0      0     0     0
12494 7dc3f943 1bb2b697 FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12497 5bd17c94 fe8e0586 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12492 34940348 dd0721be FLT ACT ACT     0   497      1497          0      0     0     0      0   497   497    497     0     0      0     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12498 09e744d8 e1e347c0 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12965 5fed87cb 30bb4c55 ACT ACT ACT     1    46      1190    1048576      1     1     1     45    45    45      0     0     0      0     0     0      0     0     0
  PID     UUID  SESSION DS0 DS1 DS2   UPW   DSW  NEXT_JOB   WRITE_BO    IP0   IP1   IP2     D0    D1    D2     S0    S1    S2    ER0   ER1   ER2    EC0   EC1   EC2
12494 7dc3f943 1bb2b697 FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12497 5bd17c94 fe8e0586 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12492 34940348 dd0721be FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12498 09e744d8 e1e347c0 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12494 7dc3f943 1bb2b697 FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12497 5bd17c94 fe8e0586 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12492 34940348 dd0721be FLT ACT ACT     0     0      1498          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12498 09e744d8 e1e347c0 FLT ACT ACT     0     0      2201          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12494 7dc3f943 1bb2b697  LR ACT ACT    20   199      1699      17920     10    37    37     24   162   162    165     0     0      3     0     0      0     0     0
12497 5bd17c94 fe8e0586  LR ACT ACT    11   181      2384       8704     10    19    19     24   162   162    147     0     0      3     0     0      0     0     0
12492 34940348 dd0721be  LR ACT ACT     2   164      1664          0      2     2     2     24   162   162    138     0     0      3     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12498 09e744d8 e1e347c0  LR ACT ACT     8   176      2379       6144     10    14    14     24   162   162    142     0     0      3     0     0      0     0     0
12501 dbb21d8c 932671af OFL ACT ACT     0     0      1000          0      0     0     0      0     0     0      0     0     0      0     0     0      0     0     0
12965 5fed87cb 30bb4c55 ACT ACT ACT     0    46      1190          0      0     0     0     46    46    46      0     0     0      0     0     0      0     0     0
12501 dbb21d8c 932671af ACT ACT ACT     0     1      1001        512      1     0     0      0     1     1      0     0     0      0     0     0      0     0     0

ss == "wait_quorum" ? "WQ" :
ss == "wait_active" ? "WA" :
ss == "connecting" ? "CON" :
ss == "Active" ? "ACT" :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to share this function with all the other scripts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe... Let me see if I can find an easy way.
I think it's "possible", but I don't want the callers of the scripts have to do too much, or remember too much.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see if I can find an easier way.. in a follow up commit :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want these changes so I can triage what is going on when multiple downstairs come and go while doing LR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could look at what OPTE does, which has a few shared type and function definitions for these kinds of things.

@leftwo leftwo merged commit ebcf035 into main Sep 29, 2025
17 checks passed
@leftwo leftwo deleted the alan/dtrace-log-polish branch September 29, 2025 21:26
iximeow added a commit to oxidecomputer/omicron that referenced this pull request Oct 18, 2025
Propolis changes:

* oxidecomputer/propolis#950
* oxidecomputer/propolis#952
* oxidecomputer/propolis#951
* oxidecomputer/propolis#954
* oxidecomputer/propolis#957
* oxidecomputer/propolis#960
* oxidecomputer/propolis#961
* oxidecomputer/propolis#955

Crucible changes:

* oxidecomputer/crucible#1773
* oxidecomputer/crucible#1774
* oxidecomputer/crucible#1780
* oxidecomputer/crucible#1778

Crucible shouldn't have functional changes here, Propolis' big ones are
@sunshowers' work moving Propolis to versioned APIs, plus propolis#960
turning the crank on MAXCPU.

propolis#961 changes the initial Milan CPU profile one last time before
the release in service of propolis#959. Propolis will clear [this
bit](https://github.com/oxidecomputer/omicron/blob/d74f5e3f1ae0a378dcdb9795a0ada2426702b046/nexus/src/app/instance_platform/cpu_platform.rs#L423).
Later we want to actually set up leaf 8000_001E, so after this merges
I'll have a followup to remove that leaf from the inital Milan
definition to keep the profile constant when `propolis-server` is
smarter about the leaf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants