Skip to content

Conversation

andrewjstone
Copy link
Contributor

@andrewjstone andrewjstone commented Oct 15, 2025

Primarily this includes attestations, concurrent session handshakes, and better errors.

name = "dice-verifier"
version = "0.2.0"
source = "git+https://github.com/oxidecomputer/dice-util?rev=3cc953c8d0ace2f20cbcf3920b0771d25301960a#3cc953c8d0ace2f20cbcf3920b0771d25301960a"
version = "0.3.0-pre0"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flihp Is it safe to merge this change?

@andrewjstone
Copy link
Contributor Author

I still need to test this change. I think I need to test it on @a4x2 and a racklet. Sprockets is only used during RSS and so existing systems should not be affected.

@labbott @flihp Anything in particular I should be concerned about?

@andrewjstone andrewjstone requested a review from flihp October 15, 2025 23:26
Copy link
Contributor

@labbott labbott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'll let you test on a4x2 before giving the final approval.

Comment on lines +65 to +66
// TODO: Once we have a corpus, use it.
// Will we ever have one at RSS time?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should because the expectation is that sleds have gotten a mupdate but we should review this

@andrewjstone
Copy link
Contributor Author

Tests pass on a4x2 with commit 0b77bc1 and https://github.com/oxidecomputer/testbed/pull/157

@andrewjstone
Copy link
Contributor Author

Both production smf and non-production were taken from #7906

andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
andrewjstone added a commit that referenced this pull request Oct 18, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
@andrewjstone
Copy link
Contributor Author

Tested on madrid, and RSS completed:

BRM42220082 # cat `svcs -L sled-agent` | grep Handoff
{"msg":"Handoff to Nexus is complete","v":0,"name":"SledAgent","level":30,"time":"2025-10-20T22:42:58.496597994Z","hostname":"BRM42220082","pid":665,"component":"RSS","file":"sled-agent/src/rack_setup/service.rs:1066"}

sprockets appears to be working as expected given we don't have a real corpus yet

BRM42220082 # cat `svcs -L sled-agent` | grep sprockets | looker
00:00:57.094Z INFO SledAgent (SledAgentSprocketsServer): Started listening
    file = sled-agent/src/bootstrap/sprockets_server.rs:51
    local_addr = [fdb0:a840:2504:190::1]:12346
00:06:12.002Z INFO SledAgent (SledAgentSprocketsServer): TCP connection accepted
    file = sled-agent/src/bootstrap/sprockets_server.rs:77
    remote_addr = [fdb0:a840:2504:190::1]:55968
00:06:12.129Z INFO SledAgent (SledAgentSprocketsServer): Certificate => CN=trust-quorum-dhe,O=Oxide Computer Company,C=US
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.129Z INFO SledAgent (SledAgentSprocketsServer): Certificate => CN=device-id,O=Oxide Computer Company,C=US
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.129Z INFO SledAgent (SledAgentSprocketsServer): Certificate => CN=PDV2:913-0000019:RRR:BRM42220082,O=Oxide Computer Company,C=US
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.129Z INFO SledAgent (SledAgentSprocketsServer): Certificate => CN=Platform Identity Staging Signer A1,O=Oxide Computer Company,C=US
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.331Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=trust-quorum-dhe,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.331Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=device-id,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.331Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=PDV2:913-0000019:RRR:BRM42220082,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.331Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=Platform Identity Staging Signer A1,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.331Z INFO SledAgent (BootstrapAgentRssHandler): Loaded keys and certs
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:58
00:06:12.338Z INFO SledAgent (BootstrapAgentRssHandler): verifier succeeded against root w/ subject: CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:284
00:06:12.338Z INFO SledAgent (BootstrapAgentRssHandler): Signature verified successfully
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:351
00:06:12.443Z INFO SledAgent (SledAgentSprocketsServer): verifier succeeded against root w/ subject: CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:284
00:06:12.443Z INFO SledAgent (SledAgentSprocketsServer): Signature verified successfully
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:351
00:06:12.548Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=trust-quorum-dhe,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.548Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=device-id,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.548Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=PDV2:913-0000019:RRR:BRM42220082,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.548Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=Platform Identity Staging Signer A1,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.548Z INFO SledAgent (BootstrapAgentRssHandler): Loaded keys and certs
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:58
00:06:12.553Z INFO SledAgent (BootstrapAgentRssHandler): verifier succeeded against root w/ subject: CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:284
00:06:12.553Z INFO SledAgent (BootstrapAgentRssHandler): Signature verified successfully
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:351
00:06:12.763Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=trust-quorum-dhe,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.763Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=device-id,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.763Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=PDV2:913-0000019:RRR:BRM42220082,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.763Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=Platform Identity Staging Signer A1,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.763Z INFO SledAgent (BootstrapAgentRssHandler): Loaded keys and certs
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:58
00:06:12.768Z INFO SledAgent (BootstrapAgentRssHandler): verifier succeeded against root w/ subject: CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:284
00:06:12.768Z INFO SledAgent (BootstrapAgentRssHandler): Signature verified successfully
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:351
00:06:12.988Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=trust-quorum-dhe,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.988Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=device-id,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.988Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=PDV2:913-0000019:RRR:BRM42220082,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.988Z INFO SledAgent (BootstrapAgentRssHandler): Certificate => CN=Platform Identity Staging Signer A1,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:72
00:06:12.988Z INFO SledAgent (BootstrapAgentRssHandler): Loaded keys and certs
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:58
00:06:12.994Z INFO SledAgent (BootstrapAgentRssHandler): verifier succeeded against root w/ subject: CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:284
00:06:12.994Z INFO SledAgent (BootstrapAgentRssHandler): Signature verified successfully
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/keys.rs:351
00:06:13.104Z INFO SledAgent (SledAgentSprocketsServer): Running with protocol version 2
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/server.rs:180
00:06:13.104Z INFO SledAgent (BootstrapAgentRssHandler): Running with protocol version 2
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:311
00:06:13.104Z INFO SledAgent (BootstrapAgentRssHandler): Running with protocol version 2
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:311
00:06:13.104Z INFO SledAgent (BootstrapAgentRssHandler): Running with protocol version 2
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:311
00:06:13.769Z INFO SledAgent (SledAgentSprocketsServer): Cert chain from peer "PDV2:913-0000019:RRR:BRM42220082" verified against root "CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US"
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/server.rs:209
00:06:13.769Z INFO SledAgent (SledAgentSprocketsServer): TQ & attestation cert chains agree on platform id
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/server.rs:219
00:06:14.094Z INFO SledAgent (BootstrapAgentRssHandler): Running with protocol version 2
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:311
00:06:14.099Z INFO SledAgent (BootstrapAgentRssHandler): Cert chain from peer "PDV2:913-0000019:RRR:BRM42220082" verified against root "CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US"
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:344
00:06:14.099Z INFO SledAgent (BootstrapAgentRssHandler): TQ & attestation cert chains agree on platform id
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:354
00:06:14.100Z INFO SledAgent (SledAgentSprocketsServer): Peer attestation verified
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/server.rs:246
00:06:14.100Z INFO SledAgent (SledAgentSprocketsServer): Peer measurements appraisal failed: Measurements are not a subset of reference measurements
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/server.rs:259
00:06:14.100Z INFO SledAgent (SledAgentSprocketsServer): Sprockets handshake completed
    file = sled-agent/src/bootstrap/sprockets_server.rs:87
    remote_addr = [fdb0:a840:2504:190::1]:55968
00:06:14.101Z INFO SledAgent (BootstrapAgentRssHandler): Peer attestation verified
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:382
00:06:14.101Z INFO SledAgent (BootstrapAgentRssHandler): Peer measurements appraisal failed Measurements are not a subset of reference measurements
    BootstrapAgentClient = [fdb0:a840:2504:190::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:397
00:06:14.903Z INFO SledAgent (BootstrapAgentRssHandler): Cert chain from peer "PDV2:913-0000019:RRR:BRM42220011" verified against root "CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US"
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:344
00:06:14.903Z INFO SledAgent (BootstrapAgentRssHandler): TQ & attestation cert chains agree on platform id
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:354
00:06:14.909Z INFO SledAgent (BootstrapAgentRssHandler): Cert chain from peer "PDV1:913-0000019:006:BRM44220007" verified against root "CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US"
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:344
00:06:14.909Z INFO SledAgent (BootstrapAgentRssHandler): TQ & attestation cert chains agree on platform id
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:354
00:06:15.125Z INFO SledAgent (BootstrapAgentRssHandler): Cert chain from peer "PDV2:913-0000019:RRR:BRM06240029" verified against root "CN=Platform Identity Staging Root A,O=Oxide Computer Company,C=US"
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:344
00:06:15.125Z INFO SledAgent (BootstrapAgentRssHandler): TQ & attestation cert chains agree on platform id
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:354
00:06:15.219Z INFO SledAgent (BootstrapAgentRssHandler): Peer attestation verified
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:382
00:06:15.219Z INFO SledAgent (BootstrapAgentRssHandler): Peer measurements appraisal failed Measurements are not a subset of reference measurements
    BootstrapAgentClient = [fdb0:a840:2504:211::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:397
00:06:15.229Z INFO SledAgent (BootstrapAgentRssHandler): Peer attestation verified
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:382
00:06:15.229Z INFO SledAgent (BootstrapAgentRssHandler): Peer measurements appraisal failed Measurements are not a subset of reference measurements
    BootstrapAgentClient = [fdb0:a840:2504:492::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:397
00:06:16.194Z INFO SledAgent (BootstrapAgentRssHandler): Peer attestation verified
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:382
00:06:16.194Z INFO SledAgent (BootstrapAgentRssHandler): Peer measurements appraisal failed Measurements are not a subset of reference measurements
    BootstrapAgentClient = [fdb0:a840:2504:b54::1]:12346
    file = /home/build/.cargo/git/checkouts/sprockets-882d17aeeb0cb343/7da1f0b/tls/src/client.rs:397
00:06:16.889Z INFO SledAgent (SledAgentSprocketsServer): Connection closed
    file = sled-agent/src/bootstrap/sprockets_server.rs:92
    remote_addr = [fdb0:a840:2504:190::1]:55968
    ```

Primarily this includes concurrent session handshakes and better errors.
This is necessary for testing latest code on a4x2
andrewjstone added a commit that referenced this pull request Oct 20, 2025
Builds on #9232

This is the first step in wrapping the `trust_quorum::Node` so that it
can be used in an async context and integrated with sled-agent. Only
the sprockets networking has been fully integrated so far such that
each `NodeTask` has a `ConnMgr` that sets up a full mesh of sprockets
connections. A test for this connectivity behavior has been written but
the code is not wired into the production code yet.

Messages can be sent between `NodeTasks` over sprockets connections.
Each connection exists in it's own task managed by an `EstablishedConn`.
The main `NodeTask` task sends messages to and receives messages from
this task to interact with the outside world via sprockets. Currently
only `Ping` messages are sent over the wire as a means to keep the
connections alive and detect disconnects.

A `NodeHandle` allows one to interact with the `NodeTask`. Currently
only three operations are implemented with messages defined in
`NodeApiRequest`. The user can instruct the node who it's peers
are on the bootstrap network to establish connectivity, can poll
for connectivity status, and can shutdown the node. All of this
functionality is used in the accompanying test.

It's important to re-iterate that this code only implements connectivity
between trust quorum nodes and no actual trust quorum messages are sent.
They can't be as a handle can not yet initiate a reconfiguration or
LRTQ upgrade. That behavior will come in a follow up. This PR is large
enough.

A lot of this code is similar to the LRTQ connection management code,
except that it operates over sprockets rather than TCP channels. This
introduces some complexity, but it is mostly abstracted away into the
`SprocketsConfig`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants