[network metrics] instance network interface schema #6414

zeeshanlakhani · 2024-08-22T19:48:26Z

This is related to the ongoing work in plumbing instance/guest metrics through to an oximeter producer in propolis.

Includes:

Rewriting the comment for trait Target in Oximeter to not being prepped for deprecation, as it does come in handy for custom needs
The main differentiating field among instance network information here is interface_id, which is exposed to propolis via the client's NetworkInterfaceRequest. We currently match the nic.slot to the port.slot to pass which of the requested nics interface_id uuids are part of the request.

Dependencies

Propolis => [network metrics] instance network interface metrics propolis#747

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. It depends on https://code.illumos.org/c/illumos-gate/+/3630. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of viona device instance. - The number of the device instance matches the kstat we'll look for. - We track this mapping via a type alias `InterfaceIdentifiers`, which is passed through from Machine initialization to vm objects (and vm objects shared), the latter to which we can use it as part of registering the oximeter producer. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, updating the `interface_id` appropriately, which is tracked in `interface_ids`. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. It depends on https://code.illumos.org/c/illumos-gate/+/3630. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of viona device instance. - The number of the device instance matches the kstat we'll look for. - We track this mapping via a type alias `InterfaceIdentifiers`, which is passed through from Machine initialization to vm objects (and vm objects shared), the latter to which we can use it as part of registering the oximeter producer. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, updating the `interface_id` appropriately, which is tracked in `interface_ids`. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests Dependencies ------------ - [ ] oxidecomputer/omicron#6414 - [ ] https://code.illumos.org/c/illumos-gate/+/3630

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of the device instance of viona. - The device instance number matches the Kstat we'll look for. - We track this mapping via a type alias `InterfaceIdentifiers`, which is passed through from Machine initialization to VM objects (and VM objects shared), the latter of which we can use as part of registering the oximeter producer. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, appropriately updating the `interface_id`, which is tracked in `interface_ids`. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests Dependencies ------------ - [ ] oxidecomputer/omicron#6414 - [ ] https://code.illumos.org/c/illumos-gate/+/3630

bnaecker

Looks good, thanks for renaming this. I think there's still a pretty big open question on the sled identifiers, see my comment. It applies to this and a few other schema too, so we should settle on something before the next release. Not necessarily this PR, if we want to put something on the board and drop it later.

oximeter/oximeter/schema/instance-network-interface.toml

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of the device instance of viona. - The device instance number matches the Kstat we'll look for. - We track this mapping via a type alias `InterfaceIdentifiers`, which is passed through from Machine initialization to VM objects (and VM objects shared), the latter of which we can use as part of registering the oximeter producer. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, appropriately updating the `interface_id`, which is tracked in `interface_ids`. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests Dependencies ------------ - [ ] oxidecomputer/omicron#6414 - [ ] https://code.illumos.org/c/illumos-gate/+/3630

zeeshanlakhani · 2024-08-26T13:14:43Z

@bnaecker Updated to remove those sled fields.

bnaecker

Looks good! Thanks for your patience while we sort out the sled-identifier issue!

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of the device instance of viona. - The device instance number matches the Kstat we'll look for. - We track this mapping via a type alias `InterfaceIdentifiers`, which is passed through from Machine initialization to VM objects (and VM objects shared), the latter of which we can use as part of registering the oximeter producer. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, appropriately updating the `interface_id`, which is tracked in `interface_ids`. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests Dependencies ------------ - [ ] oxidecomputer/omicron#6414 - [ ] https://code.illumos.org/c/illumos-gate/+/3630

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of the device instance of viona. - The device instance number matches the Kstat we'll look for. - We track this mapping via a type alias `InterfaceIdentifiers`, which is passed through from Machine initialization to VM objects (and VM objects shared), the latter of which we can use as part of registering the oximeter producer. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, appropriately updating the `interface_id`, which is tracked in `interface_ids`. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests Dependencies ------------ - [X] oxidecomputer/omicron#6414 - [X] https://code.illumos.org/c/illumos-gate/+/3630

package-manifest.toml

Prep to update lock to account for changes in omicron: - [ ] oxidecomputer/omicron#6414 - [ ] and used in oxidecomputer/propolis#747

This is related to the ongoing work in [plumbing instance/guest metrics](oxidecomputer/propolis#742) through to an oximeter producer in propolis. Includes: - Rewriting the comment for `trait Target` in Oximeter to not being prepped for deprecation, as it does come in handy for custom needs - The main differentiating field among instance network information here is `interface_id`, which is exposed to propolis via the client's `NetworkInterfaceRequest`. We currently match the `nic.slot` to the `port.slot` to pass which of the requested nics `interface_id` uuids are part of the request.

zeeshanlakhani · 2024-08-29T22:03:29Z

@bnaecker @leftwo deps are properly aligned now (with oxidecomputer/crucible#1438 and oxidecomputer/propolis#747).

leftwo

Seems like you got the dance steps figured out here on how to get these changes
rolled across the three repos.

Prep to update lock to account for changes in omicron: - [x] oxidecomputer/omicron#6414 - [x] and used in oxidecomputer/propolis#747

@pfmooney

Overview ======== This is related to the ongoing work in [plumbing instance/guest metrics](#742) through to an oximeter producer in propolis. Includes: * An update to `NetworkInterfaceRequest` to include an interface_id, which is part of the request on the [sled instance in Omicron](oxidecomputer/omicron#6414). * Within initialization, with concrete types, we create interface_ids mapping nic UUIDs => `minor` number of the device instance of viona. This is done as a Vec because it is limited in size (typically the MAX networks interfaces per VM). - The device instance number matches the Kstat we'll look for. - We track this mapping via a type alias `NetworkInterfaceIds` which is then passed to a `track_network_interfaces_kstats` function which is part of `initialize_network_devices` and adds the target to the kstat sampler. * A new stats module for collecting network_interface metrics. - In `to_samples` we generate multiple network interface targets based on vnics per instance, appropriately updating the `interface_id`, which comes from the network interface identifiers updated in network device initialization. * Move kstat_types out of virtual machine-specific stats and use its mocking for other stats tests * move stats.rs to stats/mod.rs Dependencies ------------ - [X] oxidecomputer/omicron#6414 - [X] https://code.illumos.org/c/illumos-gate/+/3630 Thanks to @pfmooney for exposing kstats directly through the viona (VirtIO-Net) driver as part of https://code.illumos.org/c/illumos-gate/+/3630. This work would be nothing without his!

iliana · 2024-09-03T15:53:51Z

Heads up — this merged PR updated the commits for Crucible and Propolis to commits that are on branches. We've had issues with that before, particularly since merged branches are squashed and then deleted in other repos.

zeeshanlakhani · 2024-09-03T16:02:56Z

@iliana Yep. I have a new PR now setting up everything off commits from main branches: #6453. With the bidirectional dependencies between propolis and omicron for this change (affecting crucible in the middle), it would have broken otherwise. But, with 6453 going green, everything will point to revs off main. I discussed this with @bnaecker and @leftwo. Sadly, there's not a great way to handle these bidirectional deps.

iliana · 2024-09-03T16:20:56Z

Ah I see. Thanks!

zeeshanlakhani requested review from bnaecker and pfmooney August 22, 2024 19:48

zeeshanlakhani mentioned this pull request Aug 23, 2024

Enable Guest/Instance network interface(s) metrics oxidecomputer/propolis#742

Closed

4 tasks

zeeshanlakhani mentioned this pull request Aug 23, 2024

[network metrics] instance network interface metrics oxidecomputer/propolis#747

Merged

3 tasks

bnaecker reviewed Aug 23, 2024

View reviewed changes

oximeter/oximeter/schema/instance-network-interface.toml Outdated Show resolved Hide resolved

oximeter/oximeter/schema/instance-network-interface.toml Outdated Show resolved Hide resolved

zeeshanlakhani force-pushed the zl/instance-network-interface branch from efe2ba7 to 90dd028 Compare August 25, 2024 15:49

bnaecker approved these changes Aug 26, 2024

View reviewed changes

zeeshanlakhani force-pushed the zl/instance-network-interface branch 2 times, most recently from 5d31963 to 66afcaf Compare August 28, 2024 16:03

zeeshanlakhani force-pushed the zl/instance-network-interface branch from 66afcaf to 07bb3d1 Compare August 28, 2024 16:12

leftwo reviewed Aug 29, 2024

View reviewed changes

package-manifest.toml Outdated Show resolved Hide resolved

zeeshanlakhani mentioned this pull request Aug 29, 2024

[prep deps] omicron branch oxidecomputer/crucible#1438

Merged

2 tasks

zeeshanlakhani added a commit to oxidecomputer/crucible that referenced this pull request Aug 29, 2024

[prep deps] omicron branch

eab4670

Prep to update lock to account for changes in omicron: - [ ] oxidecomputer/omicron#6414 - [ ] and used in oxidecomputer/propolis#747

zeeshanlakhani added 4 commits August 29, 2024 21:58

deps

0fc7f2a

[deps]

0f0ee19

[prep deps]: align propolis / crucible commits/revs

9bac9ba

zeeshanlakhani force-pushed the zl/instance-network-interface branch from 500212c to 9bac9ba Compare August 29, 2024 21:59

zeeshanlakhani requested a review from leftwo August 29, 2024 22:51

[deps update] match current HEAD of propolis

b510cc4

leftwo approved these changes Aug 30, 2024

View reviewed changes

[prep deps]

e92145d

zeeshanlakhani merged commit 5bf5f09 into main Sep 3, 2024

zeeshanlakhani deleted the zl/instance-network-interface branch September 3, 2024 13:03

zeeshanlakhani added a commit to oxidecomputer/crucible that referenced this pull request Sep 3, 2024

[prep deps] omicron branch (#1438)

9d82a99

Prep to update lock to account for changes in omicron: - [x] oxidecomputer/omicron#6414 - [x] and used in oxidecomputer/propolis#747

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[network metrics] instance network interface schema #6414

[network metrics] instance network interface schema #6414

Uh oh!

zeeshanlakhani commented Aug 22, 2024 •

edited

Loading

Uh oh!

bnaecker left a comment

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Aug 26, 2024

Uh oh!

bnaecker left a comment

Uh oh!

Uh oh!

zeeshanlakhani commented Aug 29, 2024

Uh oh!

leftwo left a comment

Uh oh!

iliana commented Sep 3, 2024

Uh oh!

zeeshanlakhani commented Sep 3, 2024 •

edited

Loading

Uh oh!

iliana commented Sep 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[network metrics] instance network interface schema #6414

[network metrics] instance network interface schema #6414

Uh oh!

Conversation

zeeshanlakhani commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependencies

Uh oh!

bnaecker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zeeshanlakhani commented Aug 26, 2024

Uh oh!

bnaecker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zeeshanlakhani commented Aug 29, 2024

Uh oh!

leftwo left a comment

Choose a reason for hiding this comment

Uh oh!

iliana commented Sep 3, 2024

Uh oh!

zeeshanlakhani commented Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iliana commented Sep 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zeeshanlakhani commented Aug 22, 2024 •

edited

Loading

zeeshanlakhani commented Sep 3, 2024 •

edited

Loading