Resource Utilization #1782

smklein · 2022-10-03T23:32:00Z

Part of #1734 , specifically this bit.

This PR adds a table called resource_usage, which exists for silos, organizations, and projects. Currently, it only contains information about each collection's disk usage.

API exposure
- Emit this information to Clickhouse (metrics are passed to the producer on every modification)
- Add a metrics-based API for querying such historical info (done, under /system/metrics/resource-utilization. Happy to update this API as it's useful, but I went with something minimal for expediency).
Correctness
- Add CTE to update all collections up to the root
- Ensure each query avoid full-table scans
- Ensure that each update of resource_usage is atomic (part of a transaction, saga, or CTE)
- Ensure that the disk usage accounting is accurate. Currently, we only consider region allocations / deallocations; accurately accounting for snapshots will require incorporating Provision destination volume for snapshot blocks #1752.
- Add integration tests

After merging, I'd like to do the following:

Emit some amount of "total capacity" info, to contextualize the currently used amount. This makes much more sense at a physical view (sled, rack, fleet) than user view.
Expand the "collections" to include a "fleet" object. This will be particularly useful for operators.
Make the accounting of "utilization" more accurate. It currently is not accounting for: Metadata (e.g., crucible's sqlite dbs), system usage (CRDB, Clickhouse, the OS itself, etc).
Expand the usage information to account for CPU usage, RAM, and other globally-shared resources.
Ensuring idempotency has been punted to Tracking issue for saga idempotency safety #2094 , though this should work with the new CTEs

nexus/src/app/sagas/disk_create.rs

nexus/src/app/instance.rs

nexus/src/app/sagas/instance_create.rs

smklein · 2022-10-07T15:45:38Z

Hey all, I'm sending this out for review a little earlier than I normally would.

Correctness-wise, this PR should work, if we are ignoring idempotency issues.
However, there are some cases where saga nodes are not idempotent, and they should be. I've added documentation on places where I'm continuing to work, and I'd file bugs if we considered merging this before resolving.

david-crespo · 2022-10-07T15:45:41Z

Before I look through in more detail, I will say the API is good and works in the console. See oxidecomputer/console#1214

nexus/src/app/sagas/disk_create.rs

nexus/src/app/sagas/snapshot_create.rs

nexus/src/external_api/http_entrypoints.rs

common/src/sql/dbinit.sql

davepacheco

Approving (despite one suggestion and one question below) because regardless of those comments this is clearly a huge step in the right direction!

nexus/src/db/pool.rs

davepacheco · 2023-01-17T22:04:58Z

nexus/src/db/pool.rs

+const DISALLOW_FULL_TABLE_SCAN_SQL: &str =
+    "set disallow_full_table_scans = on; set large_full_scan_rows = 0;";
+#[cfg(test)]
+pub const ALLOW_FULL_TABLE_SCAN_SQL: &str =


Sounds good. I thought this might not work if you're not in a transaction when you use it, and that appears to be true. All of the callers in this PR are inside a transaction when they use it. So we're fine here.

It'd be neat if we had a more type-safe way to make sure this wasn't misused. That seems outside the scope of this PR. (I did suggest adding a doc comment above to mention this.)

davepacheco · 2023-01-17T22:23:54Z

nexus/src/db/datastore/virtual_provisioning_collection.rs

+            .map_err(|e| {
+                public_error_from_diesel_pool(e, ErrorHandler::Server)
+            })?;
+        self.virtual_provisioning_collection_producer


Thanks, that's helpful. I think the bit I had missed even after reading this is that the way the database update works updates the value relatively (as one would expect) and then fetches the updated [absolute] value. This makes sense. To answer my question pedantically: Nexus 2 learns about the resources allocated from P for I1 when it makes the database query to update the resources allocated from P for I2 and (as part of the same query) fetches the total resources allocated for P.

(I think it'd be worthwhile to work through this example in a block comment, but definitely not a blocker here.)

The follow-up question then is: what happens if the Clickhouse updates are processed out of order? Take the same example of two Nexus instances, P, I1, and I2. From the start: I1 is provisioned from Nexus 1, then I2 is immediately provisioned from Nexus 2. At this point, say Oximeter hasn't yet collected data from either of them. Nexus 1 has a pending update for Clickhouse that only includes I1, while Nexus 2 has a (newer) pending update for Clickhouse that includes both I1 and I2. Now suppose that Oximeter collects from Nexus 2, stores that in Clickhouse, then collects from Nexus 1. How do we prevent the older value clobbering the correct value?

david-crespo

Forgot I was a reviewer. Looks great from my point of view. Excited to turn this back on in the console.

…2157) When we create a disk, the saga generates both a disk_id and a volume_id We use both ids when we create the disk database record However, the `volume_id` here is only used as the key to store the volume data in the database. That volume_id is never exposed to anyone outside Nexus. For the actual volume data structure and later the VolumeConstructionRequest, we use the value in the disk_id, but call it volume_id. However, Propolis (who needs to call Nexus and tell it to remove a read only parent only has the top level disk UUID. The internal volume_id is not exposed. This adds a new internal API endpoint that will take the disk UUID, then lookup the volume_id and then call the volume layer remove read only parent saga. Adding a layer like this also allows tests that operate only on a volume level to not require any changes. Co-authored-by: Alan Hanson <alan@oxide.computer>

Currently, wizard is the main entrypoint for wicket. To support uploading artifacts, We're going to change it so that it has to go through a clap command layer. Also add a `default-run` so that `cargo run -p wicket` runs the wicket binary.

Previously, we were showing the next component ID in both the previous and next spots.

Add a command that can upload data to wicketd, via stdin from wicket. This also makes wicket behave like a login shell: read arguments passed in with `-c`. Tested by creating a user locally with wicket as the login shell, then running: ``` % cat foo.bin | ssh wicket-test@localhost upload my-artifact 0.1.0 Jan 13 22:12:25.946 INFO read artifact my-artifact:0.1.0 (11957848 bytes) from stdin, uploading to wicketd Jan 13 22:12:25.972 INFO successfully uploaded my-artifact:0.1.0 (11957848 bytes) to wicketd ```

Bumps [trybuild](https://github.com/dtolnay/trybuild) from 1.0.75 to 1.0.76. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/trybuild/releases">trybuild's releases</a>.</em></p> <blockquote> <h2>1.0.76</h2> <ul> <li>Use subdirectory within target/tests/ (<a href="https://github-redirect.dependabot.com/dtolnay/trybuild/issues/219">#219</a>, thanks <a href="https://github.com/ijackson"><code>@ijackson</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/trybuild/commit/696423a32c1c20601c38e3c9f718b6c7bf833719"><code>696423a</code></a> Release 1.0.76</li> <li><a href="https://github.com/dtolnay/trybuild/commit/cd40633c016085e1a85fd211e5fc91cab6afc653"><code>cd40633</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/dtolnay/trybuild/issues/219">#219</a> from ijackson/namespace-test-directory</li> <li><a href="https://github.com/dtolnay/trybuild/commit/dc30697e24707461913fa4aa15583f3eb69c68dc"><code>dc30697</code></a> Use subdirectory within target/tests/</li> <li>See full diff in <a href="https://github.com/dtolnay/trybuild/compare/1.0.75...1.0.76">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=trybuild&package-manager=cargo&previous-version=1.0.75&new-version=1.0.76)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [clap](https://github.com/clap-rs/clap) from 4.0.32 to 4.1.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.1.1</h2> <h2>[4.1.1] - 2023-01-14</h2> <h3>Fixes</h3> <ul> <li><em>(error)</em> Small softening attempt for "unexpected argument" error</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.1.1] - 2023-01-14</h2> <h3>Fixes</h3> <ul> <li><em>(error)</em> Small softening attempt for "unexpected argument" error</li> </ul> <h2>[4.1.0] - 2023-01-13</h2> <h3>Compatibility</h3> <p>MSRV changed to 1.64.0</p> <p>For apps with custom <code>--help</code> and <code>--version</code> flags:</p> <ul> <li>Descriptions for <code>--help</code> and <code>--version</code> changed</li> </ul> <p>When apps have errors imitating clap's error style:</p> <ul> <li>Error message style was changed, including <ul> <li>Moving away from "did you mean" to tips</li> <li>Leading letter is lower case</li> <li>"For more" added some punctuation</li> </ul> </li> </ul> <h3>Features</h3> <ul> <li><code>ArgMatches::get_occurrences</code> support for argument values to be grouped by their occurrence</li> </ul> <h3>Fixes</h3> <ul> <li><em>(derive)</em> Allow <code>upgrade_from</code> when arguments / subcommands are explicitly marked as required</li> <li><em>(help)</em> Try be more clearer and succinct with <code>--help</code> and <code>--version</code> (also helps with overflow)</li> <li><em>(error)</em> Try to be more clearer and succinct with error messages</li> <li><em>(error)</em> Officially adopt <a href="https://rustc-dev-guide.rust-lang.org/diagnostics.html#suggestion-style-guide">an error style guide</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/74a82d708524ac88941e9fe4318fe5842ed74b68"><code>74a82d7</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/06f392a0d2ddbcd371d5dc65bcddbcd565430c4f"><code>06f392a</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/4d913fa4d1efee4391ec185495efb64b4aba0aa7"><code>4d913fa</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/clap-rs/clap/issues/4639">#4639</a> from epage/error</li> <li><a href="https://github.com/clap-rs/clap/commit/162a556dbad89ade96ea35052812bebc6c79ddc9"><code>162a556</code></a> fix(error): Try to soften unexpected argument/value errors</li> <li><a href="https://github.com/clap-rs/clap/commit/34d856b449a66719360d10cbc9334e09c9bb2769"><code>34d856b</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/889ca7a5372f91b073f9bd1eef593f3c5868af56"><code>889ca7a</code></a> chore: Bump versions for 4.1</li> <li><a href="https://github.com/clap-rs/clap/commit/2bafb9b75d97ec84fa51612c9d6898f33c9a39c8"><code>2bafb9b</code></a> docs(contrib): Define a compatibility policy for help/error output</li> <li><a href="https://github.com/clap-rs/clap/commit/a41ca2edb05b429a317dd97215c3eee0e4d3c97f"><code>a41ca2e</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/523adc20e7ae1a4c5bf7caa261671b2ded814ac8"><code>523adc2</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/clap-rs/clap/issues/4635">#4635</a> from epage/stablize</li> <li><a href="https://github.com/clap-rs/clap/commit/b4f111a978fb9d1d07a05940fc92802029d9df23"><code>b4f111a</code></a> feat: Stablize <code>ArgMatches::get_occurrences</code></li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/v4.0.32...v4.1.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=clap&package-manager=cargo&previous-version=4.0.32&new-version=4.1.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam H. Leventhal <ahl@oxide.computer>

Bumps [debug-ignore](https://github.com/sunshowers-code/debug-ignore) from 1.0.3 to 1.0.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/sunshowers-code/debug-ignore/releases">debug-ignore's releases</a>.</em></p> <blockquote> <h2>debug-ignore 1.0.5</h2> <h3>Fixed</h3> <ul> <li>Fixed link to documentation off of main in readme.</li> </ul> <h2>debug-ignore 1.0.4</h2> <h3>Added</h3> <ul> <li>Add a <code>From</code> impl for <code>DebugIgnore</code> (<a href="https://github-redirect.dependabot.com/sunshowers-code/debug-ignore/issues/3">#3</a>).</li> </ul> <p><a href="https://github-redirect.dependabot.com/sunshowers-code/debug-ignore/issues/3">#3</a>: <a href="https://github-redirect.dependabot.com/sunshowers-code/debug-ignore/issues/3">sunshowers-code/debug-ignore#3</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/sunshowers-code/debug-ignore/blob/main/CHANGELOG.md">debug-ignore's changelog</a>.</em></p> <blockquote> <h2>[1.0.5] - 2023-01-07</h2> <h3>Fixed</h3> <ul> <li>Fixed link to documentation off of main in readme.</li> </ul> <h2>[1.0.4] - 2023-01-07</h2> <h3>Added</h3> <ul> <li>Add a <code>From</code> impl for <code>DebugIgnore</code> (<a href="https://github-redirect.dependabot.com/sunshowers-code/debug-ignore/issues/3">#3</a>).</li> </ul> <p><a href="https://github-redirect.dependabot.com/sunshowers-code/debug-ignore/issues/3">#3</a>: <a href="https://github-redirect.dependabot.com/sunshowers-code/debug-ignore/issues/3">sunshowers-code/debug-ignore#3</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/47a911cefdba591b75fba4a3c8e208558d11ebfd"><code>47a911c</code></a> [debug-ignore] version 1.0.5</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/3fe8ddac20c43e8c4301e8f3b1d1fe5bc90ecf41"><code>3fe8dda</code></a> fix link to documentation in readme</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/91417ee991e741c7c71cfc9a734493b8eaf55fb3"><code>91417ee</code></a> [debug-ignore] version 1.0.4</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/3f65de7183a12df836c48b3f4bb69de709e3627b"><code>3f65de7</code></a> add From to example in lib.rs</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/ace6d600f145001b3b0dcdd5c078ddf61183552e"><code>ace6d60</code></a> update changelog</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/80748372372319dd06e37318c95b1390c42584fc"><code>8074837</code></a> add release.toml</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/5456884cd08255fe09b537bc7a413b70897557f9"><code>5456884</code></a> add From impl</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/5d36c5787fcf236bb440122b98773408c148790b"><code>5d36c57</code></a> use cargo test</li> <li><a href="https://github.com/sunshowers-code/debug-ignore/commit/799fe226916a2f76f8a1c9b21398059b9227e2d1"><code>799fe22</code></a> update GHA workflows</li> <li>See full diff in <a href="https://github.com/sunshowers-code/debug-ignore/compare/1.0.3...1.0.5">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=debug-ignore&package-manager=cargo&previous-version=1.0.3&new-version=1.0.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [dropshot](https://github.com/oxidecomputer/dropshot) from `ce5deee` to `120e168`. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/oxidecomputer/dropshot/commit/120e168525018dca4c9c3bed47d398c55fdd4d6b"><code>120e168</code></a> Bump usdt dependency to support stable toolchains (<a href="https://github-redirect.dependabot.com/oxidecomputer/dropshot/issues/522">#522</a>)</li> <li><a href="https://github.com/oxidecomputer/dropshot/commit/3b1195d2eaa0094a5c17b8e44f26b5959e2c6f31"><code>3b1195d</code></a> Update Rust to v1.66.1 (<a href="https://github-redirect.dependabot.com/oxidecomputer/dropshot/issues/559">#559</a>)</li> <li><a href="https://github.com/oxidecomputer/dropshot/commit/ebf16d0fc041e2ccef87be2af5bdfdedf8e779c9"><code>ebf16d0</code></a> Better detection of features necessary for USDT probes (<a href="https://github-redirect.dependabot.com/oxidecomputer/dropshot/issues/561">#561</a>)</li> <li><a href="https://github.com/oxidecomputer/dropshot/commit/3ffc64346bd3fb1cfdebf0612318359a8a175357"><code>3ffc643</code></a> move some api_description functions into new schema_util (<a href="https://github-redirect.dependabot.com/oxidecomputer/dropshot/issues/560">#560</a>)</li> <li><a href="https://github.com/oxidecomputer/dropshot/commit/bf64ab7f61c3f72f3bbcc42f9a3f61ccbe508115"><code>bf64ab7</code></a> move extractors and generic schema utilities out of handler.rs (<a href="https://github-redirect.dependabot.com/oxidecomputer/dropshot/issues/554">#554</a>)</li> <li>See full diff in <a href="https://github.com/oxidecomputer/dropshot/compare/ce5deee415cd1734b07342640fa09af8e7c3a7d0...120e168525018dca4c9c3bed47d398c55fdd4d6b">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [progenitor](https://github.com/oxidecomputer/progenitor) from `634bf98` to `1ef131a`. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/oxidecomputer/progenitor/commit/1ef131a244cf30b9c4213fdb31674fe071cb0043"><code>1ef131a</code></a> Add yaml support (<a href="https://github-redirect.dependabot.com/oxidecomputer/progenitor/issues/227">#227</a>)</li> <li><a href="https://github.com/oxidecomputer/progenitor/commit/6eaacdfc93b6b74c65b96ee68ba7d06e626ac8eb"><code>6eaacdf</code></a> Add method for referring to Client's inner type (<a href="https://github-redirect.dependabot.com/oxidecomputer/progenitor/issues/302">#302</a>)</li> <li><a href="https://github.com/oxidecomputer/progenitor/commit/96af7b5a09498a87e895ef4b9b6f5aca648c3e40"><code>96af7b5</code></a> documentation examples sometimes are not formatted properly (<a href="https://github-redirect.dependabot.com/oxidecomputer/progenitor/issues/303">#303</a>)</li> <li><a href="https://github.com/oxidecomputer/progenitor/commit/582e20c397ab4792492ed220dd9414373348321c"><code>582e20c</code></a> slashes are not properly %-encoded in path parameters (<a href="https://github-redirect.dependabot.com/oxidecomputer/progenitor/issues/304">#304</a>)</li> <li><a href="https://github.com/oxidecomputer/progenitor/commit/a5fea0b0611585eb1c4b8bee651089f8b8547d89"><code>a5fea0b</code></a> fix a warning with base64 (<a href="https://github-redirect.dependabot.com/oxidecomputer/progenitor/issues/300">#300</a>)</li> <li>See full diff in <a href="https://github.com/oxidecomputer/progenitor/compare/634bf98b053cc494191365e2072b3580f4b03df6...1ef131a244cf30b9c4213fdb31674fe071cb0043">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam H. Leventhal <ahl@oxide.computer>

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.17.2 to 0.17.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/console-rs/indicatif/releases">indicatif's releases</a>.</em></p> <blockquote> <h2>0.17.3</h2> <p>A small maintenance release:</p> <ul> <li>address a bunch of lints by <a href="https://github.com/danieleades"><code>@danieleades</code></a> in <a href="https://github-redirect.dependabot.com/console-rs/indicatif/pull/498">console-rs/indicatif#498</a></li> <li>Remove <code>"fs"</code> feature in tokio dependency. by <a href="https://github.com/azriel91"><code>@azriel91</code></a> in <a href="https://github-redirect.dependabot.com/console-rs/indicatif/pull/504">console-rs/indicatif#504</a></li> <li>Update estimate when using a steady ticker by <a href="https://github.com/aawsome"><code>@aawsome</code></a> in <a href="https://github-redirect.dependabot.com/console-rs/indicatif/pull/495">console-rs/indicatif#495</a></li> <li>Refactoring and internal improvements</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/console-rs/indicatif/compare/0.17.2...0.17.3">https://github.com/console-rs/indicatif/compare/0.17.2...0.17.3</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/console-rs/indicatif/commit/84f46b476ad192e58f96b4f91766d1fa958dc735"><code>84f46b4</code></a> 0.17.3</li> <li><a href="https://github.com/console-rs/indicatif/commit/2954b1a24ac5f1900a7861992e4825bff643c9e2"><code>2954b1a</code></a> Update estimate when using a steady ticker</li> <li><a href="https://github.com/console-rs/indicatif/commit/51e85970d8234d73d1a5d6326d8574390bee35e3"><code>51e8597</code></a> Remove <code>"fs"</code> feature in tokio dependency (<a href="https://github-redirect.dependabot.com/console-rs/indicatif/issues/504">#504</a>)</li> <li><a href="https://github.com/console-rs/indicatif/commit/d54fb0ef4c314b3c73fc94372a97f14c4bd32d9e"><code>d54fb0e</code></a> Bump actions/checkout from 2 to 3</li> <li><a href="https://github.com/console-rs/indicatif/commit/16ce199ad106ff17f8ede49640c4eead95a50634"><code>16ce199</code></a> simplify build matrix</li> <li><a href="https://github.com/console-rs/indicatif/commit/1a04604dc6ca58aadc543082d81ee87ca5d0bfe7"><code>1a04604</code></a> add dependabot config</li> <li><a href="https://github.com/console-rs/indicatif/commit/64f357022ea1b0f63b3b3db32b89773609177bb6"><code>64f3570</code></a> avoid accessing 'Default' trait directly</li> <li><a href="https://github.com/console-rs/indicatif/commit/bfeedb210719aa683470c70b0e153d173ce93fd2"><code>bfeedb2</code></a> nest 'or' patterns</li> <li><a href="https://github.com/console-rs/indicatif/commit/75a331165e5196e2a813276f6e49835a8f24ecce"><code>75a3311</code></a> collapse single match blocks</li> <li><a href="https://github.com/console-rs/indicatif/commit/c582ebc616b493fd7c3980ad7cdae0b344347bdc"><code>c582ebc</code></a> use implicit iter loops</li> <li>Additional commits viewable in <a href="https://github.com/console-rs/indicatif/compare/0.17.2...0.17.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=indicatif&package-manager=cargo&previous-version=0.17.2&new-version=0.17.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Crucible changes: Enforce write order of blocks and dirty bit (#1798) Remove unnecessary `unused_async` (#1795) Add `crucible-downstairs validate` subcommand (#1792) Remove incorrect `expect(unused)` annotation (#1789) Handle errors in get-up-state.sh (#1787) Handle final live-repair flush being skipped on all downstairs (#1783) Offline downstairs go to faulted when LR starts (#1777) update drift to pick up fix for OpenAPI type graph cycles (#1785) make Crucible APIs versioned (#1782) Propolis had no changes.

Just Crucible changes, propolis did not change (other than Crucible) Crucible changes: Enforce write order of blocks and dirty bit (#1798) Remove unnecessary `unused_async` (#1795) Add `crucible-downstairs validate` subcommand (#1792) Remove incorrect `expect(unused)` annotation (#1789) Handle errors in get-up-state.sh (#1787) Handle final live-repair flush being skipped on all downstairs (#1783) Offline downstairs go to faulted when LR starts (#1777) update drift to pick up fix for OpenAPI type graph cycles (#1785) make Crucible APIs versioned (#1782) --------- Co-authored-by: Alan Hanson <alan@oxide.computer>

First pass of resource utilization

b1f9749

smklein commented Oct 3, 2022

View reviewed changes

nexus/src/app/sagas/disk_create.rs Outdated Show resolved Hide resolved

smklein added 18 commits October 3, 2022 19:36

less weird formatting

d14f8fc

Tweak CTE to avoid using 'OR'; prefer 'UNION' to avoid full table scans

0430d1a

Merge with snapshots

e8c937f

Add test

55c1e81

test snapshot usage accounting

98467b2

Emit disk usage info to clickhouse

f838365

Add external API endpoint for querying metrics endpoints

f1e4293

fix json spec, minor tweaks

fda4c3c

Fix unauth tests

b20fefd

Boost timeout to not miss utilization

5ff619e

Merge branch 'main' into resource-usage

acb91ee

Add CPUs provisioned

5c33dba

Add resource usage for non-default silo, add fleet to DB

4c48467

Silos know which fleets they belong to, fleet resource accounting

356d4fc

Okay, fleet 'resource_usage' now created / tested, should be working

ba874bb

Actually use CPU accounting saga node

e0f9919

Add test for CPU usage

229480b

RAM provisionining metrics are plumbed

d2c5d9d

smklein commented Oct 7, 2022

View reviewed changes

nexus/src/app/instance.rs Outdated Show resolved Hide resolved

nexus/src/app/sagas/instance_create.rs Outdated Show resolved Hide resolved

smklein requested review from davepacheco, david-crespo and jmpesp October 7, 2022 15:44

minor docs / label updates

2d6f145

smklein mentioned this pull request Oct 7, 2022

Expose silo_id from /session/me endpoint #1796

Merged

jmpesp previously requested changes Oct 7, 2022

View reviewed changes

nexus/src/app/sagas/disk_create.rs Outdated Show resolved Hide resolved

nexus/src/app/sagas/snapshot_create.rs Show resolved Hide resolved

nexus/src/external_api/http_entrypoints.rs Show resolved Hide resolved

common/src/sql/dbinit.sql Outdated Show resolved Hide resolved

Merge branch 'main' into resource-usage

36f7f8f

davepacheco approved these changes Jan 17, 2023

View reviewed changes

david-crespo approved these changes Jan 17, 2023

View reviewed changes

leftwo and others added 20 commits January 17, 2023 20:04

[wicket] add missing serde feature on hex dependency (#2163)

ade45d5

[wicket] move wizard into its own file (#2164)

eaef871

Currently, wizard is the main entrypoint for wicket. To support uploading artifacts, We're going to change it so that it has to go through a clap command layer. Also add a `default-run` so that `cargo run -p wicket` runs the wicket binary.

[wicket] fix call to get_prev_component_id (#2167)

516acbc

Previously, we were showing the next component ID in both the previous and next spots.

Docs on ALLOW_FULL_TABLE_SCAN_SQL

653479b

fix mis-merge

3db02c7

Collect timestamps for Oximeter in CRDB to act as tie-breaker

f96cb95

Merge branch 'main' into resource-usage

5f49bc3

Fix bad merge

57c7389

mitigate races in metrics test

e18210e

Merge branch 'main' into resource-usage

a1feec2

Merge branch 'main' into resource-usage

52af08a

metrics test needs a wider time window, especially when under load

9f498f0

smklein merged commit b33b247 into main Jan 18, 2023

smklein deleted the resource-usage branch January 18, 2023 20:09

smklein mentioned this pull request Jan 20, 2023

thread 'integration_tests::instances::test_instance_metrics' panicked at 'attempt to subtract with overflow' #2186

Closed

luqmana mentioned this pull request Apr 27, 2023

nexus: revamp live migration saga #2892

Merged

jmpesp mentioned this pull request May 23, 2023

API for the current rack utilization #3149

Closed

leftwo mentioned this pull request Oct 24, 2025

Update crucible #9280

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Resource Utilization #1782

Resource Utilization #1782

Uh oh!

smklein commented Oct 3, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smklein commented Oct 7, 2022

Uh oh!

david-crespo commented Oct 7, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davepacheco left a comment

Uh oh!

Uh oh!

davepacheco Jan 17, 2023

Uh oh!

davepacheco Jan 17, 2023

Uh oh!

david-crespo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Resource Utilization #1782

Resource Utilization #1782

Uh oh!

Conversation

smklein commented Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smklein commented Oct 7, 2022

Uh oh!

david-crespo commented Oct 7, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davepacheco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davepacheco Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

davepacheco Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

david-crespo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

smklein commented Oct 3, 2022 •

edited

Loading