Collect metrics for allocated CPU, storage, and memory

[david-crespo]

In the console we want to show system-level utilization and capacity over time for CPU, storage, and memory (ideally also broken down by silo, org, project, etc). If I understand correctly, at any given point in time, the relevant data is available in CockroachDB in the instance and disk tables:

omicron/nexus/db-model/src/schema.rs

Lines 105 to 106 in 508906b

	ncpus -> Int8,
	memory -> Int8,

omicron/nexus/db-model/src/schema.rs

Line 24 in 508906b

size_bytes -> Int8,

However, we want to see these values as they change over time. We discussed this at the 9/22 product council and it seemed reasonable that Nexus would send metrics to ClickHouse whenever events happen that change the allocations (likely creation and deletion of instances and disks). Nexus is already set up to be a metrics producer for request latencies:

omicron/nexus/src/context.rs

Lines 115 to 121 in d1fbdd2

	let producer_registry = ProducerRegistry::with_id(config.deployment.id);
	producer_registry
	.register_producer(internal_latencies.clone())
	.unwrap();
	producer_registry
	.register_producer(external_latencies.clone())
	.unwrap();

There are some interesting problems to solve around accumulated values and filtering by project, org, etc. Say a disk is created and we want to record the change in allocated space. We can send a metric to ClickHouse that records the size delta, i.e., the size of the created disk, along with IDs for project, org, and silo to facilitate aggregate queries. But the data we actually want to display in the console is the accumulation of all these changes over time. So to show the disk allocation from time t₁ to t₂, we will need the accumulation of all changes from the beginning of time t₀ to t₁ as the baseline value, and then starting from that we calculate the cumulative change for each point between t₁ and t₂ and return that as the result of the query.

As @bnaecker mentioned here on a related issue, we could calculate and store the cumulative value in each row along with the delta. That way, when we're interested in what happens between t₁ and t₂, we can at least skip the calculation for t₀ to t₁. But wait! If we want to be able to filter by individual disk, project, org, or silo, we'd also need to maintain a cumulative value for each of those groupings. So the row stored in ClickHouse would end up looking like this

storage allocation metric
------------
timestamp
size_bytes (delta bytes)
disk_id
disk_acc_value (bytes)
project_id
project_acc_value (bytes)
org_id
org_acc_value (bytes)
silo_id 
silo_acc_value (bytes)
fleet_acc_value (bytes)

I hope the accumulation side of the problem is common to most metrics and can be solved in a general way.

[leftwo]

For disk metrics, we (being Nexus) can know how much "raw" disk space we have allocated or is free.
For disks that have been created, we don't know how "full" those disks are, as that determination is made at the filesystem layer and we (Nexus) does not have have access to that data.

I think this is consistent with what you are asking for, but I wanted to be sure.

[david-crespo]

Yes, @ahl's thought was that raw allocation is both more immediately useful and easier to get.

[bnaecker]

The linked issue was referring to keeping track of all changes to a cumulative metric, a particular data type used in oximeter. I'm not sure the same logic applies in this case, since the size of any given disk over time (assuming we allow resizes) or even the size of a collection of disks with some property (e.g., project ID) is really a gauge -- it's an instantaneous measurement of the size of that set of disks at any point in time.

What we're storing in values like fleet_acc_bytes is really something I think we should compute on the fly, as the result of a query. For example, we would sum the size of all disks in that case, or all disks with a given project_id for project_acc_value. Why not let the database do its job, and compute those values for us from a query?

[david-crespo]

We want to see these values over time. So unless we want to write down the size of every disk every 10 seconds, then we need to store deltas at events when the total changes, right? Which means the total allocated size at any point in time is the sum of all the deltas from the beginning of time. I assumed we would not want to do that calculation on the fly every time, just like we do not want to do that for the cumulative values.

[smklein]

Including some ideas that we've discussed, even if to just explain why we aren't using them:

Do not store any "aggregated" resource usage in CRDB - recalculate it periodically in a background job, send to Clickhouse.

This approach basically suggests periodically calling SUM over some tables, using e.g. the size_bytes_allocated column for each disk.

Pros:

• Simpler tables
• No overhead / complexity added to operations which consume resources

Cons:

• The background job work would be extremely complex if we're calculating this sum on a project, org, silo, and fleet level. That means periodically iterating over all collections, which is prohibitive.

Determination: Not doing this; the con is too costly.

Store only the deltas in CRDB, flush them periodically to Clickhouse

This means CRDB would add new columns (or tables?) associated with each project, org, silo, and fleet, and update these values when the appropriate resource is modified.

Pros:

• In the steady state, CRDB is doing nothing
• Clickhouse can manage aggregation, which it's already good at

Cons:

• Requires "walking up the tree", making each resource-modifying operation in CRDB have write amplification
• Database denormalization makes code more difficult to write, and makes it easier for data to enter an inconsistent state
• This converts Clickhouse to be a "source-of-truth" for information about capacity, since totals would not be calculated within CRDB. This increases the risks associated with disconnecting, or pruning information from the metrics DB.

Determination: Unclear if this is viable. The associated risks seem high; it forces a significant dependency on Clickhouse for correctness, not just heuristics.

Store a row in a resource_usage table (in CRDB) for every project, org, silo, and fleet. Update all resources back to the root on each resource modification.

This implies that an operation like "create an instance which uses 2GiB memory" would cause:

UPDATE resource_usage
SET memory = memory + 2GiB
WHERE id = project_id OR id = org_id OR id = silo_id OR id = fleet_id;

Pros:

• CRDB maintains position as arbiter of truth.
• Data could (and should!) still be offloaded to Clickhouse for historical perspective.

Cons:

• Requires "walking up the tree", making each resource-modifying operation in CRDB have write amplification
• Database denormalization makes code more difficult to write, and makes it easier for data to enter an inconsistent state

Determination: This might be viable.

[smklein]

To be much more concrete on that last proposal:

CREATE TABLE omicron.public.resource_usage (
  -- Matches the ID of the associated project, organization, silo, or fleet.
  id UUID PRIMARY KEY,

  -- Number of CPUs currently used by running instances
  ncpus -> Int8,

  -- Amount of memory currently used by running instances
  memory -> Int8,

  -- Amount of disk space currently used by all provisioned disks
  disk_bytes -> Int8
);

I'm going to refer to "project, organization, silo, or fleet" as a "collection" from here on, for simplicity.

• Whenever we create a collection, we create a row in the resource_usage table.
• Whenever we hard delete a collection, we delete the corresponding resource_usage row.
• Whenever we perform an operation that alters the resource usage...
  • We update every collection containing that resource.
  • We can use an oximeter producer to offload this information periodically to Clickhouse, for historical aggregation.

This means that CRDB can, at any time, get a precise view of "how much resource usage is provisioned to a particular collection" at the current moment in time. This could be utilized to implement per-{project,org,silo} resource caps in the future.

[david-crespo]

That sounds good to me. Tactically, if we start with fleet-level stats only, we could do a proof of concept for the job sends data to Clickhouse part and leave the CRDB resource_usage side for followup work. That works on the assumption these queries are tolerable to run in a job:

select sum(size_bytes) from disk where time_deleted is null
select sum(ncpus), sum(memory) from instance where time_deleted is null

[askfongjojo]

@david-crespo @smklein - Can this be considered done? Given what is already in place for keeping track of resource usage, we probably want to mark this closed. Future enhancements will likely be covered in some new tickets.

[david-crespo]

Yes, I think so.

[bnaecker]

Seems like there's consensus that this is done, and follow-up work can be tracked separately.