The 'sled_cpu:cpu_nsec' metric only reports values for the last 100 cpu_ids of the host

[wfchandler]

The sled_cpu:cpu_nsec metric only reports values for the last 100 cpu_ids, e.g., on a gimlet cpus 28 -> 127, on a cosmo 156 -> 255

# gimlet
root@oxz_switch1:~# omdb oxql -e 'get sled_cpu:cpu_nsec | filter timestamp > @now() - 2h && sled_serial == "BRM42220077"' 2>/dev/null | grep cpu_id | awk '{print $2}' | sort -nu
28
29
30
...
125
126
127

# cosmo
support@oxz_switch0:~$ omdb oxql -e 'get sled_cpu:cpu_nsec | filter timestamp > @now() - 2h && sled_serial == "2PNX0JCM"' 2>/dev/null | grep cpu_id | awk '{print $2}' | sort -nu
156
157
158
...
253
254
255

This has been observed on dogfood, colo, and customer racks.

[wfchandler]

CC @bnaecker @jmcarp

[bnaecker]

Interesting. It looks like this might be an OxQL bug, since the actual field values show up in the ClickHouse database on Dogfood:

oxz_clickhouse_aa646c82-c6d7-4d0c-8401-150130927759.local :) select distinct field_value from fields_u32 where timeseries_name = 'sled_cpu:cpu_nsec' and field_name = 'cpu_id';

SELECT DISTINCT field_value
FROM fields_u32
WHERE (timeseries_name = 'sled_cpu:cpu_nsec') AND (field_name = 'cpu_id')

Query id: 96bfaf45-8741-4b9c-aae6-f398e4004dce

┌─field_value─┐
│           0 │
│           1 │
│           2 │
│           3 │
│           4 │
│           5 │
│           6 │
│           7 │
│           8 │
│           9 │
│          10 │
│          11 │
│          12 │
│          13 │
│          14 │
│          15 │
│          16 │
│          17 │
│          18 │
│          19 │
│          20 │
│          21 │
│          22 │
│          23 │
│          24 │
│          25 │
│          26 │
│          27 │
│          28 │
│          29 │
│          30 │
│          31 │
│          32 │
│          33 │
│          34 │
│          35 │
│          36 │
│          37 │
│          38 │
│          39 │
│          40 │
│          41 │
│          42 │
│          43 │
│          44 │
│          45 │
│          46 │
│          47 │
│          48 │
│          49 │
│          50 │
│          51 │
│          52 │
│          53 │
│          54 │
│          55 │
│          56 │
│          57 │
│          58 │
│          59 │
│          60 │
│          61 │
│          62 │
│          63 │
│          64 │
│          65 │
│          66 │
│          67 │
│          68 │
│          69 │
│          70 │
│          71 │
│          72 │
│          73 │
│          74 │
│          75 │
│          76 │
│          77 │
│          78 │
│          79 │
│          80 │
│          81 │
│          82 │
│          83 │
│          84 │
│          85 │
│          86 │
│          87 │
│          88 │
│          89 │
│          90 │
│          91 │
│          92 │
│          93 │
│          94 │
│          95 │
│          96 │
│          97 │
│          98 │
│          99 │
│         100 │
│         101 │
│         102 │
│         103 │
│         104 │
│         105 │
│         106 │
│         107 │
│         108 │
│         109 │
│         110 │
│         111 │
│         112 │
│         113 │
│         114 │
│         115 │
│         116 │
│         117 │
│         118 │
│         119 │
│         120 │
│         121 │
│         122 │
│         123 │
│         124 │
│         125 │
│         126 │
│         127 │
└─────────────┘

128 rows in set. Elapsed: 0.004 sec. Processed 16.38 thousand rows, 850.47 KB (4.57 million rows/s., 237.06 MB/s.)
Peak memory usage: 391.28 KiB.

[jmcarp]

Those timeseries also appear to have recent data in them:

SELECT DISTINCT
    timeseries_name,
    timeseries_key,
    field_value
FROM fields_u32
WHERE (timeseries_name = 'sled_cpu:cpu_nsec') AND (field_name = 'cpu_id') AND (field_value = 9)

Query id: a47bf985-2549-4cde-aca1-089835bb36d9

    ┌─timeseries_name───┬───────timeseries_key─┬─field_value─┐
 1. │ sled_cpu:cpu_nsec │  3669601487346368268 │           9 │
 2. │ sled_cpu:cpu_nsec │  5939249003927999762 │           9 │
 3. │ sled_cpu:cpu_nsec │  7432554866499941122 │           9 │
 4. │ sled_cpu:cpu_nsec │  8917725383257819410 │           9 │
 5. │ sled_cpu:cpu_nsec │  9335399875910169544 │           9 │
 6. │ sled_cpu:cpu_nsec │ 11274982388271390870 │           9 │
 7. │ sled_cpu:cpu_nsec │ 12991911122846929320 │           9 │
 8. │ sled_cpu:cpu_nsec │ 15029230976736234306 │           9 │
 9. │ sled_cpu:cpu_nsec │ 17167221071055480887 │           9 │
10. │ sled_cpu:cpu_nsec │ 17651160603557916902 │           9 │
    └───────────────────┴──────────────────────┴─────────────┘

10 rows in set. Elapsed: 0.004 sec. Processed 8.19 thousand rows, 515.44 KB (2.20 million rows/s., 138.43 MB/s.)
Peak memory usage: 586.49 KiB.

SELECT
    timestamp,
    datum
FROM oximeter.measurements_cumulativeu64
WHERE (timeseries_name = 'sled_cpu:cpu_nsec') AND (timeseries_key = 17651160603557916902)
ORDER BY timestamp DESC
LIMIT 3

Query id: 4c0f04f8-cf2d-43a1-b2ac-d6a6ac36aab5

   ┌─────────────────────timestamp─┬───────datum─┐
1. │ 2026-06-03 21:31:48.715961240 │ 23705179144 │
2. │ 2026-06-03 21:31:18.690038322 │ 23124563656 │
3. │ 2026-06-03 21:30:48.664355331 │ 22549063786 │
   └───────────────────────────────┴─────────────┘

3 rows in set. Elapsed: 0.009 sec. Processed 57.34 thousand rows, 2.12 MB (6.53 million rows/s., 241.88 MB/s.)
Peak memory usage: 4.18 MiB.

[bnaecker]

I'm vaguely suspicious of these queries from the log:

oxz_clickhouse_aa646c82-c6d7-4d0c-8401-150130927759.local :) select query from system.query_log where query like '%sled_cpu:cpu_nsec%' and query like '%another_sort%' order by event_time desc limit 10;

SELECT query
FROM system.query_log
WHERE (query LIKE '%sled_cpu:cpu_nsec%') AND (query LIKE '%another_sort%')
ORDER BY event_time DESC
LIMIT 10

Query id: 53c4e684-dcaa-4079-b0c8-a9e24f90ad4e

┌─query────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ select * from system.query_log where query like '%sled_cpu:cpu_nsec%' and query like '%another_sort%' order by event_time desc limit 10 format Vertical; │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─query──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ select * from system.query_log where query like '%sled_cpu:cpu_nsec%' and query like '%another_sort%' order by event_time desc limit 1 format Vertical;                                                                                                    │
│ select * from system.query_log where query like '%sled_cpu:cpu_nsec%' and query like '%another_sort%' order by event_time desc limit 1;                                                                                                                    │
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:34:48.506168188') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:33:48.660673547') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─query──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:32:48.531614825') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:31:48.468218822') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:30:48.508692443') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:29:48.475886818') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
│ WITH another_sort_bites_the_dust AS (SELECT timeseries_key, start_time, timestamp, datum FROM oximeter.measurements_cumulativeu64 WHERE timeseries_name = 'sled_cpu:cpu_nsec' AND ((greater(timestamp, '2026-06-03 21:28:48.494621058') AND timeseries_key IN (...))) ORDER BY timeseries_key, timestamp DESC LIMIT 1 BY timeseries_key) SELECT * FROM another_sort_bites_the_dust ORDER BY timeseries_key, timestamp LIMIT 1000001 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

10 rows in set. Elapsed: 0.029 sec. Processed 256.49 thousand rows, 264.27 MB (8.76 million rows/s., 9.03 GB/s.)
Peak memory usage: 317.30 MiB.

I believe those are the measurement queries resulting from my OxQL query above. It's been quite a while since I looked at the code, but I'm not sure where that timestamp comes from. It seems plausible that we're inadvertently truncating the first N CPUs from this query, if they happen to occur just after that timestamp. I don't have much evidence for that at this point, but I think we can dig a little.

[bnaecker]

The plot thickens. We seem to have different numbers of timeseries when the CPU ID is something < 28 vs when it's a value >= 28:

oxz_clickhouse_aa646c82-c6d7-4d0c-8401-150130927759.local :) select count(distinct *) from fields_u32 where timeseries_name = 'sled_cpu:cpu_nsec' and field_name = 'cpu_id' and field_value = 27;

SELECT countDistinct(*)
FROM fields_u32
WHERE (timeseries_name = 'sled_cpu:cpu_nsec') AND (field_name = 'cpu_id') AND (field_value = 27)

Query id: 76fbab35-93aa-438d-9c5a-adaac3c6e6db

┌─uniqExact(timeseries_name, timeseries_key, field_name, field_value)─┐
│                                                                  10 │
└─────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.008 sec. Processed 8.19 thousand rows, 515.36 KB (1.08 million rows/s., 68.23 MB/s.)
Peak memory usage: 830.10 KiB.

oxz_clickhouse_aa646c82-c6d7-4d0c-8401-150130927759.local :) select count(distinct *) from fields_u32 where timeseries_name = 'sled_cpu:cpu_nsec' and field_name = 'cpu_id' and field_value = 28;

SELECT countDistinct(*)
FROM fields_u32
WHERE (timeseries_name = 'sled_cpu:cpu_nsec') AND (field_name = 'cpu_id') AND (field_value = 28)

Query id: 83137369-7727-4bf4-972d-babe3e5212a4

┌─uniqExact(timeseries_name, timeseries_key, field_name, field_value)─┐
│                                                                  70 │
└─────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.008 sec. Processed 8.57 thousand rows, 523.91 KB (1.03 million rows/s., 62.90 MB/s.)
Peak memory usage: 841.88 KiB.

[jmcarp]

Here's a weird one. We can see the sql queries for a given oxql query if we ask for them:

joshcarp@j0sh ~/c/o/oximeter (main)> time oxide --insecure --profile r2 api -X POST /v1/system/timeseries/query -H 'Content-Type: application/json' --field query='get sled_cpu:cpu_nsec | filter timestamp > @now() - 5m | filter sled_serial == "BRM44220011" | filter cpu_id == 9 | last 1' --field include_summaries=true | jq -r '.query_summaries[0].query'
SELECT fields_u32_pivot.cpu_id AS cpu_id, fields_uuid_pivot.rack_id AS rack_id, fields_uuid_pivot.sled_id AS sled_id, fields_string_pivot.sled_model AS sled_model, fields_u32_pivot.sled_revision AS sled_revision, fields_string_pivot.sled_serial AS sled_serial, fields_string_pivot.state AS state, fields_string_pivot.timeseries_key AS timeseries_key FROM (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'sled_model') AS sled_model, anyIf(field_value, field_name = 'sled_serial') AS sled_serial, anyIf(field_value, field_name = 'state') AS state
                            FROM oximeter.fields_string
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_string_pivot JOIN (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'cpu_id') AS cpu_id, anyIf(field_value, field_name = 'sled_revision') AS sled_revision
                            FROM oximeter.fields_u32
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_u32_pivot ON fields_string_pivot.timeseries_key = fields_u32_pivot.timeseries_key JOIN (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'rack_id') AS rack_id, anyIf(field_value, field_name = 'sled_id') AS sled_id
                            FROM oximeter.fields_uuid
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_uuid_pivot ON fields_string_pivot.timeseries_key = fields_uuid_pivot.timeseries_key WHERE and(equals(cpu_id, 9), equals(sled_serial, 'BRM44220011'))

But running this query in clickhouse gives no rows. So I must have been wrong about these fields existing above—I forgot to join all the relevant field tables, and only proved that some sleds had metrics with cpu_id < 28.

[bnaecker]

The above differences in counts appears to be related to the sled model and possibly revision. We only have 70-ish distinct CPU IDs for sleds with model `913-0000023', while we have 100 for '913-0000019':

oxz_clickhouse_aa646c82-c6d7-4d0c-8401-150130927759.local :) with data as (SELECT fields_u32_pivot.cpu_id AS cpu_id, fields_uuid_pivot.rack_id AS rack_id, fields_uuid_pivot.sled_id AS sled_id, fields_string_pivot.sled_model AS sled_model, fields_u32_pivot.sled_revision AS sled_revision, fields_string_pivot.sled_serial AS sled_serial, fields_string_pivot.state AS state, fields_string_pivot.timeseries_key AS timeseries_key FROM (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'sled_model') AS sled_model, anyIf(field_value, field_name = 'sled_serial') AS sled_serial, anyIf(field_value, field_name = 'state') AS state
                            FROM oximeter.fields_string
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_string_pivot JOIN (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'cpu_id') AS cpu_id, anyIf(field_value, field_name = 'sled_revision') AS sled_revision
                            FROM oximeter.fields_u32
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_u32_pivot ON fields_string_pivot.timeseries_key = fields_u32_pivot.timeseries_key JOIN (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'rack_id') AS rack_id, anyIf(field_value, field_name = 'sled_id') AS sled_id
                            FROM oximeter.fields_uuid
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_uuid_pivot ON fields_string_pivot.timeseries_key = fields_uuid_pivot.timeseries_key) select count(distinct cpu_id) from data where sled_model = '913-0000019';

WITH data AS
    (
        SELECT
            fields_u32_pivot.cpu_id AS cpu_id,
            fields_uuid_pivot.rack_id AS rack_id,
            fields_uuid_pivot.sled_id AS sled_id,
            fields_string_pivot.sled_model AS sled_model,
            fields_u32_pivot.sled_revision AS sled_revision,
            fields_string_pivot.sled_serial AS sled_serial,
            fields_string_pivot.state AS state,
            fields_string_pivot.timeseries_key AS timeseries_key
        FROM
        (
            SELECT
                timeseries_key,
                anyIf(field_value, field_name = 'sled_model') AS sled_model,
                anyIf(field_value, field_name = 'sled_serial') AS sled_serial,
                anyIf(field_value, field_name = 'state') AS state
            FROM oximeter.fields_string
            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
            GROUP BY timeseries_key
        ) AS fields_string_pivot
        INNER JOIN
        (
            SELECT
                timeseries_key,
                anyIf(field_value, field_name = 'cpu_id') AS cpu_id,
                anyIf(field_value, field_name = 'sled_revision') AS sled_revision
            FROM oximeter.fields_u32
            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
            GROUP BY timeseries_key
        ) AS fields_u32_pivot ON fields_string_pivot.timeseries_key = fields_u32_pivot.timeseries_key
        INNER JOIN
        (
            SELECT
                timeseries_key,
                anyIf(field_value, field_name = 'rack_id') AS rack_id,
                anyIf(field_value, field_name = 'sled_id') AS sled_id
            FROM oximeter.fields_uuid
            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
            GROUP BY timeseries_key
        ) AS fields_uuid_pivot ON fields_string_pivot.timeseries_key = fields_uuid_pivot.timeseries_key
    )
SELECT countDistinct(cpu_id)
FROM data
WHERE sled_model = '913-0000019'

Query id: 5c3feb24-b9b2-4049-afaf-a59462a9892f

┌─uniqExact(cpu_id)─┐
│               100 │
└───────────────────┘

1 row in set. Elapsed: 0.067 sec. Processed 68.83 thousand rows, 3.86 MB (1.03 million rows/s., 57.52 MB/s.)
Peak memory usage: 8.66 MiB.

oxz_clickhouse_aa646c82-c6d7-4d0c-8401-150130927759.local :) with data as (SELECT fields_u32_pivot.cpu_id AS cpu_id, fields_uuid_pivot.rack_id AS rack_id, fields_uuid_pivot.sled_id AS sled_id, fields_string_pivot.sled_model AS sled_model, fields_u32_pivot.sled_revision AS sled_revision, fields_string_pivot.sled_serial AS sled_serial, fields_string_pivot.state AS state, fields_string_pivot.timeseries_key AS timeseries_key FROM (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'sled_model') AS sled_model, anyIf(field_value, field_name = 'sled_serial') AS sled_serial, anyIf(field_value, field_name = 'state') AS state
                            FROM oximeter.fields_string
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_string_pivot JOIN (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'cpu_id') AS cpu_id, anyIf(field_value, field_name = 'sled_revision') AS sled_revision
                            FROM oximeter.fields_u32
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_u32_pivot ON fields_string_pivot.timeseries_key = fields_u32_pivot.timeseries_key JOIN (
                            SELECT
                                timeseries_key, anyIf(field_value, field_name = 'rack_id') AS rack_id, anyIf(field_value, field_name = 'sled_id') AS sled_id
                            FROM oximeter.fields_uuid
                            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
                            GROUP BY timeseries_key
                        ) AS fields_uuid_pivot ON fields_string_pivot.timeseries_key = fields_uuid_pivot.timeseries_key) select count(distinct cpu_id) from data where sled_model = '913-0000023';

WITH data AS
    (
        SELECT
            fields_u32_pivot.cpu_id AS cpu_id,
            fields_uuid_pivot.rack_id AS rack_id,
            fields_uuid_pivot.sled_id AS sled_id,
            fields_string_pivot.sled_model AS sled_model,
            fields_u32_pivot.sled_revision AS sled_revision,
            fields_string_pivot.sled_serial AS sled_serial,
            fields_string_pivot.state AS state,
            fields_string_pivot.timeseries_key AS timeseries_key
        FROM
        (
            SELECT
                timeseries_key,
                anyIf(field_value, field_name = 'sled_model') AS sled_model,
                anyIf(field_value, field_name = 'sled_serial') AS sled_serial,
                anyIf(field_value, field_name = 'state') AS state
            FROM oximeter.fields_string
            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
            GROUP BY timeseries_key
        ) AS fields_string_pivot
        INNER JOIN
        (
            SELECT
                timeseries_key,
                anyIf(field_value, field_name = 'cpu_id') AS cpu_id,
                anyIf(field_value, field_name = 'sled_revision') AS sled_revision
            FROM oximeter.fields_u32
            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
            GROUP BY timeseries_key
        ) AS fields_u32_pivot ON fields_string_pivot.timeseries_key = fields_u32_pivot.timeseries_key
        INNER JOIN
        (
            SELECT
                timeseries_key,
                anyIf(field_value, field_name = 'rack_id') AS rack_id,
                anyIf(field_value, field_name = 'sled_id') AS sled_id
            FROM oximeter.fields_uuid
            WHERE timeseries_name = 'sled_cpu:cpu_nsec'
            GROUP BY timeseries_key
        ) AS fields_uuid_pivot ON fields_string_pivot.timeseries_key = fields_uuid_pivot.timeseries_key
    )
SELECT countDistinct(cpu_id)
FROM data
WHERE sled_model = '913-0000023'

Query id: c8fa87e1-db6b-477d-98bd-c2866f716cd5

┌─uniqExact(cpu_id)─┐
│                72 │
└───────────────────┘

1 row in set. Elapsed: 0.075 sec. Processed 109.14 thousand rows, 6.35 MB (1.46 million rows/s., 84.80 MB/s.)
Peak memory usage: 10.65 MiB.

It's also useful to note we're only getting the last 100 CPUs for 913-0000019 and the first 72 CPUs for 913-0000023.

[bnaecker]

Looks like we're dropping samples:

0x〉get kstat_sampler:samples_dropped | filter target_name == "sled_cpu" | last 1

kstat_sampler:samples_dropped

 hostname: BRM42220057
 target_id: 13624226742619834788
 target_name: sled_cpu
   [2026-06-03 20:31:51.596307200, 2026-06-03 22:12:40.714947286]: [285560]

 hostname: BRM44220005
 target_id: 4533553309558607539
 target_name: sled_cpu
   [2026-06-03 20:42:18.861602813, 2026-06-03 22:12:26.191189105]: [255100]

 hostname: BRM42220009
 target_id: 15442156270748774863
 target_name: sled_cpu
   [2026-06-03 19:38:23.240304503, 2026-06-03 22:12:33.952189706]: [446720]

 hostname: BRM22250001
 target_id: 15561978571055507045
 target_name: sled_cpu
   [2026-06-03 21:23:02.683023678, 2026-06-03 22:12:45.412726034]: [58000]

 hostname: BRM42220017
 target_id: 12280116768737819899
 target_name: sled_cpu
   [2026-06-03 20:05:42.382368556, 2026-06-03 22:12:47.018004587]: [361540]

 hostname: BRM27230045
 target_id: 8263107911170423284
 target_name: sled_cpu
   [2026-06-03 19:20:24.286369820, 2026-06-03 22:12:44.227195634]: [495980]

 hostname: BRM44220010
 target_id: 8867022197890707382
 target_name: sled_cpu
   [2026-06-03 20:50:37.754398478, 2026-06-03 22:12:25.224195576]: [231600]

 hostname: BRM42220016
 target_id: 4094064889712653352
 target_name: sled_cpu
   [2026-06-03 19:46:29.317158530, 2026-06-03 22:12:29.904332436]: [412860]

 hostname: BRM42220006
 target_id: 10357125765981304162
 target_name: sled_cpu
   [2026-06-03 19:27:30.620902989, 2026-06-03 22:12:45.973096050]: [474960]

 hostname: BRM42220014
 target_id: 9400342352715252881
 target_name: sled_cpu
   [2026-06-03 19:54:54.199134464, 2026-06-03 22:12:31.728945679]: [390720]

 hostname: BRM42220051
 target_id: 17545891633865439388
 target_name: sled_cpu
   [2026-06-03 20:23:42.741191845, 2026-06-03 22:12:44.149022770]: [307780]

 hostname: BRM44220011
 target_id: 8635689701382024369
 target_name: sled_cpu
   [2026-06-03 20:58:52.275637932, 2026-06-03 22:12:45.900039650]: [209520]

 hostname: BRM13250012
 target_id: 15657696958500352037
 target_name: sled_cpu
   [2026-06-03 21:09:47.594610245, 2026-06-03 22:12:40.875310815]: [57780]

 hostname: BRM42220031
 target_id: 11988938253002909027
 target_name: sled_cpu
   [2026-06-03 20:15:35.877025046, 2026-06-03 22:12:24.952562417]: [331860]

Josh pointed to the const limit on the the per-target kstat sampler queue here:

omicron/oximeter/instruments/src/kstat/sampler.rs

Line 1212 in c5c595d

pub const DEFAULT_SAMPLE_LIMIT: usize = 500;

That's definitely not going to work for this case. We have 128 CPUs, with 5 states (idle, kernel, user, dtrace, intr), which is 640 samples. We'll never get everything in one sample interval.

[bnaecker]

This target needs to override the default sample limit, and probably also the one recording the per-zone stats as well. I'm wondering if we actually want to have no default here at all, to force the developer to right-size the queue based on their expected sample rates.