Multi series support for event/metric line charts and alerts #171

MikeShi42 · 2023-12-29T23:22:54Z

Introduces a new /chart/series endpoint that will take an array of ChartSeries (logs or metrics) and return them back in series_${i}.data columns. Can also return in ratio mode if seriesReturnType is specified and 2 series are submitted.
Creates a new HDXMultiSeriesTimeChart to render a chart with multiple series. I have a draft of mostly-done HDXMultiSeriesTableChart as well but I figured I'd spare the PR from blowing up further.
Introduces ChartSeriesFormCompact, a hacky-ish UI to fit multiple series in an edit form easily (part of EditMultiSeriesChartForm which will be reused for tables).
Modifies checkAlert to operate on multiple series, mostly through getMultiSeriesChartLegacyFormat which will flatten a multi-series query back into the { data:.., ts_bucket:..., group:... } format.
Adds real CH-backed tests to checkAlert.test.ts and clickhouse.test.ts with some util functions.

changeset-bot · 2023-12-29T23:22:57Z

🦋 Changeset detected

Latest commit: 6fd5d7e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

wrn14897 · 2023-12-30T00:46:49Z

packages/api/src/utils/testUtils.ts

@@ -0,0 +1,142 @@
+import * as clickhouse from '@/clickhouse';


nit: can move this to fixtures.ts

wrn14897 · 2023-12-30T01:14:34Z

packages/api/src/clickhouse/index.ts

+      SqlString.raw(
+        isRate
+          ? rateMetricSource
+          : // Max/Min aggs are the same for both Sum and Gauge metrics


Not sure I fully understand this. So if Mas/Min aggs are the same for both, why do we need extra aggFn checking here ?

This is moved from the existing getMetricsChart function;

In a future PR I actually intend to get rid of this condition, since the sumMetricSource and gaugeMetricSource are nearly identical after the agg fn bug fix I talked about on Slack.

This logic was here because the min/max aggFn didn't require the more expensive sumMetricSource query, even for sum type metrics. However, the gaugeMetricSource and sumMetricSource will in the future be the same query in general, so this shouldn't really matter.

wrn14897 · 2023-12-30T01:21:06Z

packages/api/src/clickhouse/__tests__/clickhouse.test.ts

@@ -1,4 +1,14 @@
+import _ from 'lodash';


Great tests!! not required in this PR, but I'd write tests for buildEventSeriesQuery and queryMultiSeriesChart to make sure queries are properly generated

Ty - I think testing those fns are pretty low ROI test-wise compared to testing it all e2e to make sure the values are actually correct, otherwise I think those tests alone are just going to test fragile implementation details.

wrn14897 · 2023-12-30T01:28:17Z

packages/api/src/clickhouse/index.ts

+    query,
+    format: 'JSON',
+    clickhouse_settings: {
+      additional_table_filters: buildLogStreamAdditionalFilters(


hmm....this is critical and tricky. How do we apply filters separately for log and metric tables ? for now at least we need to execute two queries and merge them afterward

We don't today - the series endpoint can only take series completely of logs or completely of metrics, so we shouldn't need to worry about multi-source charts for now.

I can probably add more validation logic (probably in follow up PR) to throw if that's not true. (For now it'll just create non-sensical queries, but I don't expect us to enter that state to begin with).

wrn14897 · 2023-12-30T01:37:07Z

packages/api/src/clickhouse/index.ts

+  seriesReturnType?: SeriesReturnType;
+}) => {
+  let queries: { query: string; hasGroupBy: boolean }[] = [];
+  if ('table' in series[0] && series[0].table === 'logs') {


This assumes all series have the same table ?

wrn14897 · 2023-12-30T01:47:39Z

packages/api/src/clickhouse/index.ts

+  });
+};
+
+export const getMultiSeriesChartLegacyFormat = async ({


I think we need to test this guy to make sure it outputs the same data as the original chart method

wrn14897 · 2023-12-30T01:49:47Z

packages/api/src/routers/api/chart.ts

+    query: z.object({
+      endTime: z.string(),
+      granularity: z.nativeEnum(clickhouse.Granularity).optional(),
+      startTime: z.string(),
+      seriesReturnType: z.optional(z.nativeEnum(clickhouse.SeriesReturnType)),
+    }),


nit: can move this to body later

wrn14897 · 2023-12-30T01:52:03Z

packages/api/src/tasks/checkAlerts.ts

@@ -336,8 +336,8 @@ export const processAlert = async (now: Date, alert: AlertDocument) => {
    // Logs Source
    let checksData:
      | Awaited<ReturnType<typeof clickhouse.checkAlert>>
-      | Awaited<ReturnType<typeof clickhouse.getLogsChart>>
      | Awaited<ReturnType<typeof clickhouse.getMetricsChart>>


can also move getMetricsChart return type

wrn14897 · 2023-12-30T02:00:44Z

packages/api/src/clickhouse/index.ts

+  };
+};
+
+export const queryMultiSeriesChart = async ({


I can't really read this one and it hurts my brain. I think a simple test helps (at least I can see what the output query looks like). What I can tell so far is it concats a bunch of CTE and merges them afterward

Which lines were too dense to read? If the code can't be read - best that I rewrite the code to make it readable. Fwiw this function literally does just concats a bunch of CTEs and then merges them.

If you want to see the query output - I heard there's a cool tool to observe your systems ;)

wrn14897 · 2023-12-30T04:10:22Z

packages/api/src/clickhouse/index.ts

+
+  const selectClause = [
+    isCountFn
+      ? 'toFloat64(count()) as data'


do we want to do this for other counts ?

Aghhh good catch

wrn14897 · 2023-12-30T04:13:37Z

packages/api/src/clickhouse/index.ts

+  maxNumGroups: number;
+  tableVersion: number | undefined;
+  teamId: string;
+  seriesReturnType?: 'ratio' | 'column';


nit: SeriesReturnType

wrn14897 · 2023-12-30T05:06:30Z

packages/api/src/clickhouse/index.ts

+          seriesReturnType === 'column'
+            ? `greatest(${queries
+                .map((_, i) => `series_${i}.data`)
+                .join(', ')})`


style suggestion: any chance we can move this to the format 2nd arg ? (to improve readability)

wrn14897 · 2023-12-30T05:09:51Z

packages/api/src/clickhouse/index.ts

+  // For now only supports same-table series with the same groupBy
+
+  const seriesCTEs = SqlString.raw(
+    'WITH ' + queries.map((q, i) => `series_${i} AS (${q.query})`).join(',\n'),


style suggestion: I'd move 'WITH' to the final query so its clear that we are building a bunch of CTEs

wrn14897 · 2023-12-30T05:13:02Z

packages/api/src/clickhouse/index.ts

+  let leftJoin = '';
+  // Join every series after the first one
+  for (let i = 1; i < queries.length; i++) {
+    leftJoin += `LEFT JOIN series_${i} AS series_${i} ON series_${i}.ts_bucket=series_0.ts_bucket${


curious why do we need 'AS' here ?

more sql is not more better? 😢

wrn14897 · 2024-01-02T18:18:24Z

packages/api/src/clickhouse/index.ts

+  if (dataType === MetricsDataType.Gauge || dataType === MetricsDataType.Sum) {
+    selectClause.push(
+      aggFn === AggFn.Count
+        ? 'COUNT(value) as data'


nit: I wonder if we also want to do toFloat64 here

Oh good catch, we actually don't technically support the Count aggfn in metrics (not sure why anyone would want it either). Not sure why this is here. I'll ticket this to see if we should just clean it up later instead.

wrn14897

Great work! Very excited about this 🚢 🚢 🚢

wrn14897 reviewed Dec 30, 2023

View reviewed changes

MikeShi42 added 3 commits December 31, 2023 00:00

Multi series support for event/metric line charts and alerts

4fcc499

Address comments, fix bugs, add table chart support

9731133

Add changeset

7f667c7

MikeShi42 force-pushed the mikeshi/multi-series-chart branch from f4ea1ad to 7f667c7 Compare December 31, 2023 08:00

wrn14897 reviewed Jan 2, 2024

View reviewed changes

wrn14897 approved these changes Jan 2, 2024

View reviewed changes

MikeShi42 added the automerge label Jan 2, 2024

Merge branch 'main' into mikeshi/multi-series-chart

6fd5d7e

kodiakhq bot merged commit 3b8effe into main Jan 2, 2024
4 checks passed

kodiakhq bot deleted the mikeshi/multi-series-chart branch January 2, 2024 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi series support for event/metric line charts and alerts #171

Multi series support for event/metric line charts and alerts #171

MikeShi42 commented Dec 29, 2023

changeset-bot bot commented Dec 29, 2023 •

edited

Loading

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023

MikeShi42 Dec 30, 2023

wrn14897 Dec 30, 2023

MikeShi42 Dec 30, 2023

wrn14897 Dec 30, 2023

MikeShi42 Dec 30, 2023

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023 •

edited

Loading

MikeShi42 Dec 30, 2023

wrn14897 Dec 30, 2023

MikeShi42 Dec 30, 2023

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023 •

edited

Loading

wrn14897 Dec 30, 2023

wrn14897 Dec 30, 2023

MikeShi42 Dec 30, 2023

wrn14897 Jan 2, 2024

MikeShi42 Jan 2, 2024

wrn14897 left a comment

		@@ -0,0 +1,142 @@
		import * as clickhouse from '@/clickhouse';

Multi series support for event/metric line charts and alerts #171

Multi series support for event/metric line charts and alerts #171

Conversation

MikeShi42 commented Dec 29, 2023

changeset-bot bot commented Dec 29, 2023 • edited Loading

🦋 Changeset detected

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wrn14897 Dec 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wrn14897 Dec 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wrn14897 left a comment

Choose a reason for hiding this comment

changeset-bot bot commented Dec 29, 2023 •

edited

Loading

wrn14897 Dec 30, 2023 •

edited

Loading

wrn14897 Dec 30, 2023 •

edited

Loading