Skip to content

fix: better source validation and refine required source fields#1895

Merged
kodiakhq[bot] merged 21 commits intomainfrom
aaron/sources-but-better
Mar 19, 2026
Merged

fix: better source validation and refine required source fields#1895
kodiakhq[bot] merged 21 commits intomainfrom
aaron/sources-but-better

Conversation

@knudtty
Copy link
Copy Markdown
Contributor

@knudtty knudtty commented Mar 12, 2026

Summary

Large refactor changing the TSource type to a true discriminated union. This means that the expected fields for kind: 'log' will differ from those for 'trace', 'session', 'metrics'. This avoids the current laissez faire source type that currently exists, and required extensive changes across the api and app packages. Also includes a nice addition to useSource - you can now specify a kind field, which will properly infer the type of the returned source.

This also makes use of discriminators in mongoose. This does change a bit of the way that we create and update sources. Obvious changes to sources have also been made, namely making timeValueExpression required on sources. Care has been taken to avoid requiring a migration.

How to test locally or on Vercel

  1. yarn dev
  2. Play around with the app, especially around source creation, source edits, and loading existing sources from a previous version

References

  • Linear Issue: References HDX-3352
  • Related PRs:

Ref: HDX-3352

@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Mar 12, 2026

🦋 Changeset detected

Latest commit: a48ca67

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel
Copy link
Copy Markdown

vercel bot commented Mar 12, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview, Comment Mar 19, 2026 0:55am

Request Review

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 12, 2026

PR Review

Good overall design — the discriminated union approach is sound and the backward-compatibility tests for Session sources are a nice touch.

Critical / Must-fix:

  • ⚠️ createdAt not preserved on kind change (controllers/sources.ts:81-88) → The replaceOne replacement object includes updatedAt: new Date() but never copies existing.createdAt, so changing a source's kind silently resets its creation timestamp. Fix: add createdAt: existing.createdAt to the replacement object.

  • ⚠️ Legacy non-Session sources silently dropped from external API (external-api/v2/sources.ts:78-91) → applyLegacyDefaults only backfills timestampValueExpression for Session sources. Legacy Log/Trace/Metric sources that predate this field being required will fail SourceSchema.safeParse and be silently omitted from the GET /api/v2/sources response (logged as error + filtered out). If such documents exist in production, they'd disappear from the API. Consider adding defaults for other kinds, or at minimum test that each kind handles the missing-timestampValueExpression case.

Minor quality issues:

  • 🔹 The switch in formatExternalSource (external-api/v2/sources.ts:61-74) calls source.toJSON({ getters: true }) identically for all 4 cases. The only value is the exhaustiveness check via satisfies never on the default. Consider collapsing the cases or adding a comment explaining the intent, since it looks like dead code on first read.

  • 🔹 Two @ts-expect-error suppressions in controllers/sources.ts (lines 47, 61) work around discriminator model type incompatibilities. These are explained in comments and appear safe, but worth revisiting if Mongoose types improve.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 12, 2026

E2E Test Results

All tests passed • 91 passed • 3 skipped • 992s

Status Count
✅ Passed 91
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 3

Tests ran across 4 shards in parallel.

View full report →

@knudtty knudtty requested review from a team and pulpdrew and removed request for a team March 18, 2026 03:22
Copy link
Copy Markdown
Contributor

@pulpdrew pulpdrew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Massive improvement! Just a couple questions and minor suggestions

Comment thread packages/api/src/models/source.ts Outdated
Comment thread packages/app/src/components/Sources/SourceForm.tsx
Comment thread packages/common-utils/src/types.ts
Comment thread packages/common-utils/src/types.ts
Comment thread packages/app/src/SessionsPage.tsx Outdated
Comment thread packages/app/src/components/DBSessionPanel.tsx Outdated
Comment thread packages/api/src/tasks/usageStats.ts Outdated
Comment thread packages/api/src/routers/api/sources.ts Outdated
Comment thread packages/api/src/controllers/sources.ts
Comment thread packages/app/src/components/Sources/SourceForm.tsx Outdated
pulpdrew
pulpdrew previously approved these changes Mar 18, 2026
Copy link
Copy Markdown
Contributor

@pulpdrew pulpdrew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread packages/api/src/tasks/usageStats.ts Dismissed
Comment thread packages/api/src/controllers/sources.ts Dismissed
Copy link
Copy Markdown
Contributor

@pulpdrew pulpdrew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@kodiakhq kodiakhq bot merged commit ce85064 into main Mar 19, 2026
20 checks passed
@kodiakhq kodiakhq bot deleted the aaron/sources-but-better branch March 19, 2026 12:56
knudtty added a commit that referenced this pull request Apr 16, 2026
## Summary

Large refactor changing the TSource type to a true discriminated union. This means that the expected fields for `kind: 'log'` will differ from those for `'trace', 'session', 'metrics'`.  This avoids the current laissez faire source type that currently exists, and required extensive changes across the api and app packages. Also includes a nice addition to `useSource` - you can now specify a `kind` field, which will properly infer the type of the returned source. 

This also makes use of discriminators in mongoose. This does change a bit of the way that we create and update sources. Obvious changes to sources have also been made, namely making `timeValueExpression` required on sources. Care has been taken to avoid requiring a migration.

### How to test locally or on Vercel

1. `yarn dev`
2. Play around with the app, especially around source creation, source edits, and loading existing sources from a previous version

### References

- Linear Issue: References HDX-3352
- Related PRs:

Ref: HDX-3352
Copilot AI pushed a commit that referenced this pull request Apr 20, 2026
## Summary

Large refactor changing the TSource type to a true discriminated union. This means that the expected fields for `kind: 'log'` will differ from those for `'trace', 'session', 'metrics'`.  This avoids the current laissez faire source type that currently exists, and required extensive changes across the api and app packages. Also includes a nice addition to `useSource` - you can now specify a `kind` field, which will properly infer the type of the returned source. 

This also makes use of discriminators in mongoose. This does change a bit of the way that we create and update sources. Obvious changes to sources have also been made, namely making `timeValueExpression` required on sources. Care has been taken to avoid requiring a migration.

### How to test locally or on Vercel

1. `yarn dev`
2. Play around with the app, especially around source creation, source edits, and loading existing sources from a previous version

### References

- Linear Issue: References HDX-3352
- Related PRs:

Ref: HDX-3352
Co-authored-by: peter-leonov-ch <209667683+peter-leonov-ch@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants