Repository navigation
feat: add PostHog exporter for AI observability - #10393
Conversation
Adds a new observability package for PostHog integration. Includes tracing implementation, tests, and configuration.
…nd add comprehensive test coverage
…es, and changeset
Streamline PostHog exporter docs and reference to match the concise format used by other exporters (Langfuse, Braintrust, LangSmith, Arize).
🦋 Changeset detectedLatest commit: 0d4bcef The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@camachoh1 is attempting to deploy a commit to the Mastra Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (1)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including You can disable this status message by setting the WalkthroughAdds a new PostHog AI observability exporter package including implementation, tests, packaging and build configs, ESLint/Vitest setup, and user/reference documentation for integrating Mastra traces with PostHog LLM Analytics. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Files/areas to inspect closely:
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
observability/posthog/src/tracing.ts (2)
315-347: Consider surfacing more LLM attributes (resultType, finishReason, etc.) in PostHog properties
buildGenerationPropertiescurrently mapsmodel,provider, token usage, core parameters, and streaming, but other usefulModelGenerationAttributesfields (likeresultType,finishReason, and potentially additional parameters) are not forwarded.To reduce information loss and make AI dashboards richer, consider adding a few more well‑named properties, for example:
if (attrs.parameters) { if (attrs.parameters.temperature !== undefined) props.$ai_temperature = attrs.parameters.temperature; if (attrs.parameters.maxOutputTokens !== undefined) props.$ai_max_tokens = attrs.parameters.maxOutputTokens; } if (attrs.streaming !== undefined) props.$ai_stream = attrs.streaming; + + if (attrs.resultType) props.$ai_result_type = attrs.resultType; + if (attrs.finishReason) props.$ai_finish_reason = attrs.finishReason;(You can adjust naming to match any internal conventions, but having these in PostHog would help downstream analysis.)
137-203: Optional: guard against orphaned traces if SPAN_ENDED never arrives
traceMapentries are removed only incleanupSpanwhen the last span in a trace ends. If, for any reason, aSPAN_ENDEDevent is never emitted (process crash, instrumentation bug, etc.), that trace’s entry will stick around for the life of the exporter.For long‑running processes with high traffic this could accumulate. If you expect such scenarios, consider adding a simple TTL‑based cleanup (e.g., periodically pruning traces whose last
startTimeis older than some cutoff) or a max-size guard fortraceMap. Not urgent, but it’s a straightforward resilience improvement.observability/posthog/src/tracing.test.ts (1)
252-273: Optional: avoid double‑shutdown in tests to keep mocks simplerWith the top‑level
afterEachalready callingexporter.shutdown(), theCleanuptest’s explicitawait exporter.shutdown();results inshutdownbeing invoked twice for that test. It’s harmless (and your mock doesn’t assert on call counts), but you could rely solely on the sharedafterEachand remove the explicit call here to keep the lifecycle pattern uniform.Also applies to: 268-273
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (13)
.changeset/add-posthog-exporter.md(1 hunks)docs/src/content/en/docs/observability/tracing/exporters/posthog.mdx(1 hunks)docs/src/content/en/reference/observability/tracing/exporters/posthog.mdx(1 hunks)observability/posthog/README.md(1 hunks)observability/posthog/eslint.config.js(1 hunks)observability/posthog/package.json(1 hunks)observability/posthog/src/index.ts(1 hunks)observability/posthog/src/tracing.test.ts(1 hunks)observability/posthog/src/tracing.ts(1 hunks)observability/posthog/tsconfig.build.json(1 hunks)observability/posthog/tsconfig.json(1 hunks)observability/posthog/tsup.config.ts(1 hunks)observability/posthog/vitest.config.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
.changeset/*.md
⚙️ CodeRabbit configuration file
.changeset/*.md: Review the changeset file (which will become part of a CHANGELOG.md file) according to these guidelines:
- The target audience are developers
- Write short, direct sentences that anyone can understand. Avoid commit messages, technical jargon, and acronyms. Use action-oriented verbs (Added, Fixed, Improved, Deprecated, Removed)
- Avoid generic phrases like "Update code", "Miscellaneous improvements", or "Bug fixes"
- Highlight outcomes rather than internal code details
- Add context like links to issues or PRs when relevant
- If the change is a breaking change or is adding a new feature, ensure that a code example is provided showing its (changed) usage
- Keep the formatting easy-to-read and scannable. If necessary, use bullet points or subheadings (subheadings must start with heading level 4 - ####)
- For larger, more substantial changes, also answer the "Why" behind the changes
Files:
.changeset/add-posthog-exporter.md
🧬 Code graph analysis (2)
observability/posthog/src/tracing.ts (2)
observability/mastra/src/exporters/base.ts (1)
BaseExporterConfig(17-22)packages/core/src/observability/types/tracing.ts (3)
TracingEvent(790-793)AnyExportedSpan(429-429)ModelGenerationAttributes(96-123)
observability/posthog/src/tracing.test.ts (1)
observability/posthog/src/tracing.ts (1)
PosthogExporter(51-413)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Socket Security: Pull Request Alerts
🔇 Additional comments (12)
observability/posthog/eslint.config.js (1)
1-6: LGTM!The ESLint configuration correctly uses top-level await to load the shared config factory and exports the result. This pattern aligns with the async config pattern used across the observability suite.
observability/posthog/src/index.ts (1)
1-1: LGTM!The barrel export correctly re-exports the tracing module, establishing the public API surface for the PostHog observability package.
observability/posthog/tsconfig.json (1)
1-5: LGTM!The TypeScript configuration correctly extends the base Node.js config, includes source files and build config, and appropriately excludes test files from the main compilation scope.
observability/posthog/tsconfig.build.json (1)
1-9: LGTM!The build configuration correctly extends the base configs, sets appropriate output/root directories, and excludes test and mock files from the distribution build.
observability/posthog/vitest.config.ts (1)
1-8: LGTM!The Vitest configuration correctly sets up the Node environment and discovers test files from the expected location, aligning with the test file exclusions in the TypeScript configs.
observability/posthog/tsup.config.ts (1)
1-17: LGTM!The tsup configuration correctly sets up dual-format builds (ESM/CJS), enables code splitting and tree-shaking, and uses a custom type generation hook instead of built-in DTS generation. This aligns with the package's build requirements.
docs/src/content/en/docs/observability/tracing/exporters/posthog.mdx (1)
1-107: LGTM!The documentation is comprehensive and well-structured, covering installation, prerequisites, configuration options, and special modes (serverless and privacy). Code examples are clear and demonstrate both basic and advanced usage patterns. The inclusion of environment variable setup and regional deployment options enhances usability.
docs/src/content/en/reference/observability/tracing/exporters/posthog.mdx (1)
119-131: Span type mapping matches implementationThe span-type → PostHog event mapping here (
MODEL_GENERATION/MODEL_STEP→$ai_generation, everything else →$ai_span) is consistent withmapToPostHogEventintracing.ts, which is great for keeping docs and behavior in sync.observability/posthog/README.md (1)
35-150: README usage and feature documentation look solidThe installation, Mastra integration example, feature list (AI tracing, token usage, serverless, privacy mode), and metadata/cost-tracking sections all accurately reflect the exporter behavior in
tracing.tsand provide enough detail for users to get started and reason about what gets sent to PostHog.observability/posthog/package.json (1)
1-62: Package configuration looks consistent and ready for publishExports, entrypoints (
main/types), runtime deps (posthog-node,@mastra/observability), and the peer dependency on@mastra/coreall look coherent and aligned with howtracing.tsis built and consumed. No issues from this file.observability/posthog/src/tracing.ts (1)
51-191: Exporter core behavior (init, span lifecycle, batching) looks correctThe constructor’s API-key guard, serverless-aware
buildClientConfig, span start/end handling (latency, distinctId resolution, cleanup), and the$ai_generation/$ai_spanrouting all line up well with both the tracing types and the tests. Error handling viatry/catchin_exportTracingEventpluslogger.errorkeeps failures from bubbling into callers.observability/posthog/src/tracing.test.ts (1)
1-773: Comprehensive test coverage for exporter behaviorThe suite exercises initialization paths, span lifecycle (including out‑of‑order and concurrent traces), distinctId resolution, LLM property mapping, privacy mode, error handling, event spans, and message formatting against the concrete
PosthogExporterimplementation. This gives high confidence the exporter will behave as documented.
- Add usage example to changeset - Fix logLevel default mismatch (INFO vs warn) in reference docs - Fix misleading enablePrivacyMode comment in README
| type: span.type, | ||
| }); | ||
|
|
||
| if (span.parentSpanId === undefined || !traceData.distinctId) { |
There was a problem hiding this comment.
why do we need a check for root node here? span.parentSpanId === undefined?
| timestamp: new Date(endTime), | ||
| }); | ||
|
|
||
| this.cleanupSpan(span.traceId, span.id); |
There was a problem hiding this comment.
it might be better to just delete this span from traceData here. You already have a valid traceData object... don't need to re-pull and test for it in the helper.
| if (!this.config.enablePrivacyMode) { | ||
| if (span.input) props.$ai_input = this.formatMessages(span.input, 'user'); | ||
| if (span.output) props.$ai_output_choices = this.formatMessages(span.output, 'assistant'); | ||
| } |
There was a problem hiding this comment.
maybe we should just let the posthog SDK take care of removing these items when this config is set, instead of removing the data ourselves?
epinzur
left a comment
There was a problem hiding this comment.
Added a few minor comments. Overall this looks great! Awesome work. Thanks!
… cleanup, use SDK privacy mode - Remove redundant parentSpanId check in handleSpanStarted - Inline span cleanup in handleSpanEnded to avoid extra Map lookup - Use PostHog SDK's native privacyMode config instead of manual property stripping - Update tests to verify privacyMode config is passed correctly
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
observability/posthog/src/tracing.ts (2)
51-75: Prefer an explicit nullable type forclientovernull as anyWhen the API key is missing,
clientis set tonull as anyeven though it’s declared asPostHog. This works but sidesteps type safety and invites accidental misuse. Consider declaring it asprivate client: PostHog | null(initialised tonull) and relying on normal null‑checks in_exportTracingEvent/shutdowninstead of theas anyescape hatch.
117-137: UseTracingEventTypeconstants instead of string literals and clarifySPAN_UPDATEDbehaviour
event.typeis compared against hard‑coded strings ('span_started','span_ended'), andSPAN_UPDATEDis currently a no‑op. Importing and using theTracingEventTypeenum here would give compile‑time safety if values change, and a brief comment or assertion about intentionally ignoring updates would make the behaviour clearer for future maintainers.observability/posthog/src/tracing.test.ts (2)
31-35: Optionally resetexporterafter shutdown to avoid stale references
afterEachshuts down the currentexporterinstance but keeps the variable pointing at it. Today every test assignsexporterbefore use, so this is safe, but settingexporter = undefined as any(or similar) after shutdown would guard against future tests that accidentally rely on a leftover instance.
770-787: Consider a deterministic ID generator increateSpan
createSpanusesMath.random()for defaultidandtraceId. Since no assertions depend on those values this is fine, but using a simple counter or fixed default values would make debugging easier and avoid any chance of future tests becoming flaky if they start inspecting IDs.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
observability/posthog/src/tracing.test.ts(1 hunks)observability/posthog/src/tracing.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Run
pnpm typecheckto validate TypeScript types across all packages
Files:
observability/posthog/src/tracing.test.tsobservability/posthog/src/tracing.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Run
pnpm prettier:formatto format code andpnpm formatto run linting with auto-fix across all packages
Files:
observability/posthog/src/tracing.test.tsobservability/posthog/src/tracing.ts
**/*.{test,spec}.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{test,spec}.{ts,tsx}: Use Vitest for testing framework in test files
Co-locate test files with source code using naming patterns like *.test.ts or *.spec.ts
Files:
observability/posthog/src/tracing.test.ts
🧠 Learnings (5)
📚 Learning: 2025-11-24T16:42:04.235Z
Learnt from: CR
Repo: mastra-ai/mastra PR: 0
File: packages/codemod/AGENTS.md:0-0
Timestamp: 2025-11-24T16:42:04.235Z
Learning: Applies to packages/codemod/src/test/**/*.test.ts : Include test cases for multiple occurrences of the transformation pattern, aliased imports, type imports, and mixed imports to verify the codemod works consistently
Applied to files:
observability/posthog/src/tracing.test.ts
📚 Learning: 2025-11-24T16:42:04.235Z
Learnt from: CR
Repo: mastra-ai/mastra PR: 0
File: packages/codemod/AGENTS.md:0-0
Timestamp: 2025-11-24T16:42:04.235Z
Learning: Applies to packages/codemod/src/test/__fixtures__/**/*.ts : Create test fixtures by copying examples DIRECTLY from migration guides in `docs/src/content/en/guides/migrations/upgrade-to-v1/` without hallucinating or inventing changes
Applied to files:
observability/posthog/src/tracing.test.ts
📚 Learning: 2025-11-24T16:42:04.235Z
Learnt from: CR
Repo: mastra-ai/mastra PR: 0
File: packages/codemod/AGENTS.md:0-0
Timestamp: 2025-11-24T16:42:04.235Z
Learning: Applies to packages/codemod/src/test/__fixtures__/**/*.ts : In output fixtures, ensure all NEGATIVE test cases remain EXACTLY IDENTICAL to their input fixture counterparts to verify the codemod only transforms intended patterns
Applied to files:
observability/posthog/src/tracing.test.ts
📚 Learning: 2025-11-24T16:42:04.235Z
Learnt from: CR
Repo: mastra-ai/mastra PR: 0
File: packages/codemod/AGENTS.md:0-0
Timestamp: 2025-11-24T16:42:04.235Z
Learning: Applies to packages/codemod/src/test/__fixtures__/**/*.ts : In input fixtures, include both POSITIVE test cases (patterns that should transform) and NEGATIVE test cases (unrelated code with similar names/patterns that should NOT transform)
Applied to files:
observability/posthog/src/tracing.test.ts
📚 Learning: 2025-11-24T16:40:29.603Z
Learnt from: CR
Repo: mastra-ai/mastra PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-11-24T16:40:29.603Z
Learning: Applies to packages/**/*.{ts,tsx} : Use telemetry decorators for observability across components
Applied to files:
observability/posthog/src/tracing.test.ts
🧬 Code graph analysis (1)
observability/posthog/src/tracing.test.ts (1)
observability/posthog/src/tracing.ts (1)
PosthogExporter(51-409)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Socket Security: Pull Request Alerts
🔇 Additional comments (2)
observability/posthog/src/tracing.ts (1)
313-336: Generation and token‑usage property mapping looks correct and well‑coveredThe mapping from
ModelGenerationAttributesinto$ai_*properties (model/provider, v4/v5 token fields with the right precedence, extra usage fields) is clear and matches the expectations encoded in the tests. No changes needed here.observability/posthog/src/tracing.test.ts (1)
276-376: Nice coverage of span type mapping and LLM property extractionThe tests around
Span Type MappingandLLM Generation Propertiesexercise the key branches inmapToPostHogEventandbuildGenerationProperties(MODEL_GENERATION vs MODEL_STEP, chunk attributes, v4/v5 token formats, and defaults). This gives good confidence that the exporter’s core analytics surface behaves as intended.
| private extractCustomMetadata(span: AnyExportedSpan): Record<string, any> { | ||
| const { userId, sessionId, ...customMetadata } = span.metadata ?? {}; | ||
| return customMetadata; | ||
| } | ||
|
|
||
| private buildGenerationProperties(span: AnyExportedSpan): Record<string, any> { | ||
| const props: Record<string, any> = {}; | ||
| const attrs = (span.attributes ?? {}) as ModelGenerationAttributes; | ||
|
|
||
| props.$ai_model = attrs.model || 'unknown-model'; | ||
| props.$ai_provider = attrs.provider || 'unknown-provider'; | ||
|
|
||
| if (span.input) props.$ai_input = this.formatMessages(span.input, 'user'); | ||
| if (span.output) props.$ai_output_choices = this.formatMessages(span.output, 'assistant'); | ||
|
|
||
| if (attrs.usage) { | ||
| const { usage } = attrs; | ||
| const inputTokens = usage.inputTokens ?? usage.promptTokens; | ||
| const outputTokens = usage.outputTokens ?? usage.completionTokens; | ||
| const totalTokens = usage.totalTokens; | ||
|
|
||
| if (inputTokens !== undefined) props.$ai_input_tokens = inputTokens; | ||
| if (outputTokens !== undefined) props.$ai_output_tokens = outputTokens; | ||
| if (totalTokens !== undefined) props.$ai_total_tokens = totalTokens; | ||
|
|
||
| if (usage.reasoningTokens !== undefined) props.reasoning_tokens = usage.reasoningTokens; | ||
| if (usage.cachedInputTokens !== undefined) props.cached_input_tokens = usage.cachedInputTokens; | ||
| } | ||
|
|
||
| if (attrs.parameters) { | ||
| if (attrs.parameters.temperature !== undefined) props.$ai_temperature = attrs.parameters.temperature; | ||
| if (attrs.parameters.maxOutputTokens !== undefined) props.$ai_max_tokens = attrs.parameters.maxOutputTokens; | ||
| } | ||
| if (attrs.streaming !== undefined) props.$ai_stream = attrs.streaming; | ||
|
|
||
| return { ...props, ...this.extractErrorProperties(span), ...this.extractCustomMetadata(span) }; | ||
| } |
There was a problem hiding this comment.
Forwarding all metadata may leak PII, even in privacy mode
extractCustomMetadata forwards everything in span.metadata except userId and sessionId into PostHog event properties. If callers ever include emails, names, raw prompts, or other PII in metadata, it will be sent unredacted, regardless of enablePrivacyMode. Consider:
- documenting that
metadatamust be pre‑sanitized, and/or - introducing an allowlist/denylist or per‑exporter option to restrict which metadata keys are forwarded.
🤖 Prompt for AI Agents
In observability/posthog/src/tracing.ts around lines 308 to 344,
extractCustomMetadata currently forwards all span.metadata (except
userId/sessionId) which can leak PII; change it to honor privacy and only
forward safe metadata: add a configurable allowlist/denylist (or per-exporter
option) on the class that is consulted here, and if enablePrivacyMode is true
then either only forward the allowlisted keys or forward none; ensure keys not
allowed are omitted (not redacted) from the returned record, and update
callers/docs to require pre-sanitized metadata when no allowlist is configured.
There was a problem hiding this comment.
this is already solved higher up in the stack. See: https://mastra.ai/docs/observability/ai-tracing/processors/sensitive-data-filter
where this data can be cleaned for all exporters in the system.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Fetch the sensitive data filter documentation
curl -s "https://mastra.ai/docs/observability/ai-tracing/processors/sensitive-data-filter"
Length of output: 104826
@epinzur, you're absolutely right! I see that the SensitiveDataFilter processor operates at the correct level in the observability pipeline—it processes spans before they reach any exporters. This means metadata is already sanitized by the time it gets to the PostHog exporter's extractCustomMetadata method.
This is a well-designed architecture: centralizing sensitive data filtering in a processor that runs once for all exporters, rather than requiring each exporter to implement its own filtering logic. The documentation confirms that when using the default configuration, the SensitiveDataFilter is automatically included and will redact fields like passwords, tokens, API keys, etc. from all span data including metadata.
Thank you for the clarification!
✏️ Learnings added
Learnt from: epinzur
Repo: mastra-ai/mastra PR: 10393
File: observability/posthog/src/tracing.ts:308-344
Timestamp: 2025-11-24T18:27:56.140Z
Learning: In the Mastra observability pipeline, the SensitiveDataFilter processor runs before exporters and automatically redacts sensitive data from span metadata. Exporters (like PosthogExporter) receive pre-sanitized data, so they don't need to implement their own PII filtering logic. Documentation: https://mastra.ai/docs/observability/ai-tracing/processors/sensitive-data-filter
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
Co-authored-by: Richard LoRicco <rtlo9962@gmail.com> Co-authored-by: Daniel Lew <51924260+DanielSLew@users.noreply.github.com>
Adds a PostHog exporter to send Mastra LLM tracing data to PostHog's AI analytics platform. This addresses community request #7508.
The implementation maps Mastra's span-based observability to PostHog's event model (
$ai_generation,$ai_span), handling token usage, latency tracking, and parent-child relationships. Includes comprehensive test coverage, documentation, and configurable batching for both long-running and serverless environments.Co-Authored-By: Richard LoRicco rtlo9962@gmail.com
Summary by CodeRabbit
New Features
Documentation
Tests
Chores
✏️ Tip: You can customize this high-level summary in your review settings.