Skip to content

Conversation

@ericallam
Copy link
Member

When using custom OTLP exporters via telemetry.exporters and
This occurred when tasks were triggered without a parent trace
context (e.g., via API or dashboard). In this scenario: - Spans were
correctly rewritten to use the generated externalTraceId - Logs kept
their original internal trace ID due to a bug in the early return logic

Root Cause

In ExternalLogRecordExporterWrapper.transformLogRecord(), the early
return condition incorrectly included !this.externalTraceContext:

if (!logRecord.spanContext || !this.externalTraceId || 
!this.externalTraceContext) {   return logRecord;  // Bug: Returns early
when externalTraceContext is undefined }

// This fallback logic was never reached:
const externalTraceId = this.externalTraceContext
  ? this.externalTraceContext.traceId
  : this.externalTraceId;

Fix

  1. Reordered logic in transformLogRecord(): Move the
  2. externalTraceId calculation before the early return, and check the
  3. culated value instead of this.externalTraceContext:
const externalTraceId = this.externalTraceContext
  ? this.externalTraceContext.traceId
  : this.externalTraceId;

if (!logRecord.spanContext || !externalTraceId) {
  return logRecord;
}
  1. Clarified _isExternallySampled logic: Updated both
  2. ExternalSpanExporterWrapper and ExternalLogRecordExporterWrapper
  3. explicitly handle the case where there's no external trace context
  4. a generated externalTraceId exists:
this._isExternallySampled = externalTraceContext
  ? isTraceFlagSampled(externalTraceContext.traceFlags)
  : !!externalTraceId;

Impact

Logs and spans from the same task run will now have matching trace IDs
when exported to external observability tools, enabling proper trace
correlation regardless of whether the task was triggered with or without a parent trace context.
telemetry.logExporters in trigger.config.ts, logs and spans were
exported with different trace IDs, breaking trace correlation in
external observability tools like Datadog.

When using custom OTLP exporters via `telemetry.exporters` and 

This occurred when tasks were triggered **without** a parent trace 
context (e.g., via API or dashboard). In this scenario: - Spans were 
correctly rewritten to use the generated `externalTraceId` - Logs kept 
their original internal trace ID due to a bug in the early return logic

### Root Cause

In `ExternalLogRecordExporterWrapper.transformLogRecord()`, the early 
return condition incorrectly included `!this.externalTraceContext`:

```typescript
if (!logRecord.spanContext || !this.externalTraceId || 
!this.externalTraceContext) {   return logRecord;  // Bug: Returns early
when externalTraceContext is undefined }

// This fallback logic was never reached:
const externalTraceId = this.externalTraceContext
  ? this.externalTraceContext.traceId
  : this.externalTraceId;
```

### Fix

1. **Reordered logic in `transformLogRecord()`**: Move the 
1. `externalTraceId` calculation before the early return, and check the 
1. culated value instead of `this.externalTraceContext`:

```typescript
const externalTraceId = this.externalTraceContext
  ? this.externalTraceContext.traceId
  : this.externalTraceId;

if (!logRecord.spanContext || !externalTraceId) {
  return logRecord;
}
```

2. **Clarified `_isExternallySampled` logic**: Updated both 
2. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` 
2. explicitly handle the case where there's no external trace context 
2. a generated `externalTraceId` exists:

```typescript
this._isExternallySampled = externalTraceContext
  ? isTraceFlagSampled(externalTraceContext.traceFlags)
  : !!externalTraceId;
```

### Impact

Logs and spans from the same task run will now have matching trace IDs 
when exported to external observability tools, enabling proper trace correlation regardless of whether the task was triggered with or without a parent trace context. 
`telemetry.logExporters` in `trigger.config.ts`, logs and spans were 
exported with **different trace IDs**, breaking trace correlation in
external observability tools like Datadog.
@changeset-bot
Copy link

changeset-bot bot commented Dec 2, 2025

🦋 Changeset detected

Latest commit: ee457aa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 26 packages
Name Type
trigger.dev Patch
d3-chat Patch
references-d3-openai-agents Patch
references-nextjs-realtime Patch
references-realtime-hooks-test Patch
references-realtime-streams Patch
references-telemetry Patch
@trigger.dev/build Patch
@trigger.dev/core Patch
@trigger.dev/python Patch
@trigger.dev/react-hooks Patch
@trigger.dev/redis-worker Patch
@trigger.dev/rsc Patch
@trigger.dev/schema-to-json Patch
@trigger.dev/sdk Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch
@internal/cache Patch
@internal/clickhouse Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/schedule-engine Patch
@internal/testcontainers Patch
@internal/tracing Patch
@internal/zod-worker Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Dec 2, 2025

Walkthrough

This pull request adds a changeset entry for a patch release documenting a fix to OpenTelemetry integration in Trigger.dev. The corresponding code changes in packages/core/src/v3/otel/tracingSDK.ts modify two exporter wrapper classes to properly handle cases where externalTraceContext is undefined. The key change introduces a fallback mechanism: when externalTraceContext is missing, the code now derives the sampling flag from externalTraceId directly instead of attempting to access traceFlags. Additionally, the log record transformer now validates that externalTraceId exists before processing, preventing logs without valid trace context from being transformed.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • Changes to two related but distinct exporter wrapper classes require separate reasoning for each class's behavior
  • New fallback logic for handling undefined externalTraceContext needs verification to ensure correct trace ID propagation
  • The guard condition in transformLogRecord introduces a behavioral change that affects log filtering logic
  • The interaction between the fallback mechanism and existing trace context handling should be carefully validated

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main fix: exported logs and spans now have matching trace IDs, which directly corresponds to the primary change in the changeset.
Description check ✅ Passed The description provides comprehensive context on the bug, root cause, the fix implementation, and impact. However, it's missing the checklist items and some template sections like Testing and Screenshots that are specified in the repository template.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ea-branch-105

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
packages/core/src/v3/otel/tracingSDK.ts (3)

317-320: Sampling fallback behavior looks correct when no externalTraceContext

Using externalTraceContext ? isTraceFlagSampled(...) : !!externalTraceId makes the wrapper treat traces as sampled whenever there’s a generated externalTraceId but no upstream context, which matches the intent for internally-started tasks. The logic is sound; the !!externalTraceId bit is slightly redundant given it’s typed as string, but that’s purely cosmetic.


401-404: Consistent sampling semantics between span and log exporters

Mirroring the _isExternallySampled computation from ExternalSpanExporterWrapper into ExternalLogRecordExporterWrapper keeps span/log behavior aligned when there’s no externalTraceContext but a generated externalTraceId. This is important for ensuring both spans and logs are either exported or dropped together. Same minor note as above that !!externalTraceId is redundant but harmless.


429-433: Guarding on spanContext prevents unnecessary proxying

Early‑returning when !logRecord.spanContext (and, defensively, !externalTraceId) makes sense: logs without a span context are passed through untouched, and the proxy is only used when there’s something to rewrite. Given externalTraceId is always a non‑empty string in current usage, the second part of the condition is mostly defensive, but it doesn’t hurt readability.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2f1a72b and ee457aa.

📒 Files selected for processing (2)
  • .changeset/brown-pots-beg.md (1 hunks)
  • packages/core/src/v3/otel/tracingSDK.ts (3 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

Files:

  • packages/core/src/v3/otel/tracingSDK.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use zod for validation in packages/core and apps/webapp

Files:

  • packages/core/src/v3/otel/tracingSDK.ts
**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

Files:

  • packages/core/src/v3/otel/tracingSDK.ts
**/*.{js,ts,jsx,tsx,json,md,css,scss}

📄 CodeRabbit inference engine (AGENTS.md)

Format code using Prettier

Files:

  • packages/core/src/v3/otel/tracingSDK.ts
🧠 Learnings (3)
📓 Common learnings
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger.config.ts : Configure OpenTelemetry instrumentations and exporters in trigger.config.ts for enhanced logging
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger.config.ts : Configure OpenTelemetry instrumentations and exporters in trigger.config.ts for enhanced logging

Applied to files:

  • packages/core/src/v3/otel/tracingSDK.ts
  • .changeset/brown-pots-beg.md
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Export tasks with unique IDs within the project to enable proper task discovery and execution

Applied to files:

  • .changeset/brown-pots-beg.md
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (23)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
  • GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (2)
.changeset/brown-pots-beg.md (1)

1-5: Changeset entry looks good.

The format is correct and the summary accurately describes the bug fix. The message is clear enough for users consuming release notes, and the patch bump type is appropriate for this bug fix.

packages/core/src/v3/otel/tracingSDK.ts (1)

425-427: Deriving externalTraceId before guard fixes the mismatch bug

Computing externalTraceId from externalTraceContext.traceId with a fallback to the generated externalTraceId before any early return is exactly what’s needed so logs from tasks without a parent context still get rewritten to the same external trace ID as spans. This directly addresses the original bug around mismatched trace IDs.

@ericallam ericallam merged commit 9f27422 into main Dec 2, 2025
31 checks passed
@ericallam ericallam deleted the ea-branch-105 branch December 2, 2025 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants