Skip to content

chore: add traces to event queue #666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 18, 2025
+123 −8
Merged

chore: add traces to event queue #666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e5e2098
chore: add traces to event queue
adityachoudhari26 Sep 18, 2025
2a09b70
rabbit changes
adityachoudhari26 Sep 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/event-queue/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ EXPOSE 3124

WORKDIR apps/event-queue/dist/

CMD ["node", "index.js"]
CMD ["node", "-r", "./instrumentation-node.js", "index.js"]
9 changes: 9 additions & 0 deletions apps/event-queue/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@
"@ctrlplane/validators": "workspace:*",
"@octokit/auth-app": "catalog:",
"@octokit/rest": "catalog:",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/auto-instrumentations-node": "^0.52.1",
"@opentelemetry/exporter-logs-otlp-http": "^0.54.2",
"@opentelemetry/exporter-trace-otlp-http": "^0.54.2",
"@opentelemetry/resources": "^1.27.0",
"@opentelemetry/sdk-logs": "^0.54.2",
"@opentelemetry/sdk-node": "^0.54.2",
"@opentelemetry/sdk-trace-base": "^1.27.0",
"@opentelemetry/semantic-conventions": "^1.27.0",
"@t3-oss/env-core": "catalog:",
"dotenv": "^16.4.5",
"kafkajs": "^2.2.4",
Expand Down
22 changes: 16 additions & 6 deletions apps/event-queue/src/events/resources.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
import type * as schema from "@ctrlplane/db/schema";
import type { Event } from "@ctrlplane/events";
import { trace } from "@opentelemetry/api";

import { makeWithSpan } from "@ctrlplane/logger";

import type { Handler } from ".";
import { OperationPipeline } from "../workspace/pipeline.js";
Expand All @@ -16,12 +19,19 @@ const getResourceWithDates = (resource: schema.Resource) => {
return { ...resource, createdAt, updatedAt, lockedAt, deletedAt };
};

export const newResource: Handler<Event.ResourceCreated> = async (event) => {
const ws = await WorkspaceManager.getOrLoad(event.workspaceId);
if (ws == null) return;
const resource = getResourceWithDates(event.payload);
await OperationPipeline.update(ws).resource(resource).dispatch();
};
const newResourceTracer = trace.getTracer("new-resource");
const withSpan = makeWithSpan(newResourceTracer);

export const newResource: Handler<Event.ResourceCreated> = withSpan(
"new-resource",
async (span, event) => {
span.setAttribute("resource.id", event.payload.id);
const ws = await WorkspaceManager.getOrLoad(event.workspaceId);
if (ws == null) return;
const resource = getResourceWithDates(event.payload);
await OperationPipeline.update(ws).resource(resource).dispatch();
},
);

export const updatedResource: Handler<Event.ResourceUpdated> = async (
event,
Expand Down
65 changes: 65 additions & 0 deletions apps/event-queue/src/instrumentation-node.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";
import { NodeSDK } from "@opentelemetry/sdk-node";
import {
AlwaysOnSampler,
BatchSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";

const sdk = new NodeSDK({
resource: new Resource({
[ATTR_SERVICE_NAME]: "ctrlplane/event-queue",
}),
spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter()) as any],
logRecordProcessors: [new BatchLogRecordProcessor(new OTLPLogExporter())],
instrumentations: [
getNodeAutoInstrumentations({
"@opentelemetry/instrumentation-fs": {
enabled: false,
},
"@opentelemetry/instrumentation-net": {
enabled: false,
},
"@opentelemetry/instrumentation-dns": {
enabled: false,
},
"@opentelemetry/instrumentation-http": {
enabled: true,
},
"@opentelemetry/instrumentation-pg": {
enabled: true,
enhancedDatabaseReporting: true,
addSqlCommenterCommentToQueries: true,
},
"@opentelemetry/instrumentation-ioredis": {
enabled: true,
},
"@opentelemetry/instrumentation-winston": {
enabled: true,
logHook: (span, record) => {
record["resource.service.name"] = "ctrlplane/event-queue";
},
},
}),
],
sampler: new AlwaysOnSampler(),
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Consider configurable sampling strategy for production

Using AlwaysOnSampler in production can generate excessive trace data and incur high costs. Consider making the sampling strategy configurable.

Apply this diff to make sampling configurable:

+import { TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
+
+const samplingRatio = parseFloat(process.env.OTEL_SAMPLING_RATIO || "0.1");
+const sampler = samplingRatio >= 1 
+  ? new AlwaysOnSampler()
+  : new TraceIdRatioBasedSampler(samplingRatio);
+
 const sdk = new NodeSDK({
   // ... other config
-  sampler: new AlwaysOnSampler(),
+  sampler,
 });

Would you like me to create an issue to track adding sampling configuration?

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
sampler: new AlwaysOnSampler(),
import { TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
const samplingRatio = parseFloat(process.env.OTEL_SAMPLING_RATIO || "0.1");
const sampler = samplingRatio >= 1
? new AlwaysOnSampler()
: new TraceIdRatioBasedSampler(samplingRatio);
const sdk = new NodeSDK({
// ... other config
sampler,
});
🤖 Prompt for AI Agents 
In apps/event-queue/src/instrumentation-node.ts around line 49, the code
unconditionally uses sampler: new AlwaysOnSampler(), which can produce excessive
traces in production; change this to select the sampler from configuration (env
var or shared config) instead. Import the needed samplers (AlwaysOnSampler,
TraceIdRatioBasedSampler, ParentBasedSampler) and read something like PROCESS
env TRACE_SAMPLER (e.g., "always"|"ratio"|"parent") and TRACE_SAMPLE_RATE (a
float 0.0-1.0) or equivalent config; build the sampler with a small switch/if
(default to TraceIdRatioBasedSampler with a sensible default in production and
AlwaysOnSampler in local/dev) and pass that sampler into the tracer/provider.
Validate and coerce the sample-rate value, and document the two env vars near
the file.

Copy link
Contributor Author

@adityachoudhari26 adityachoudhari26 Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not worried about cost atm

Copy link
Contributor

@coderabbitai coderabbitai bot Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adityachoudhari26, understood! Keeping AlwaysOnSampler for now makes sense when you want full observability and cost isn't a concern. You can always optimize sampling later when needed.

✏️ Learnings added 
Learnt from: adityachoudhari26
PR: ctrlplanedev/ctrlplane#666
File: apps/event-queue/src/instrumentation-node.ts:49-49
Timestamp: 2025-09-18T17:40:04.183Z
Learning: The team prefers using AlwaysOnSampler for OpenTelemetry tracing in the event-queue service as they are not currently concerned about tracing costs and want full observability.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

});

try {
sdk.start();
console.log("Tracing initialized");
} catch (error) {
console.error("Error initializing tracing", error);
}

process.on("SIGTERM", () => {
sdk
.shutdown()
.then(() => console.log("Tracing shutdown"))
.catch((error) => console.error("Error shutting down tracing", error))
.finally(() => process.exit(0));
});
4 changes: 4 additions & 0 deletions apps/event-queue/src/instrumentation.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
export async function register() {
if (process.env.NODE_ENV === "production")
await import("./instrumentation-node.js");
}
29 changes: 28 additions & 1 deletion pnpm-lock.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading