Skip to content

Add telemetry-to-caas kind for CaaS (Collector as a Service) support - #436

Open
vkozyura wants to merge 79 commits into
mainfrom
feature/caas-support
Open

Add telemetry-to-caas kind for CaaS (Collector as a Service) support#436
vkozyura wants to merge 79 commits into
mainfrom
feature/caas-support

Conversation

@vkozyura

Copy link
Copy Markdown
Contributor
  • Add telemetry-to-caas kind definition in package.json
  • Add getCredsForCaaS() to extract credentials from caas-service binding
  • Add augmentCaaSCreds() to configure OTLP endpoint URL
  • Handle CaaS in tracing and metrics exporters

Note: CaaS requires mTLS authentication with SAP-signed certificates. The certificate must be obtained separately via BTP Certificate Service.

- Add telemetry-to-caas kind definition in package.json
- Add getCredsForCaaS() to extract credentials from caas-service binding
- Add augmentCaaSCreds() to configure OTLP endpoint URL
- Handle CaaS in tracing and metrics exporters

Note: CaaS requires mTLS authentication with SAP-signed certificates.
The certificate must be obtained separately via BTP Certificate Service.

@hyperspace-pr-bot hyperspace-pr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is generally well-structured, but has one logic bug: when only a gRPC OTLP endpoint is present in the CaaS binding, credentials.url is set to undefined (the falsy http value), silently breaking the exporter. Please address the flagged issues before merging.

PR Bot Information

Version: 1.26.5

  • Event Trigger: pull_request.opened
  • LLM: anthropic--claude-4.6-sonnet
  • Correlation ID: be7942dc-b11a-457c-a83d-2aafac5eb2c3
  • File Content Strategy: Full file content

Comment thread lib/utils.js Outdated
Comment thread lib/utils.js
Comment thread lib/tracing/index.js Outdated
Comment thread lib/utils.js Fixed
Comment thread lib/utils.js Fixed
Comment thread lib/utils.js Fixed
This file should not be committed to the feature branch.
@vkozyura
vkozyura force-pushed the feature/caas-support branch 6 times, most recently from 4f2f45d to 1906cd7 Compare August 31, 2026 13:21
Comment thread lib/logging/index.js Outdated
Comment thread lib/exporter/LazyExporter.js Outdated
const cds = require('@sap/cds')
const LOG = cds.log('telemetry')

const MAX_BUFFER_SIZE = 1000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how much is this?

@vkozyura vkozyura Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let ai calculate it :)

MAX_BUFFER_SIZE = 1000 buffers batches, not items:

Signal Batch Size Item Size 1000 Batches
Traces 512 spans ~2 KB ~1 GB
Metrics varies ~1 KB ~100 MB
Logs 512 records ~1 KB ~500 MB

This is way too much. The ZTI window is typically seconds. 10-50 batches is plenty

solution: changed to 10 (16MB in worst case)

Comment thread lib/utils.js
Comment on lines +182 to +193
const ztiAgentFactory = createZTIAgentFactory()
if (ztiAgentFactory) {
credentials.httpAgentOptions = ztiAgentFactory
credentials.useZTI = true
return
}

const staticAgentFactory = createStaticAgentFactory()
if (staticAgentFactory) {
credentials.httpAgentOptions = staticAgentFactory
return
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the agent factory gets called every time a new connection is opened, correct? in that case, i think we don't need this distinction here. that agent factory should just always use cds.env.requires.telemetry.x509, which gets updated by the svid watcher in case of ztis.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The agent factory is called once, not per-Connection http-exporter-transport.js:23-34 and cached for the process lifetime.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. still the agent factory should be something like:

() => {
  const { cert, key } = cds.env.requires.telemetry.x509
  const agent = new https.Agent({ cert, key, keepAlive: true })
  cds.on('svid', ({ cert, key }) => Object.assign(agent.options, { cert, key }))
  return agent
}

the watcher will not remain in telemetry. other services will need to support as well. the more we decouple now the better. question is how to handle missing cert during startup...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of a lazy exporter, could we create the standard exporter and temporarily swap their export function with one that buffers, maybe even using the file system to reduce memory load?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still need buffer logic somewhere if first span completes and triggers export() and SVID files don't exist yet - swapping export function is too hacky for me :( and using file system looks like overkill for a seconds-long window. i reduced buffersize to 10 - enough for few seconds and only 16MB in worst scenario.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imho, swapping the export function is less hacky than the lazy exporter. or at least less risky. as demonstrated by the temporality bug. export is a single, clearly defined function.

@sjvans sjvans left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vkozyura — the ZTI rework cleared all the earlier bot/CodeQL findings (the shared-env clobbering, base64/PEM handling, the unescaped RegExp, the URL-substring checks). A few things left before this can go into 2.1.0.

Blocking

1. CaaS metrics silently downgrade DELTA → CUMULATIVE on the ZTI path
createCaaSExporter returns a LazyExporter (lib/exporter/LazyExporter.js), which only exposes export/shutdown/forceFlush. PeriodicExportingMetricReader looks for exporter.selectAggregationTemporality, doesn't find it on the wrapper, and falls back to the default CUMULATIVE selector — so the temporalityPreference ??= DELTA we set on the inner OTLP exporter is ignored. Every other kind in this repo emits DELTA. static-x509 returns the real exporter directly so it's fine there; this only bites the (recommended) ZTI path, and since no test covers CaaS metrics it went unnoticed. I'd rather fix the root cause than proxy one more method — see design note B.

2. try/catch around the logs exporter still masks setup errors (lib/logging/index.js:38-44)
Same point as my earlier comment — it swallows a missing/broken logs-exporter module into null (logging silently disabled) for all kinds, inconsistent with tracing/metrics which still fail loudly via _require. Please drop it.

Design — two components reinvent platform machinery

Root fact worth stating up front: the OTLP http transport calls the httpAgentOptions factory exactly once and caches the returned Agent for the process lifetime (@opentelemetry/otlp-exporter-basehttp-exporter-transport.js _loadUtils). Two consequences:

A. Cert handling — one source of truth + one rotation mechanism.
Today there are two parallel cert paths: createStaticAgentFactory closure-captures the decoded x509 once (so static certs never rotate on renewal), and createZTIAgentFactoryzti.js reimplements file reads plus _cached/_paths/_rotatingAgent module globals plus an fs.watch+debounce. Let's collapse this: an SVID watcher keeps cds.env.requires.telemetry.x509 current, and a single rotating agent sources cert/key from there and destroy()s its sockets on change. Note rotation still needs the self-refreshing agent + destroy() — because OTel caches the Agent, merely reading x509 inside the factory wouldn't rotate. The win is the single source of truth, unifying the static and ZTI paths, and static certs becoming rotatable for free.

B. LazyExporter — override, don't replace.
Because the agent factory is only invoked on the first send(), the OTLP exporter can be constructed before the SVID files exist — so the "defer construction until certs are ready" premise mostly doesn't hold. Prefer constructing the real exporter, and if we truly can't lose the first-export window, override just its export (FS-backed if we want crash durability) so the full exporter interface — including the temporality selector from blocker #1 — stays intact. A partial hand-rolled stand-in will keep leaking methods; this is exactly how #1 happened. The 1000-item in-memory drop-oldest ring also duplicates what BatchSpanProcessor/PeriodicExportingMetricReader already do and loses everything on restart.

Minor

  • package.json still declares mtls_service_pattern — nothing reads it anymore (leftover from the removed getCredsForCaaSMtls). Remove.
  • zti.js requires https at module top level — per OTel's own guidance for agent factories, load it lazily (as createStaticAgentFactory does) so it can't preempt @opentelemetry/instrumentation-http patching, now that #475 enabled HTTP instrumentation.
  • CHANGELOG: use ## Version 2.1.0 - tbd (our convention for the unreleased section).
  • The BatchLogRecordProcessor({ exporter }) / SimpleLogRecordProcessor({ exporter }) change is correct for @opentelemetry/sdk-logs@0.221 but touches all kinds, not just CaaS — worth splitting into its own PR.

Before it can ride 2.1.0

  • Retarget to develop — main-based PRs re-trigger the release ancestry problem.
  • Port test/caas.test.js off jestdevelop is on vitest (#474); jest.fn/jest.isolateModules won't resolve there.
  • Add a CaaS metrics test (would have caught blocker #1) and assert the signal-suffixed exporter URL (baseUrl + '/v1/traces' etc.) against a real exporter — the in-memory exporter ignores config.url, so that path is currently unverified.

@sjvans

sjvans commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

the SVID watcher should emit an event on cds (e.g., cds.emit('svid', { ... })) that the self-refreshing agent and the exporter can react on.

FS-backed if we want crash durability

crash durability is not the point, minimized memory usage is

@vkozyura

vkozyura commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

1. CaaS metrics silently downgrade DELTA → CUMULATIVE on the ZTI path `

I went with the quick fix (Option A) to unblock the PR

@vkozyura

vkozyura commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author
  • package.json still declares mtls_service_pattern — nothing reads it anymore (leftover from the removed getCredsForCaaSMtls). Remove.

removed

@vkozyura

vkozyura commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

"zti.js only extends https.Agent to hold certificates — it doesn't make any HTTP requests itself

@vkozyura

vkozyura commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author
  • The BatchLogRecordProcessor({ exporter }) / SimpleLogRecordProcessor({ exporter }) change is correct for @opentelemetry/sdk-logs@0.221 but touches all kinds, not just CaaS — worth splitting into its own PR.

Added @opentelemetry/sdk-logs: >=0.221 as optional peerDependency — npm will warn if users have an older incompatible version, but won't require it if logging isn't used

@sjvans

sjvans commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

1. CaaS metrics silently downgrade DELTA → CUMULATIVE on the ZTI path `

I went with the quick fix (Option A) to unblock the PR

which option a?

@vkozyura

vkozyura commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

the SVID watcher should emit an event on cds (e.g., cds.emit('svid', { ... })) that the self-refreshing agent and the exporter can react on.

FS-backed if we want crash durability

crash durability is not the point, minimized memory usage is

for now #436 (comment). we still can discuss emitting an event as future work. bli`?

@sjvans

sjvans commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

the SVID watcher should emit an event on cds (e.g., cds.emit('svid', { ... })) that the self-refreshing agent and the exporter can react on.

FS-backed if we want crash durability

crash durability is not the point, minimized memory usage is

for now #436 (comment). we still can discuss emitting an event as future work. bli`?

there's no rush, we can spend the time now.

@vkozyura

vkozyura commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Before it can ride 2.1.0

postpone until the content is clarified

@sjvans

sjvans commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

"zti.js only extends https.Agent to hold certificates — it doesn't make any HTTP requests itself

it may or may not make a difference, still we should follow the instruction

@sjvans

sjvans commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
  • The BatchLogRecordProcessor({ exporter }) / SimpleLogRecordProcessor({ exporter }) change is correct for @opentelemetry/sdk-logs@0.221 but touches all kinds, not just CaaS — worth splitting into its own PR.

Added @opentelemetry/sdk-logs: >=0.221 as optional peerDependency — npm will warn if users have an older incompatible version, but won't require it if logging isn't used

already solved on develop:

? new BatchLogRecordProcessor(logProcessorArg(exporter))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants