Skip to content

feat(telemetry): extensible log pipeline + rotating file exporter#22

Merged
lIang70 merged 2 commits into
mainfrom
feat/sdk-log-extensible
Apr 20, 2026
Merged

feat(telemetry): extensible log pipeline + rotating file exporter#22
lIang70 merged 2 commits into
mainfrom
feat/sdk-log-extensible

Conversation

@lIang70

@lIang70 lIang70 commented Apr 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Introduce an extensible logging pipeline in sdk/telemetry and add a rotating file exporter in sdkx/telemetry/logfile. Together they unblock the long-standing gap of "the server cannot persist its logs to disk", which on macOS effectively means flowcraft logs returns nothing useful (the VM only forwards stdout/stderr that the host never sees).

This PR ships the infrastructure; the wiring change in feat/flowcraft (adding LogConfig.File* fields and pointing the server at a virtio-fs-shared file) is a follow-up.

sdk/telemetry — new public API

API Role
WithLogProcessor(p sdklog.Processor) LogOption The canonical OTel pattern: register any number of processors.
ConsoleProcessors(min) []sdklog.Processor Ready-made stdout/stderr split sink (Exporter + BatchProcessor + SeverityFilter).
NewPlainTextExporter(w io.Writer) sdklog.Exporter Standard sdklog.Exporter. The previous hand-rolled batching Processor is gone — async batching, queue management and shutdown draining now come from OTel's BatchProcessor.
NewSeverityFilter(base, min, max) sdklog.Processor Severity gate honouring OTel's Enabled protocol so dropped records are short-circuited before formatting.
FormatPlainTextRecordLine(*sdklog.Record) []byte Exposed so downstream sinks (sdkx/logfile) can emit the same on-screen format.

sdk/telemetry — deprecations (v0.2.0 removal)

WithLogConsole, WithLogMinSeverity, WithLogExporter are marked Deprecated: with inline migration guides. Behaviour preserved on v0.1.x — calling InitLog() with no options still yields the default console sink.

Migration:

`go
// before
telemetry.InitLog(ctx)

// after
telemetry.InitLog(ctx,
telemetry.WithLogProcessor(telemetry.ConsoleProcessors(otellog.SeverityInfo)...),
)
`

sdkx/telemetry/logfile — new package

go exp, _ := logfile.NewExporter(logfile.Config{ Path: "/var/log/flowcraft/server.log", MaxSizeMB: 100, MaxBackups: 7, MaxAgeDays: 30, }) telemetry.InitLog(ctx, telemetry.WithLogProcessor(sdklog.NewBatchProcessor(exp)), )

Lives in sdkx because lumberjack would otherwise be an unwanted transitive dependency for sdk core consumers.

Compatibility & rollout

  • ✅ All existing call sites (sdk tests, plugin, examples) continue to compile and behave identically — deprecation is annotation-only.
  • ✅ No version drift on the OTel stack:
    • sdk: pulls go.opentelemetry.io/otel/sdk/log/logtest@v0.16.0 as a direct test dep, everything else stays on v1.40 / v0.16.
    • sdkx: pulls sdk/log v0.16.0 + sdk/log/logtest v0.16.0 + lumberjack.v2 v2.2.1, everything else stays on v1.40 / v0.16.
  • The auto-tag workflow will tag sdk first, then open a chore/bump-sdkx-sdk-vX.Y.Z PR. After that PR is merged, sdkx gets tagged with the new logfile package.

Test plan

  • cd sdk && go test -count=1 ./... — full suite green
  • cd sdkx && go test -count=1 ./... — full suite green (logfile package included)
  • cd plugin && go build ./... && go vet ./... — green
  • cd examples/voice-pipeline && go build ./... && go vet ./... — green
  • gofmt -l ./... clean
  • New TestInitLog_ForwardCompatible_ConsoleViaProcessor exercises the recommended migration path end-to-end.
  • New TestPlainTextExporter_Export_WrappedInBatchProcessor validates the Exporter+BatchProcessor composition that ConsoleProcessors actually produces.

Made with Cursor

lIang70 added 2 commits April 20, 2026 13:27
…on path

Add building blocks for composing OTel log sinks:
- WithLogProcessor: register any number of sdklog.Processor (the
  canonical OTel pattern; replaces the single-slot WithLogExporter)
- ConsoleProcessors: ready-made stdout/stderr split sink, internally
  wired as Exporter + BatchProcessor + SeverityFilter
- NewPlainTextExporter: sdklog.Exporter formatting records as plain text
  (the previous hand-rolled batching Processor is replaced; async
  batching, queue management and shutdown draining now come from OTel's
  BatchProcessor)
- NewSeverityFilter: severity gate Processor with min and max bounds,
  honouring OTel's Enabled protocol so dropped records are short-circuited
  before formatting
- FormatPlainTextRecordLine: exposed so downstream sinks (sdkx/logfile)
  can emit records in the same on-screen format

Deprecate WithLogConsole, WithLogMinSeverity and WithLogExporter with
inline migration guides. Behaviour is preserved on v0.1.x — calling
InitLog with no options still yields the default console sink — and
removal is scheduled for v0.2.0.

Pull go.opentelemetry.io/otel/sdk/log/logtest@v0.16.0 as a direct test
dependency so log records in tests can be constructed without hitting
the default attribute-length truncation. No version drift on the rest
of the OTel stack.

Made-with: Cursor
New sdklog.Exporter that writes plain-text log records to a local file
with size/age/backup-based rotation via natefinch/lumberjack.

Output format matches sdk/telemetry's console sink (uses
telemetry.FormatPlainTextRecordLine), so file logs read identically to
what a developer sees on stderr.

Typical wiring:

    exp, _ := logfile.NewExporter(logfile.Config{
        Path:       "/var/log/flowcraft/server.log",
        MaxSizeMB:  100,
        MaxBackups: 7,
        MaxAgeDays: 30,
    })
    telemetry.InitLog(ctx,
        telemetry.WithLogProcessor(sdklog.NewBatchProcessor(exp)),
    )

Lives in sdkx because lumberjack would be an unwanted transitive
dependency for sdk core consumers.

Made-with: Cursor
@lIang70
lIang70 merged commit 2b5580e into main Apr 20, 2026
8 checks passed
@lIang70
lIang70 deleted the feat/sdk-log-extensible branch April 20, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant