feat(otel): add OpenTelemetry instrumentation for core memory operations - #2207
feat(otel): add OpenTelemetry instrumentation for core memory operations#2207henrikrexed wants to merge 6 commits into
Conversation
Instrument the core add / search / update / delete memory operations with OpenTelemetry spans, metrics, and logs, following the memory-semconv conventions. The L1/L2/L3 memory tiers are mapped onto the `memory.tier` attribute. - New `memos.telemetry` module: lightweight helpers that emit spans/metrics and degrade to no-ops when OpenTelemetry is not installed or configured, so there is zero runtime cost for users who do not opt in. - The application remains responsible for configuring the OTel SDK/exporters; this change only instruments the library (correct library/app separation). - Metrics cover data stored/indexed/utilized, per-tier utilization, query result/hit counts, and search latency. The search query is captured as a span attribute with cardinality/PII in mind. - Adds optional `opentelemetry-*` dependencies under an extra in pyproject. - Adds tests/test_telemetry.py (all green without an OTel backend present). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…port, product-API metrics Extends the core memory-semconv instrumentation so the telemetry is actually exported on every entry-point MemOS runs behind, and validated end-to-end. - telemetry.py: env bootstrap (configure_from_env), W3C TraceContext propagation, metrics (memory.operations, memory.operation.duration, memory.result.count) and span-correlated logs, alongside the add/search/update/delete spans. - api/server_api.py: bootstrap the OTel SDK from env + FastAPIInstrumentor so the REST entry-point emits server spans and propagates traceparent. - api/mcp_serve.py: bootstrap OTel in MOSMCPServer.__init__ so the MCP entry-point exports the memory.* telemetry too (otherwise it runs against the no-op provider and emits nothing over MCP); best-effort server-side MCP context extraction. - multi_mem_cube/single_cube.py: instrument the product-API add/search path. - pyproject.toml: opentelemetry-instrumentation-fastapi in the [otel] extra. - tests: telemetry + MCP-bootstrap regression guards (13 passing). All signals validated against a live OTel collector → Tempo + Dynatrace: memory.tier semconv spans + memory_* metrics on service.name=memos, connected end-to-end under an agent trace. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
There was a problem hiding this comment.
Pull request overview
Adds an OpenTelemetry instrumentation module and wires it into MemOS core memory operations, plus an optional dependency extra and basic unit tests intended to validate span/metric emission.
Changes:
- Introduces
src/memos/telemetry.pywith span/metric/log helpers and aconfigure()routine. - Wraps
MOSCore.search()in amemory_span(...)and decoratesadd/update/deletewith@instrument_op(...). - Adds an
oteloptional-dependency extra and a newtests/test_telemetry.pysuite.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
tests/test_telemetry.py |
New unit tests validating span attributes and in-memory metrics for the telemetry helpers. |
src/memos/telemetry.py |
New telemetry helper module defining semconv constants plus span/metric/log utilities and an SDK configure function. |
src/memos/mem_os/core.py |
Instruments core memory operations (search/add/update/delete) via the new telemetry helpers. |
pyproject.toml |
Adds an [otel] optional-dependency extra for OpenTelemetry packages. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from memos.telemetry import ( | ||
| MEMORY_CUBE_ID, | ||
| MEMORY_ITEM_COUNT, | ||
| MEMORY_RESULT_COUNT, | ||
| MEMORY_SESSION_ID, |
| def configure( | ||
| service_name: str = "memos", | ||
| service_version: str = "0.0.0", | ||
| otlp_endpoint: str = "http://localhost:4317", | ||
| export_interval_ms: int = 5000, | ||
| ) -> None: | ||
| """ | ||
| Configure OTel providers and install them globally. | ||
|
|
||
| Call once at application startup (e.g., from MOSConfig or CLI entry-point). | ||
| Safe to call multiple times; subsequent calls are no-ops. |
| from opentelemetry import metrics, trace | ||
| from opentelemetry._logs import get_logger_provider, set_logger_provider | ||
| from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter |
| import importlib.util | ||
| import pathlib | ||
| import sys | ||
| import types | ||
| import pytest | ||
|
|
||
| from opentelemetry.sdk.metrics import MeterProvider | ||
| from opentelemetry.sdk.metrics.export import InMemoryMetricReader | ||
| from opentelemetry.sdk.trace import TracerProvider | ||
| from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter | ||
| from opentelemetry.sdk.trace.export import SimpleSpanProcessor | ||
| from opentelemetry import trace, metrics |
🤖 Open Code ReviewTarget: PR #2207 🔍 OpenCodeReview found 13 issue(s) in this PR. 1.
|
✅ Automated Test Results: PASSEDAll tests passed (6/6 executed). memos_python_core/changed-repo-python: 6/6. Duration: 1s [advisory, non-gating] AI-generated tests on branch test/auto-gen-dac9d2f6ee0314fb-20260804185426: 181/184 passed, 3 failed — these do NOT affect the PR verdict; review the branch manually. Branch: |
…mport, lint, test guards
Resolves the Copilot review comments on the upstream PR:
- telemetry.py: guard the opentelemetry imports so `import memos.telemetry`
(and thus `import memos.mem_os.core`) works on a default install WITHOUT the
[otel] extra — the instrumentation degrades to a no-op (`_OTEL_AVAILABLE`,
`_NoopSpan`, guarded configure/configure_from_env/get_tracer/memory_span).
Drops the unused `get_logger_provider` import; moves `Callable` into a
TYPE_CHECKING block; simplifies `_resolve_service_version` (no blind except).
- core.py: drop unused telemetry constants (MEMORY_CUBE_ID, MEMORY_ITEM_COUNT,
TIER_ACTIVATION, TIER_PARAMETRIC) — F401.
- tests: guard both suites with `pytest.importorskip("opentelemetry")` and drop
unused sys/types imports.
The env bootstrap the reviewer asked for (configure_from_env + FastAPI/MCP
entry-point wiring) is included in this branch (server_api.py / mcp_serve.py).
Verified: 13 tests pass; `import memos.telemetry` succeeds with opentelemetry
absent (no-op); ruff clean on telemetry.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
…ependencies) CI's `scripts/check_dependencies.py` installs main deps only (no [otel] extra) and fails on any top-level import of a non-main module. single_cube.py imported `from opentelemetry.trace import get_current_span` at module level, so the check (and thus the "Python tests" workflow) failed. - telemetry.py: add a no-op-safe `get_current_span()` (returns the active span, or _NoopSpan when OTel is absent). - single_cube.py: import `get_current_span` from `memos.telemetry` (our own package, excluded by the check) instead of from `opentelemetry` directly. Verified: no `src/memos` file has a tree.body top-level `opentelemetry` import (same AST walk check_dependencies uses); 13 tests pass; telemetry.py ruff clean; get_current_span works with opentelemetry present (real span) and absent (no-op). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
…check` The `python-tests` workflow runs `ruff format --check` (ruff 0.11.8). Reflowed the OTel additions to match: collapse now-short calls onto one line, blank lines around the nested `_NoopSpan`/decorator defs, and quote/spacing normalization in the telemetry tests. No behavior change. Verified with ruff 0.11.8 + repo config: `ruff check` and `ruff format --check` both clean on all changed files; `check_dependencies.py` finds no top-level optional imports; 13 telemetry tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
✅ Automated Test Results: PASSEDAll tests passed (13/13 executed). memos_python_core/changed-repo-python: 13/13. Duration: 1s [advisory, non-gating] AI-generated tests on branch test/auto-gen-6bfc3a89d17f9cb9-20260804194123: 135/139 passed, 4 failed — these do NOT affect the PR verdict; review the branch manually. Branch: |
…[all]
Root cause of the failing `Python tests` workflow: the PR added the `[otel]`
optional-dependency extra to pyproject.toml but never regenerated poetry.lock,
so `poetry check --lock` failed ("pyproject.toml changed significantly since
poetry.lock was last generated") and CI's strict `poetry install` (poetry 2.1.3)
errored before any test ran.
- poetry.lock: regenerated — now includes opentelemetry-* and transitive deps
(poetry check --lock passes).
- pyproject.toml: add the opentelemetry packages to the `[all]` extra (they were
the only sub-extra missing from `all`). CI's pytest step runs
`poetry install --extras all`, so this makes opentelemetry present during the
suite → the telemetry tests actually run (instead of pytest.importorskip
skipping them) and cover telemetry.py, keeping total coverage above
--cov-fail-under=28.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
✅ Automated Test Results: PASSEDAll tests passed (13/13 executed). memos_python_core/changed-repo-python: 13/13. Duration: 1s [advisory, non-gating] AI-generated tests on branch test/auto-gen-468bce880403ade4-20260804201833: 61/89 passed, 28 failed — these do NOT affect the PR verdict; review the branch manually. Branch: |
Summary
Adds first-class OpenTelemetry instrumentation to MemOS conforming to memory-semconv v0.1.0, so MemOS emits uniform, benchmark-grade telemetry — spans, metrics, and logs — on the core memory operations. It is opt-in (a no-op unless OTEL_EXPORTER_OTLP_ENDPOINT is set) and every optional dependency is imported defensively, so default installs and local runs are unaffected.
What's instrumented
Traces — spans on memory.add / memory.search / memory.update / memory.delete:
Metrics — memory.operations (counter), memory.operation.duration (ms histogram), memory.result.count (histogram), labeled by memory.operation + memory.tier.
Logs — structured, span-correlated records on each operation.
Propagation — W3C TraceContext installed globally; both the REST (FastAPI) and MCP entry-points bootstrap the SDK from env and extract inbound traceparent, so memory.* spans nest under a calling agent's trace end-to-end.
Changes
Validation
Validated end-to-end against a live OTel Collector → Tempo and Dynatrace: one connected trace CrewAI agent → LLM (Ollama) → MCP → memos (single trace_id), memory.tier=textual semconv spans, and memory_* metrics on service.name=memos. Both backends agree. Unit tests: pytest tests/test_telemetry.py tests/test_mcp_serve_telemetry.py → 13 passed.
Notes