test(observability): add ingestor QA integration suite

[ljluestc]

Summary

Describe the problem and fix in 2–5 bullets:
If this PR fixes a plugin beta-release blocker, title it fix(<plugin-id>): beta blocker - <summary> and link the matching Beta blocker: <plugin-name> - <summary> issue labeled beta-blocker. Contributors cannot label PRs, so the title is the PR-side signal for maintainers and automation.

• Problem: Issue Add thorough QA tests for observability ingestor #7834 identified that observability ingestion lacked thorough automated QA coverage beyond parser-only checks.
• Why it matters: Regressions in parsing, tailing, watch updates, or file rotation can silently reduce observability data quality without obvious runtime failures.
• What changed: Added comprehensive integration QA for ingest-to-SQLite flows, synthetic JSONL data generation/tests, parser coverage, and supporting ingestion plumbing for watch/rotation scenarios.
• What did NOT change (scope boundary): No user-facing CLI/UI behavior changes, no new external integrations, and no config default changes.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes Add thorough QA tests for observability ingestor #7834
• Related fix: add @lid format support and allowFrom wildcard handling #1
• This PR fixes a bug or regression

Root Cause (if applicable)

For bug fixes or regressions, explain why this happened, not just what changed. Otherwise write N/A. If the cause is unclear, write Unknown.

• Root cause: The observability path had limited automated QA guardrails for full pipeline behavior across real-world log formats and live file behaviors.
• Missing detection / guardrail: No deterministic integration test that validated message/session/system/cache events from JSONL input through SQLite query assertions.
• Contributing context (if known): Prior validation depended heavily on manual or ad-hoc checks.

Regression Test Plan (if applicable)

For bug fixes or regressions, name the smallest reliable test coverage that should catch this. Otherwise write N/A.

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: src/observability/ingestor.integration.test.ts, src/observability/synthetic-data.test.ts, src/observability/parsers/parsers.test.ts
• Scenario the test should lock in: Ingest synthetic logs across supported event formats, persist rows in SQLite, handle rotation to shorter files, and process live watch updates.
• Why this is the smallest reliable guardrail: Failure modes span parser, reader, cursor tracking, watcher behavior, and persistence boundaries.
• Existing test that already covers this (if any): Existing parser tests covered isolated parsing but not full ingestion lifecycle behavior.
• If no new test is added, why not: N/A

User-visible / Behavior Changes

List user-visible changes (including defaults/config).
If none, write None.
None.

Diagram (if applicable)

For UI changes or non-trivial logic flows, include a small ASCII diagram reviewers can scan quickly. Otherwise write N/A.

Before:
[JSONL logs] -> [partial parser-level checks] -> [manual validation]

After:
[JSONL logs (synthetic/realistic fixtures)]
  -> [ObservabilityIngestor: parse + tail/watch + rotation handling]
  -> [SQLite persistence]
  -> [query assertions in integration tests]

Security Impact (required)

• New permissions/capabilities? (Yes/No) No
• Secrets/tokens handling changed? (Yes/No) No
• New/changed network calls? (Yes/No) No
• Command/tool execution surface changed? (Yes/No) No
• Data access scope changed? (Yes/No) No
• If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

• OS: Ubuntu Linux
• Runtime/container: Node.js + pnpm workspace
• Model/provider: N/A
• Integration/channel (if any): Synthetic Telegram/Discord/Slack-style JSONL fixtures
• Relevant config (redacted): Temporary local directories and explicit watchedPaths in tests

Steps

1. pnpm test -- src/observability/parsers/parsers.test.ts src/observability/synthetic-data.test.ts src/observability/ingestor.integration.test.ts
2. pnpm build
3. (Optional) pnpm exec tsx scripts/generate-observability-synthetic.ts --help

Expected

• Observability parser, synthetic-data, and integration test files pass.
• Ingested events are persisted and queryable via SQLite.
• Rotation and watch-mode ingestion paths behave deterministically.
• Build succeeds.

Actual

• Passed: 3 test files, 17 tests.
• Passed: pnpm build.

Evidence

Attach at least one:

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

• Verified scenarios: End-to-end ingest to SQLite query assertions, synthetic dataset ingestion, watch-mode live update ingestion, and file-rotation handling.
• Edge cases checked: Watcher-limit constrained environment behavior (ENOSPC) and deterministic ingestion verification path.
• What you did not verify: Live external channel traffic against a running gateway instance.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.
  If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

• Backward compatible? (Yes/No) Yes
• Config/env changes? (Yes/No) No
• Migration needed? (Yes/No) No
• If yes, exact upgrade steps: N/A

Risks and Mitigations

List only real risks for this PR. Add/remove entries as needed. If none, write None.

• Risk: Large observability logs can increase ingestion time and local SQLite size.
  • Mitigation: Batch insert strategy, tracked-file cursoring, and explicit high-volume test coverage.
• Risk: OS file-watcher limits can reduce reliability of live watch mode in constrained environments.
  • Mitigation: Deterministic ingestion test coverage plus explicit watcher error handling paths.