[OMEGA-312] feat: persistent session store (omegaclaw-sessions) - #274
[OMEGA-312] feat: persistent session store (omegaclaw-sessions)#274amiroussama wants to merge 2 commits into
Conversation
Adds durable, queryable conversation/session storage backed by SQLite: sessions, messages, and tool calls with begin/record/end APIs, full-text-ish search, show, resume, and export — plus the `omegaclaw-sessions` CLI to browse them. `ingest_trace` backfills the session DB from a reasoning-trace JSONL (the format src.tracing emits), so runs get searchable session summaries — tool names, provider/model, result sizes — even when bodies weren't recorded. Pure-Python, stdlib sqlite3. Autotests/test_session_store.py (host-runnable; the trace-ingest test writes the JSONL directly, staying independent of the tracing module) + run_mandatory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6e9f4f0 to
85bb500
Compare
… + webhooks Adds a scheduler for time-based agent tasks: cron / interval / one-shot jobs and webhook subscriptions, persisted in a SQLite jobs DB, with HMAC-verified webhook delivery and the `omegaclaw-cron` CLI to add/list/remove/run jobs. Job runs are recorded via the session store for auditability. `is_safe_skill_name` is inlined (validates job/subscription ids used as filesystem path segments) so the scheduler carries no dependency on the skills subsystem. Depends on singnet#274 (session_store) — included in this branch and drops out on rebase once it merges. Pure-Python, stdlib sqlite3/hmac. Autotests/test_scheduler.py (host-runnable) + run_mandatory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alyona-snet
changed the title
paul-v-snet
left a comment
paul-v-snet
left a comment
There was a problem hiding this comment.
Additionally, once the identified issues are addressed, please make sure the relevant documentation in docs/ or README.md is created or updated accordingly.
| def ingest_trace(path: str, conn: Optional[sqlite3.Connection] = None) -> Dict[str, Any]: | ||
| """Backfill sessions from a reasoning-trace JSONL (what every run writes via ``src.tracing``). | ||
|
|
||
| Maps the ACTUAL tracing phases — ``iteration_start`` / ``llm_call`` / ``action_parse`` / | ||
| ``policy_decision`` / ``iteration_result`` / ``error`` / ``iteration_end`` — into messages + | ||
| tool calls. Produces useful **searchable summaries even without bodies** (tool names, | ||
| provider/model, result size, error codes); when ``OMEGACLAW_TRACE_BODIES`` was set, the | ||
| prompt/response/result/failed-action bodies are ingested too. Best-effort per line. | ||
| """ |
There was a problem hiding this comment.
Could you please clarify which src.tracing module this refers to? I couldn't find such a module either in the main repository or in this PR.
As a result, it looks like the reasoning-trace JSONL mentioned here is not currently generated or populated anywhere. Could you please clarify which scenario this function is intended to handle and where the corresponding trace file is expected to come from?
There was a problem hiding this comment.
Good catch — src.tracing isn't in this branch or in main; it's the module from #270 (contrib/reasoning-trace), and this PR is stacked on it. That was implied but never stated. I've reworded the docstrings so it's explicit that ingest_trace reads the JSONL #270 emits and only becomes reachable once #270 merges underneath. The phase names it maps line up 1:1 with what that module writes.
| # --------------------------------------------------------------------------- recording API | ||
|
|
||
| def begin_session(session_id: str, *, provider: str = "", channel: str = "", task: str = "", | ||
| meta: Optional[Dict[str, Any]] = None, conn: Optional[sqlite3.Connection] = None) -> str: |
There was a problem hiding this comment.
Could you please clarify how the recording API is expected to be used?
As far as I can see, none of the functions implemented in this recording API section are actually called anywhere except from tests and ingest_trace. However, ingest_trace itself does not seem to have a working way to populate the database, since the src.tracing dependency mentioned is not present either in the main repository or in this PR.
At the moment, I don't see any entry point that could initiate database recording, either from the agent or from the user side:
- Agent: there are no corresponding skills or MeTTa expressions that would allow the main loop to manage a session and record its data.
- User: the only entry point appears to be the
cmd_ingestCLI tool, which callssrc.sessions_store.ingest_trace. However, as mentioned above,ingest_tracedepends on the trace file produced bysrc.tracing, which is not present.
So, as far as I understand, the recording API is currently isolated from the rest of the system, and there is no available path that can actually populate the database.
There was a problem hiding this comment.
You're right that nothing calls the recording API yet outside tests, and that ingest_trace is currently the only population path (itself gated on #270). Rather than bolt loop-wiring onto this PR, I'd like to keep it scoped to the store + CLI + ingest and land live recording as a follow-up once #270 is in — I've documented the recording API as that integration surface so it's clear where the loop will hook in. If you'd prefer the wiring in this same PR, I'm happy to do that instead — just say the word and I'll stack it on #270.
| mock/test_transition_metta_to_remember_mock.py | ||
| mock/test_transition_pin_to_remember_mock.py | ||
| mock_websocket/test_wschat_unit.py | ||
| test_session_store.py |
There was a problem hiding this comment.
Should be moved to Autotests/unit/
There was a problem hiding this comment.
Done — moved to Autotests/unit/test_session_store.py and updated the run_mandatory entry. Passes 9/9 from the new location.
- Move test to Autotests/unit/ and fix run_mandatory path (matches repo convention); adjust _REPO_ROOT depth for the new location. - Make the src.tracing dependency explicit in docstrings: ingest_trace reads the JSONL emitted by src.tracing (PR singnet#270); this PR is stacked on it. - Clarify the recording API is the loop's integration surface (live wiring is a follow-up). - Add docs/reference-session-store.md and link it from docs/README.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
On docs: added One more thing I noticed while addressing this — the PR description has gone stale and I'll fix it: it says "Depends on #266" and claims secrets are scrubbed via the shared redactor, but #266 was closed unmerged, All of the above is pushed as |
4 participants
Description
Fixes #273. Adds a persistent, queryable session store backed by SQLite.
src/session_store.py— sessions / messages / tool_calls tables withbegin_session,record_message,record_tool_call,record_snapshot,end_session, pluslist_sessions/show/search/resume/export. Secrets in stored/searched/exported text are scrubbed via the shared redactor.ingest_trace(path)backfills the store from a reasoning-trace JSONL (iteration_start/llm_call/action_parse/policy_decision/iteration_resultphases), so runs get searchable summaries even without bodies.scripts/omegaclaw-sessions— CLI to list / show / search / resume / export sessions.Depends on #266 — shares
src/redaction.py(included in this branch; identical file, drops out on rebase once #266 merges).How Has This Been Tested?
Autotests/test_session_store.py(pure-Python, stdlibsqlite3, host-runnable), registered inrun_mandatory— begin/record/show/resume/export round-trips, search by content/tool/provider, secret redaction in search/show/export, list + missing-session handling, andingest_traceagainst a directly-written trace JSONL (kept independent of the tracing module). Run:cd Autotests && python3 test_session_store.py→ 9/9 pass.Checklist