Skip to content

test(importer): the paged hermes session enumeration is unverified against a real store #272

Description

@0xKT

Problem

list_exportable_sessions (raven/importer/scanners/hermes.py) has two paths. When
hermes counts at most the 100 rows its dry-run listing prints, one unwindowed probe answers
the question. Above that, the started_at axis is partitioned into half-open,
minute-aligned windows and probed recursively, and _uncovered reconciles the collected
ids against the header count.

#264 ships that second path, but it has only ever run against synthetic stores: the
install it was developed against had fewer than 100 exportable sessions. The PR
description flags it as one of two unverified paths. What is unproven on real data:

  • Listing rows. The parser takes the first token of any indented line. Real stores
    carry ids the timestamped default does not look like -- cron ids (cron_<job>_<stamp>)
    and bare uuid4 from ACP -- and a row that fails to parse is caught only by the
    len(ids) > expected arithmetic, which cannot see a row that parses into something
    wrong.
  • The reconciliation under movement. reopen_session() clears ended_at and only
    ended sessions are candidates, so a session can leave the candidate set mid-scan. The
    code counts that rather than raising, which is right, but it also means a genuine
    coverage bug and a benign race are indistinguishable from the outside. Nobody has seen
    what the count actually looks like on a store in use.
  • Cost. Probe counts and wall clock were measured against synthetic stores in the feat(*): add hermes as a cold-start import source #264
    discussion (17 / 27 / 79 / 159 probes for 150 / 500 / 2000 / 5000 sessions), not against
    a real one.
  • Windows. _PARTITION_FLOOR moved to 1970-01-02 in bf27da2, which closes the
    pre-epoch OSError class, but the paged path itself has never run on Windows.

Proposal

Run the paged path against a real Hermes home with more than 100 ended sessions:

  1. raven import scan and raven import run --tier full, on macOS or Linux and once on
    Windows.
  2. Record the probe count, the wall clock, and whatever _uncovered reports; confirm the
    ids that come back cover the cron and ACP shapes.
  3. Fix whatever that turns up, and pin the real row shapes as fixtures in
    tests/test_importer_hermes_scanner.py so the parser is anchored to observed output
    rather than to the format as read.

Alternatives considered

Bounded concurrency was raised and measured in the #264 discussion: asyncio.gather on
the two halves saves 10-22s at 2000-5000 sessions but, because _collect_window is
recursive, expands the tree exponentially -- peak concurrent hermes processes 28 and 72 for
those two sizes. A Semaphore(4) version keeps most of the gain. It was correctly left out
of #264: enumeration is under 1% of a full-tier import, and the correctness of the bounded
version would rest on the same synthetic data this issue exists to replace. Worth revisiting
only after the path has run for real.

Area

Memory / skills

Additional context

Introduced by #264. The two-path split is at list_exportable_sessions; the recursion is
_collect_window; the reconciliation is _uncovered. The 100-row cap is upstream
(hermes_cli/sessions_cmd.py, candidates[:100]), so the paged path is reachable by any
user with a long Hermes history, not an edge case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions