You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
list_exportable_sessions (raven/importer/scanners/hermes.py) has two paths. When
hermes counts at most the 100 rows its dry-run listing prints, one unwindowed probe answers
the question. Above that, the started_at axis is partitioned into half-open,
minute-aligned windows and probed recursively, and _uncovered reconciles the collected
ids against the header count.
#264 ships that second path, but it has only ever run against synthetic stores: the
install it was developed against had fewer than 100 exportable sessions. The PR
description flags it as one of two unverified paths. What is unproven on real data:
Listing rows. The parser takes the first token of any indented line. Real stores
carry ids the timestamped default does not look like -- cron ids (cron_<job>_<stamp>)
and bare uuid4 from ACP -- and a row that fails to parse is caught only by the len(ids) > expected arithmetic, which cannot see a row that parses into something
wrong.
The reconciliation under movement.reopen_session() clears ended_at and only
ended sessions are candidates, so a session can leave the candidate set mid-scan. The
code counts that rather than raising, which is right, but it also means a genuine
coverage bug and a benign race are indistinguishable from the outside. Nobody has seen
what the count actually looks like on a store in use.
Cost. Probe counts and wall clock were measured against synthetic stores in the feat(*): add hermes as a cold-start import source #264
discussion (17 / 27 / 79 / 159 probes for 150 / 500 / 2000 / 5000 sessions), not against
a real one.
Windows._PARTITION_FLOOR moved to 1970-01-02 in bf27da2, which closes the
pre-epoch OSError class, but the paged path itself has never run on Windows.
Proposal
Run the paged path against a real Hermes home with more than 100 ended sessions:
raven import scan and raven import run --tier full, on macOS or Linux and once on
Windows.
Record the probe count, the wall clock, and whatever _uncovered reports; confirm the
ids that come back cover the cron and ACP shapes.
Fix whatever that turns up, and pin the real row shapes as fixtures in tests/test_importer_hermes_scanner.py so the parser is anchored to observed output
rather than to the format as read.
Alternatives considered
Bounded concurrency was raised and measured in the #264 discussion: asyncio.gather on
the two halves saves 10-22s at 2000-5000 sessions but, because _collect_window is
recursive, expands the tree exponentially -- peak concurrent hermes processes 28 and 72 for
those two sizes. A Semaphore(4) version keeps most of the gain. It was correctly left out
of #264: enumeration is under 1% of a full-tier import, and the correctness of the bounded
version would rest on the same synthetic data this issue exists to replace. Worth revisiting
only after the path has run for real.
Area
Memory / skills
Additional context
Introduced by #264. The two-path split is at list_exportable_sessions; the recursion is _collect_window; the reconciliation is _uncovered. The 100-row cap is upstream
(hermes_cli/sessions_cmd.py, candidates[:100]), so the paged path is reachable by any
user with a long Hermes history, not an edge case.
Problem
list_exportable_sessions(raven/importer/scanners/hermes.py) has two paths. Whenhermes counts at most the 100 rows its dry-run listing prints, one unwindowed probe answers
the question. Above that, the
started_ataxis is partitioned into half-open,minute-aligned windows and probed recursively, and
_uncoveredreconciles the collectedids against the header count.
#264 ships that second path, but it has only ever run against synthetic stores: the
install it was developed against had fewer than 100 exportable sessions. The PR
description flags it as one of two unverified paths. What is unproven on real data:
carry ids the timestamped default does not look like -- cron ids (
cron_<job>_<stamp>)and bare uuid4 from ACP -- and a row that fails to parse is caught only by the
len(ids) > expectedarithmetic, which cannot see a row that parses into somethingwrong.
reopen_session()clearsended_atand onlyended sessions are candidates, so a session can leave the candidate set mid-scan. The
code counts that rather than raising, which is right, but it also means a genuine
coverage bug and a benign race are indistinguishable from the outside. Nobody has seen
what the count actually looks like on a store in use.
discussion (17 / 27 / 79 / 159 probes for 150 / 500 / 2000 / 5000 sessions), not against
a real one.
_PARTITION_FLOORmoved to 1970-01-02 inbf27da2, which closes thepre-epoch
OSErrorclass, but the paged path itself has never run on Windows.Proposal
Run the paged path against a real Hermes home with more than 100 ended sessions:
raven import scanandraven import run --tier full, on macOS or Linux and once onWindows.
_uncoveredreports; confirm theids that come back cover the cron and ACP shapes.
tests/test_importer_hermes_scanner.pyso the parser is anchored to observed outputrather than to the format as read.
Alternatives considered
Bounded concurrency was raised and measured in the #264 discussion:
asyncio.gatheronthe two halves saves 10-22s at 2000-5000 sessions but, because
_collect_windowisrecursive, expands the tree exponentially -- peak concurrent hermes processes 28 and 72 for
those two sizes. A
Semaphore(4)version keeps most of the gain. It was correctly left outof #264: enumeration is under 1% of a full-tier import, and the correctness of the bounded
version would rest on the same synthetic data this issue exists to replace. Worth revisiting
only after the path has run for real.
Area
Memory / skills
Additional context
Introduced by #264. The two-path split is at
list_exportable_sessions; the recursion is_collect_window; the reconciliation is_uncovered. The 100-row cap is upstream(
hermes_cli/sessions_cmd.py,candidates[:100]), so the paged path is reachable by anyuser with a long Hermes history, not an edge case.