MorletDB 0.3.14-alpha.1

Changelog

0.3.14-alpha.1

SQL version 0.3.11, with an update script from 0.3.10: run ALTER EXTENSION pg_morlet UPDATE after installing.
Python SDK 0.1.0a7. Chunk format version 6 is written by default for float32 samples where it is smaller: releases
up to 0.3.13-alpha.1 cannot read those chunks, so a downgrade needs them re-appended with
morlet.float32_chunks = off first. Existing chunks keep their format.

• Float32 samples about half the size: chunk format version 6, on by default (#326, study B8). Chunks whose
  samples are all float32 values widened to float64, such as PLC and CNC controller signals, are coded in float32
  terms, on a float32 grid or as 31-bit magnitudes with sign runs. A version-6 chunk is kept only where it is
  strictly smaller than the configured format's chunk; every other chunk is written byte for byte as before.

  • Setting: morlet.float32_chunks (on by default, off), at any level. Reads decode every version whatever
    the setting. Checked like any chunk with morlet.verify_appends on.
  • Compatibility: pg_morlet 0.3.13-alpha.1 and earlier, and their decoders, cannot read version-6 chunks.
    Compressed transfer sends samples to clients that do not report version 6; the native decoder now reports 6.
    Embedded-edition files (file format 1) never contain version 6. Before a downgrade, re-append such chunks
    with the setting off.
  • Measured (B8, frozen protocol): on 321 IU4 runs, force 3.02 and controller channels 2.29 bytes per sample
    instead of 6.60 and 6.52; table level 3.36 and 3.03; no chunk larger on B6's 30 inputs; same bytes on macOS
    arm64, Linux arm64 and amd64. Encoding integer counts that fit in float32 is slower (about 13–15 instead of
    62 M samples/s) for a 1–4 % gain (results).
  • Tests: golden chunks, a cross-platform equivalence file, property and malformed-chunk tests, Kani proofs, a
    decode_v6 fuzz target, and database tests for the setting and append verification.

• Context values build about 4× faster (#414). On encoded collections, morlet_build_context_values now
  reads a segment in one raw read (morlet_read_interval_raw) and computes its values from the bytes with the new
  internal morlet_simple_values_raw. Before, it joined 32,768-sample float8[] pieces in PL/pgSQL, and every join
  copied the growing array: about half of the time for a 220k-sample segment. simple() also checks and sums in one
  pass and takes the moments, peak and sum of squares in a second (it used six). Each sum adds the same terms in the
  same order from the same start, so every value is bit-identical.

  • Tests: against the six-pass version on continuous, quantised, signed-zero, tiny, maximal and offset data, and
    against the float8[] path on segments that span several pieces.
  • Measured on an M-series Mac: 7 channels of 220k samples took 82–87 ms instead of 320–346 ms (alternating, same
    session); simple() alone takes 1.00 ms instead of 1.40 ms.
  • Collections with float8[] chunks keep the old path.

• Prometheus metrics from the steward (#197). morlet-steward metrics --listen HOST:PORT serves /metrics in
  the Prometheus text format, using only the standard library. It reports:

  • the steward's overall state and items, heartbeat age, breakers, recent actions and waiting plans;
  • the ingest hold, and disk free, total and database bytes;
  • the last integrity pass per collection;
  • morlet_collection_status per collection and window size;
  • dead tuples and vacuum age of each collection's tables.

  A scrape only reads; queries have a 5-second timeout, and per-collection statistics are cached (60 s by default).
  It is tested offline (text format, an unreachable database) and against a database (test_steward_db.py).

• morlet_compare_summary computes the pooled spreads once (#335). For the duration deviation it called
  morlet_compare_to_history, which computed the pooled spreads only to discard them; it computed them again for
  calibrated_score. The comparison's body is now the internal morlet_history_comparison, with the spreads
  optional: morlet_compare_to_history calls it with them (same signature and results), morlet_compare_summary
  without them for the duration deviation. On a 320-segment history morlet_compare_summary took 23 ms instead of
  35 ms (p50), with identical results on every segment. The decision state (tools/mcp/decision.py) already
  compares in Python without the spreads; it calls morlet_compare_summary only for the summary quantile.

• Search can leave out the same part or the whole source recording (#384). On cyclic production
  morlet_search_similar returned the next cycles of the query's own recording, because it leaves out only the
  windows overlapping the query. The new, experimental morlet_search_similar_excluding returns the same matches
  with evidence as morlet_search_similar_evidence and takes exclude: 'window' (the default, identical results),
  'occurrence' (every current context segment whose occurrence, e.g. the part, equals the query's, in any
  recording, as the history comparison excludes the same occurrence), 'recording' or 'none'. Excluded windows are
  left out of the scans and of the max_candidates count on both paths; probe search also leaves recordings
  excluded whole out of its index scans. The MCP tool
  search_similar and the Python SDK's search_similar take exclude. The stable search functions keep their
  signatures, statements and results; their bodies moved into internal functions that accept excluded ranges.

• The image sizes shared_buffers to small memory limits (#466). Under a container memory limit below 512 MB
  the image starts PostgreSQL with shared_buffers at a quarter of the limit (at least 16 MB) instead of the
  default 128 MB, unless shared_buffers is set on the command line, with ALTER SYSTEM or in postgresql.conf;
  MORLET_AUTO_SHARED_BUFFERS=off turns it off. At 256 MB the default left as little as about 20 MB that the
  kernel could not reclaim after a bulk load, and a backend was killed once (Study M1d); with 64 MB the peak fell
  by about 80 MB with no measurable change in speed. Limits of 512 MB and more, and runs without a limit, are
  unchanged. The release workflow checks every case (deploy/test_memory_sizing.sh).

0.3.13-alpha.1

Security and data-safety release: SQL 0.3.10 (ALTER EXTENSION pg_morlet UPDATE from 0.3.9; the update also puts the
episodic-memory tables of collections already under row security under it), Python SDK 0.1.0a6. The chunk format is
unchanged; existing collections keep their layout. Steward installations run steward.py init again.

• Steward plans: the approval rules hold at the table (#439). A trigger on morlet_steward.plans enforces the
  plan workflow for every write, not only through decide() and apply(): a new plan is undecided and unapplied,
  with its author, creation time and 24-hour validity set by the database; a plan's kind, target, payload, evidence
  and validity never change; a decision is made once, by an approver other than the author, before expiry; an
  application is made once, of an approved, unchanged, unexpired plan; plans are not deleted. Existing installations
  get it by running steward.py init again.
  SQL version 0.3.10, with an update script from 0.3.9: run ALTER EXTENSION pg_morlet UPDATE after installing.

• A representative keeps its samples while it represents an episode (#440). morlet_set_level checked the
  representative only when an episode moved to level 2; a later move could take the representative to level 3 or 4,
  leaving the level-2 episode without a real example. Now, after all moves of a call (stored levels overlaid with the
  call's moves, so dry runs too), no level-2 episode may name a representative above level 1, or the whole call is
  refused; the members can leave level 2 in the same call. The representative row is locked while it is assigned, so
  concurrent moves cannot break the rule. Affects all releases with episodic memory.

• Episodic memory under row-level security (#438). morlet_enable_row_security now covers the memory tables
  (episodes, ledger, kept summaries: they follow their recording; memory runs: visible to the role that started
  them), and morlet_enable_memory applies it at once when the collection is already under it. The update to
  0.3.10 applies it to every collection already under row security, so existing memory tables are covered without
  a manual step (an update by a role that cannot alter one of them fails and rolls back).

• Envelopes build on after compaction removed the first windows (#441). morlet_build_envelopes read the stride
  from the first stored window after 0; once level-1 compaction had removed the early windows, a rebuild with the
  original stride failed (built with stride 6144, not 2048), new samples stayed unsearchable and the steward's
  envelope job could trip its breaker. When that first window does not match, the builder now checks that every
  stored window lies on the requested grid and every grid position without a window is one it skips itself (out of
  search, or samples removed); another stride's grid is still refused.

• S3 import resumes objects that were cut off (#442). morlet_s3.py import treated any recording carrying an
  object's key and ETag as proof that the whole object was imported, but the importer commits in batches, so an
  interrupted object was skipped on the next run and its remaining samples and channels never arrived. Objects are
  now imported one at a time and recorded as complete in <collection>_s3_imports only after the importer finished
  them; anything else is imported again (identical chunks are no-ops). Collections imported before have no record,
  so their objects are read once more on the next run.

• **Chunk s...

MorletDB 0.3.13-alpha.1

Changelog

0.3.13-alpha.1

Security and data-safety release: SQL 0.3.10 (ALTER EXTENSION pg_morlet UPDATE from 0.3.9; the update also puts the
episodic-memory tables of collections already under row security under it), Python SDK 0.1.0a6. The chunk format is
unchanged; existing collections keep their layout. Steward installations run steward.py init again.

• Steward plans: the approval rules hold at the table (#439). A trigger on morlet_steward.plans enforces the
  plan workflow for every write, not only through decide() and apply(): a new plan is undecided and unapplied,
  with its author, creation time and 24-hour validity set by the database; a plan's kind, target, payload, evidence
  and validity never change; a decision is made once, by an approver other than the author, before expiry; an
  application is made once, of an approved, unchanged, unexpired plan; plans are not deleted. Existing installations
  get it by running steward.py init again.
  SQL version 0.3.10, with an update script from 0.3.9: run ALTER EXTENSION pg_morlet UPDATE after installing.

• A representative keeps its samples while it represents an episode (#440). morlet_set_level checked the
  representative only when an episode moved to level 2; a later move could take the representative to level 3 or 4,
  leaving the level-2 episode without a real example. Now, after all moves of a call (stored levels overlaid with the
  call's moves, so dry runs too), no level-2 episode may name a representative above level 1, or the whole call is
  refused; the members can leave level 2 in the same call. The representative row is locked while it is assigned, so
  concurrent moves cannot break the rule. Affects all releases with episodic memory.

• Episodic memory under row-level security (#438). morlet_enable_row_security now covers the memory tables
  (episodes, ledger, kept summaries: they follow their recording; memory runs: visible to the role that started
  them), and morlet_enable_memory applies it at once when the collection is already under it. The update to
  0.3.10 applies it to every collection already under row security, so existing memory tables are covered without
  a manual step (an update by a role that cannot alter one of them fails and rolls back).

• Envelopes build on after compaction removed the first windows (#441). morlet_build_envelopes read the stride
  from the first stored window after 0; once level-1 compaction had removed the early windows, a rebuild with the
  original stride failed (built with stride 6144, not 2048), new samples stayed unsearchable and the steward's
  envelope job could trip its breaker. When that first window does not match, the builder now checks that every
  stored window lies on the requested grid and every grid position without a window is one it skips itself (out of
  search, or samples removed); another stride's grid is still refused.

• S3 import resumes objects that were cut off (#442). morlet_s3.py import treated any recording carrying an
  object's key and ETag as proof that the whole object was imported, but the importer commits in batches, so an
  interrupted object was skipped on the next run and its remaining samples and channels never arrived. Objects are
  now imported one at a time and recorded as complete in <collection>_s3_imports only after the importer finished
  them; anything else is imported again (identical chunks are no-ops). Collections imported before have no record,
  so their objects are read once more on the next run.

• Chunk shipping checks the target before writing (#443, #444). An existing target recording must have the
  source's parameters (rate, channel, unit, metadata, quantum, window, stride, normalization, extractor); before,
  shipping into an empty recording at another rate succeeded and reported identical: true. The target's existing
  chunks must equal the source's first chunks byte for byte and on the same boundaries; before, a conflicting
  prefix was only detected after the source's later chunks had been committed, leaving a mixed recording. A run now
  ships up to the source end read at its start, so a source that keeps recording no longer fails the final check.
  The SDK's export_file refuses a file recording with other parameters in the same way. Affects 0.3.12-alpha.1.

• Decision states from memory: decision.HistoryCache (#403). A process that decides cut after cut passes
  history_cache=HistoryCache() to decision_states and keeps it: each collection's current segments with values and
  its active labels stay in memory, indexed by rate, channel, unit, spec and context, and each call fetches only new
  segments (and values built since for known ones). A fingerprint read in the same snapshot (highest segment and
  label ids, numbers of current segments and active labels) reloads the pool when a segment went stale or was
  deleted, and the labels when they changed. The states are identical to the uncached ones (a test changes the data
  between calls: new cuts, late values, labels added and retracted, segments rebuilt, a recording deleted). On a
  Raspberry Pi 3B+ with ~57,000 segments, p50 per decision of 9 signals 1.5 s -> 0.68 s; the fingerprint's counts
  (~0.2 s there) are the next step.

• Decision states with long histories faster (#403). simple-v2's grid check (quantised_step) ran in pure
  Python over every value column of the history; it now uses numpy with the same float64 operations. A history's
  comparison floors are computed once, and neighbours are ranked with one sort. The states are identical to before;
  a test compares the grid check with morlet_quantised_step. On a Raspberry Pi 3B+ with ~600 earlier segments per
  signal, these parts took about 0.7 s of each decision; the history query remains the larger cost.

0.3.12-alpha.1

SQL version 0.3.9, with an update script from 0.3.8: run ALTER EXTENSION pg_morlet UPDATE after installing.
Python SDK 0.1.0a5 (reads and transfers Grizzbit 2.1 chunks). The default chunk format is unchanged (Grizzbit 1);
Grizzbit 2.1 is opt-in. Existing collections keep their layout and their value rules (simple-v1); new collections
get simple-v2.

• Lossless chunk shipping from edge to centre (#420). morlet_encoded_chunks returns a recording's stored
  chunks as bytes; morlet_append_encoded_chunk appends a chunk that another MorletDB encoded, keeping its bytes,
  after decoding it and running the usual append checks (offset, identical replay a no-op, conflicting samples
  refused, ingest hold, checkpoint, summary). tools/ingest/morlet_ship.py ships recordings with them: it creates
  the target recording with the source's parameters, resumes at the target's end, and compares the SHA-256 of all
  chunk bytes at the end (docs/chunk-shipping.md).
• MCP: negative search answers state their scope, checked by a scripted conversation (#379).
  tools/mcp/scope_conversation.py asks the server, through its own tools, the questions an analyst would ask of
  a collection with a period out of search, and fails when a search without a hit does not say what it searched
  and what it did not; the transcript is in examples/mcp-scope/TRANSCRIPT.md, and the test runs it. It found
  one weak answer: a search whose filters matched no recording said "every period of the matching recordings
  was searchable"; it now says that no recording matched and nothing was searched.
• Alarm and PLC messages as context events (#421). tools/ingest/morlet_events.py places each message of a
  log (CSV: time and message columns) as a context event on the recordings whose time range contains it, at the
  sample its time maps to (optionally only on the recordings of the message's machine), with the recording's
  measured time uncertainty (#393) where known. An identical re-import is a no-op. decision_state lists the
  events inside its segment (events: key, value, sample, offset in seconds).
• MCP trend over an order key (#334). order_by (a metadata key such as cut) orders the recordings by
  that key, numbers numerically, instead of by time. Numbered cuts of a tool, which have no clock time, then get
  a trend and change points named by the order value (first_change_order). Without a start time, the error
  now points to order_by.
• Label glossary (#333). morlet_define_label(collection, label, definition, unit, source) records what a
  label means; morlet_label_definitions(collection) lists the definitions. Context enabled from SQL 0.3.9 on
  gets the table; existing collections get it with their first definition. decision_state returns, for the
  labels it shows, their definitions (label_definitions) and the labels nobody defined (undefined_labels),
  and logged precedents carry them. In a demo session an agent had read not_sharp (flank wear above 0.2 mm)
  as the 0.3 mm replacement limit.
• MCP search_similar: exclude="occurrence" (#384). On cyclic production, the useful answer is the same cut
  on earlier parts. occurrence drops hits from the query's own part, cycle or run (the occurrence of the
  context segment that contains it), but keeps the other parts in the same recording, which exclude=recording
  drops as well. Tested on one recording of three parts.
• decision.finish_recordings: build a cut's channels on several connections (#414). Context, segments and
  values for several recordings at once, spread over the given autocommit connections (one thread each, each
  recording in its own transaction), with the same results as one by one. On a Raspberry Pi 3B+, 4
  connections took the end of a 7-channel cut from 1.21 to 0.72 s (p50, with #423); 7 connections were slower.
• Logged decisions keep their precedents (#418). decision.log_decision stores, beside the state the model
  saw, its k nearest labelled precedents (default 3): the state's examples (labelled cases from the wh...

MorletDB technical report 0.25

MorletDB: Recordings, Not Rows — Lag-Tolerant Waveform Retrieval in PostgreSQL, version 0.25.
Technical report / preprint by Zin Nam and Semir Maslo (Sulzer-BDiM). It has not been peer reviewed.

What changed in 0.25

• Related work gains two paragraphs and rows in the comparison matrix:
  • Similar past events in monitoring: Moirae (Balazinska et al., CIDR 2007) and Kwon, Xu and Balazinska (2007) return past events whose context resembles a new one. They compare scalar snapshots of monitored values, not waveforms.
  • Compressed high-frequency sensor data: ModelarDB compresses sensor columns at the edge with constant and linear models plus residuals, lossless or within an error bound, and ships the segments to object storage. Its lossless mode keeps float32 values equal, not samples bit-identical.
• No result, number or figure changes. The main report is now 16 pages.

Files

• morletdb-v0.25.pdf: the report (16 pages).
• morletdb-supplement-v0.25.pdf: the supplement.
• RELEASE_MANIFEST.json: version, commits and the PDFs' SHA-256 sums. Verify with python3 research/paper/release_manifest.py --check in a checkout of this tag.

Provenance and licence

• Measured software: E1 and D1–D2 ran on 68594d7, library SHA-256 3c786684…. Later studies ran on the implementation states named in research/paper/STUDIES.md.
• Report sources: the manifest names dfa9fc8. The tag points at its merge into main (c85a4bd).
• Licence: text and figures are CC BY 4.0; code is Apache-2.0. Copyright Zin Nam.
• Funding: none.
• AI assistance: developed with Claude (Anthropic) under the authors' direction. The authors are responsible for the content.

MorletDB 0.3.12-alpha.1

Changelog

0.3.12-alpha.1

SQL version 0.3.9, with an update script from 0.3.8: run ALTER EXTENSION pg_morlet UPDATE after installing.
Python SDK 0.1.0a5 (reads and transfers Grizzbit 2.1 chunks). The default chunk format is unchanged (Grizzbit 1);
Grizzbit 2.1 is opt-in. Existing collections keep their layout and their value rules (simple-v1); new collections
get simple-v2.

• Lossless chunk shipping from edge to centre (#420). morlet_encoded_chunks returns a recording's stored
  chunks as bytes; morlet_append_encoded_chunk appends a chunk that another MorletDB encoded, keeping its bytes,
  after decoding it and running the usual append checks (offset, identical replay a no-op, conflicting samples
  refused, ingest hold, checkpoint, summary). tools/ingest/morlet_ship.py ships recordings with them: it creates
  the target recording with the source's parameters, resumes at the target's end, and compares the SHA-256 of all
  chunk bytes at the end (docs/chunk-shipping.md).
• MCP: negative search answers state their scope, checked by a scripted conversation (#379).
  tools/mcp/scope_conversation.py asks the server, through its own tools, the questions an analyst would ask of
  a collection with a period out of search, and fails when a search without a hit does not say what it searched
  and what it did not; the transcript is in examples/mcp-scope/TRANSCRIPT.md, and the test runs it. It found
  one weak answer: a search whose filters matched no recording said "every period of the matching recordings
  was searchable"; it now says that no recording matched and nothing was searched.
• Alarm and PLC messages as context events (#421). tools/ingest/morlet_events.py places each message of a
  log (CSV: time and message columns) as a context event on the recordings whose time range contains it, at the
  sample its time maps to (optionally only on the recordings of the message's machine), with the recording's
  measured time uncertainty (#393) where known. An identical re-import is a no-op. decision_state lists the
  events inside its segment (events: key, value, sample, offset in seconds).
• MCP trend over an order key (#334). order_by (a metadata key such as cut) orders the recordings by
  that key, numbers numerically, instead of by time. Numbered cuts of a tool, which have no clock time, then get
  a trend and change points named by the order value (first_change_order). Without a start time, the error
  now points to order_by.
• Label glossary (#333). morlet_define_label(collection, label, definition, unit, source) records what a
  label means; morlet_label_definitions(collection) lists the definitions. Context enabled from SQL 0.3.9 on
  gets the table; existing collections get it with their first definition. decision_state returns, for the
  labels it shows, their definitions (label_definitions) and the labels nobody defined (undefined_labels),
  and logged precedents carry them. In a demo session an agent had read not_sharp (flank wear above 0.2 mm)
  as the 0.3 mm replacement limit.
• MCP search_similar: exclude="occurrence" (#384). On cyclic production, the useful answer is the same cut
  on earlier parts. occurrence drops hits from the query's own part, cycle or run (the occurrence of the
  context segment that contains it), but keeps the other parts in the same recording, which exclude=recording
  drops as well. Tested on one recording of three parts.
• decision.finish_recordings: build a cut's channels on several connections (#414). Context, segments and
  values for several recordings at once, spread over the given autocommit connections (one thread each, each
  recording in its own transaction), with the same results as one by one. On a Raspberry Pi 3B+, 4
  connections took the end of a 7-channel cut from 1.21 to 0.72 s (p50, with #423); 7 connections were slower.
• Logged decisions keep their precedents (#418). decision.log_decision stores, beside the state the model
  saw, its k nearest labelled precedents (default 3): the state's examples (labelled cases from the whole
  history), else its labelled neighbours, never logged decisions, with recording, sample range, distance and
  labels. When the state held none, precedents_note says so. decisions and the MCP tool return them without
  the full state. History grows and labels change, so the precedents of that moment are part of the evidence.
  Decision log version 2.
• Steward: integrity checks get a minimum share under load (soak S1). On a busy edge device the 1-minute load
  stayed above the steward's gate and the builders always had new data, so morlet_verify never ran ("public.live
  verified never"). A due verification now gets one batch at least every 15 minutes (--verify-share-every),
  before the builders and despite the load gate; disk pressure still stops it. A missing chunk summary counts as
  builder backlog (summaries_pending), not as damage: under continuous ingest new chunks lack one for a while,
  and counting them would have raised a false alarm on every pass.
• morlet_add_context no longer scans the whole context table (#414). Every call looks its batch up by id
  (identical retries are no-ops). Without an index that was a sequential scan, growing with the collection: 33
  of 48 ms per call on a Raspberry Pi 3B+ with 27,000 context rows. Context enabled from SQL 0.3.9 on gets an
  index on batch_id; morlet_index_context(collection) adds it to existing collections (once, idempotent).
  On the Pi, one call dropped from 48 to 24 ms.
• Segment values faster: p75 by selection, not a full sort (#414). simple-v1's 75th percentile takes the two
  order statistics it needs by selection (O(n)) instead of sorting every sample; the values are bit-identical (a
  test compares with the sort; on a Raspberry Pi 3B+ both builds wrote the same SHA-256 over all values). One
  220,000-sample segment: 4.75 → 1.44 ms on an M-series Mac. End of a 7-channel cut on the Pi (context,
  segments, values): p50 1.38 → 1.21 s sequentially, and 0.81 → 0.72 s when the channels are built on 4
  connections.
• Clock drift is detected and reported (#393, first part). The importer fits the rate a CSV time column
  actually shows (time_axis.fitted_rate_hz, rate_error_ppm). When the timestamps depart from start + rate by
  more than one sample, it warns per recording and in the report. MCP locate_time says per match how the time
  is derived (time_basis: start + rate) and, when an import measured the departure, the uncertainty in
  seconds and samples. Time anchors, which would correct the drift, are not implemented yet.
• Value rules simple-v2: a near-zero median on a quantised history counts as zero (#387, study V2). A p75
  history of 0, −1 and −2 converter steps had a median of half a step, so a segment at 0 deviated by exactly 1.0
  and dominated every score. Under simple-v2, a history whose distinct values (at least 3) lie on multiples of
  their smallest gap is quantised, and a median within two steps of zero is treated as zero (zero_median), in
  morlet_compare_to_history, morlet_compare_summary, morlet_history_neighbors, morlet_value_spreads and
  the decision state (history.value_rules). Values that are not quantised compare exactly as before, and the
  stored values are unchanged. The rules belong to the collection: morlet_value_rules reads them and
  morlet_set_value_rules sets them. Context enabled from SQL 0.3.9 on starts with simple-v2; existing
  collections keep simple-v1 until switched. Study V2 (evaluation/values_v2/, protocol frozen before any
  run): the artefact disappears, continuous values are unchanged, and C2's and C3's AUROCs are identical under
  both rules (0 of 37,670 and 0 of 24 history medians affected). New functions: morlet_quantised_step,
  morlet_value_floors, morlet_value_rules, morlet_set_value_rules; the 0.3.8 → 0.3.9 update script adds them.
• Importer: wide plant exports (#388). For CSV, --time-column now sets the start time (ISO 8601) and,
  without --rate, the rate from the median row spacing. The metadata records how far the timestamps depart
  from the even axis; they are not corrected. --context-columns program,block stores controller columns as
  context on every signal (runs of equal value, empty cells keep the last value) and enables context on the
  collection if needed.
• MCP trend per context segment (#385). segment_context (a JSON object such as {"phase": "cut"}) takes
  the measurement points only inside current context segments whose context contains it, from each segment's
  start (a segment shorter than samples_per_point is measured whole). Without it, trend measures from each
  recording's start, which on machining data is the idle lead-in before the cut.
• Decision state about ten times faster, all channels of a cut in one call (#403). decision.decision_states
  returns the states of several segments; decision_state is its one-segment form. The history of all segments
  that share a context is read in one query, and medians, quantiles and neighbour distances are computed in Python
  with the SQL functions' rules (checked against morlet_compare_to_history and morlet_history_neighbors in
  tools/mcp/test_decision_db.py). The SQL path read the collection three times per channel and also computed the
  pooled spreads, which the state does not use. On a synthetic history shaped like a milling study (3 tools x 315
  cuts x 7 channels), seven channels took 259 ms on an M-series Mac before and 16 ms after, and on a Raspberry Pi
  3B+ 3.8 s before and 0.26 s after (median). On the real PHM 2010 milling data on the Pi (3 tools x 315 cuts,
  100 decisions of c4 and c6 late in life, soft temperature limit active), a cut's 7 states take p50 256 ms and
  p95 296 ms, from 4.7 s and 5.1 s. decision.segments_at looks up ...

MorletDB technical report 0.24

MorletDB: Recordings, Not Rows — Lag-Tolerant Waveform Retrieval in PostgreSQL, version 0.24.
Technical report / preprint by Zin Nam and Semir Maslo (Sulzer-BDiM). It has not been peer reviewed.

What changed in 0.24

• Section "Industrial use" adds two studies by Semir Maslo, with protocols, scripts and results in the repository:
  • IU2, tool wear (evaluation/industrial_iu2/): 13 end mills run to failure (Piecuch and Żabiński 2025, CC BY), 4.88 billion samples. The protocol was frozen before the raw signals were read.
    • End of life across seven confirmation tools: AUROC 0.639 with bounded lag and 0.580 with zero lag. The difference was not significant here; on the development tools bounded lag was better.
    • Level values did better: RMS per window (morlet_interval_stats) scored 0.791. The vibration level (0.831) and spindle current (0.813) came from the dataset authors' feature table, not from MorletDB.
    • Storage: 6.47 GB of chunks, 10.6 bits per sample, against 39.1 GB as float64.
  • IU1-E, edge-sized limits (evaluation/industrial_latency/): the database container limited to 2 CPUs and 512 MB.
    • History searches and window statistics were as fast as without limits: p95 42 against 43 ms, and 1.6 against 1.5 ms.
    • Only the search over all 85,042 windows slowed down: exact search 366 against 212 ms, probes 197 against 173 ms.
    • There was no out-of-memory event. The measurement used laptop cores, not edge hardware.
  • Conclusion: bounded lag was better than zero lag everywhere, significantly on the Bosch data. Level values did as well or better on both datasets.
• Grizzbit 2.1 (study B6; opt-in, not yet released): on the six waveform inputs it was 12.6% smaller in total and smaller on all six. It decoded at least 53.1 and encoded at least 10.8 M samples/s, and wrote identical bytes on four platforms.
• Updated to match: the ledger (six rows), limitations, the AI statement and the bibliography.

Files

• morletdb-v0.24.pdf: the report (15 pages).
• morletdb-supplement-v0.24.pdf: the supplement.
• RELEASE_MANIFEST.json: version, commits and the PDFs' SHA-256 sums. Verify with python3 research/paper/release_manifest.py --check in a checkout of this tag.

Provenance and licence

• Measured software: E1 and D1–D2 ran on 68594d7, library SHA-256 3c786684…. Later studies ran on the implementation states named in research/paper/STUDIES.md; IU1, IU2 and IU1-E ran on 088f03b.
• Report sources: commit 96a63e7; the manifest names 65326c7, which only corrected its date. The tag points at that merge into main (b4869d4).
• Licence: text and figures are CC BY 4.0; code is Apache-2.0. Copyright Zin Nam.
• Funding: none.
• AI assistance: developed with Claude (Anthropic) under the authors' direction; the section "Industrial use" and the studies behind it were also written with AI assistance. The authors are responsible for the content.

MorletDB 0.3.11-alpha.1

Changelog

0.3.11-alpha.1

SQL version 0.3.8, unchanged; Python SDK 0.1.0a4. The chunk format is unchanged; existing collections keep their
layout.

• MCP server starts under python -I again (#382): morlet_mcp.py loads its sibling decision.py by path. In
  0.3.10-alpha.1 the server fails in isolated mode with No module named 'decision'; a test now imports it under -I.

• MCP server, findings from a field evaluation (#384, #386):

  • search_similar takes exclude: "recording" drops every hit in the query's recording, and "metadata:<key>"
    drops hits from recordings sharing the query's value of that key (e.g. the other channels of the same part). It
    fetches up to 100 candidates so that top_k hits remain, and reports how many were left out. The SQL option
    remains open in #384.
  • coverage treats recordings that overlap in time (channels of one acquisition) as one recorded block, so covered
    time is no longer added up per channel (covered_fraction stays at most 1), and snapshots recorded on several
    channels are recognised as snapshots. New fields: recorded_blocks, recordings_per_block.
  • locate_time says when a time falls into a gap, or before the first or after the last recording, and names the
    nearest recording on each side.
  • latest reports a too-small limit by that name instead of the SQL argument max_recordings.

• Docs (#389): morlet_read_interval_raw returns a bare bytea, while morlet_read_interval returns a table with
  samples; notes for Windows (no chmod 600 on NTFS, running the tools from a container).

• The chunk codec is now the grizzbit crate (Grizzbit 1 = chunk format versions 2 and 3, unchanged):
  crates/grizzbit holds the encoder, decoder, CRC-32C, sample limits, unit and property tests, golden-chunk test,
  Kani proofs and the codec's fuzz targets, with no dependencies. morlet_core::rawcodec re-exports it, so every
  existing path keeps compiling; morlet_core::MAX_SAMPLES and MAX_AMPLITUDE are the codec's limits.
  CodecError::Signal now carries grizzbit::SampleError (same messages; SignalError: From<SampleError>). No
  byte changes: a new frozen vector (testdata/grizzbit), written by the encoder before the move, must be
  reproduced exactly, and SQL function names and messages are unchanged. The crate is not published.

• Docs: the chunk codec is named Grizzbit: docs/grizzbit.md describes it (what it does, why it
  is lossless, where it is strong and weak in B4, lineage, versions, the planned Grizzbit 2 of #326);
  docs/chunk-format.md is retitled "Grizzbit format specification" with its content unchanged (Grizzbit 1 = chunk format
  versions 2 and 3); the README and report 0.22 (paper Section 4, supplement Section 1) name it. No format or behaviour
  change.

• Every append path proves at write time that a chunk decodes to its input (#366): with
  morlet.verify_appends on (the default), the batch append morlet_append_chunks_raw decoded each encoded chunk
  again and compared it bit for bit with its input, but the single-chunk append morlet_append_chunk (and so
  morlet_append_chunk_raw, morlet_append_chunks, and the batch append's chunk-by-chunk fallback for retries,
  overlaps and float8[] collections) stored morlet_encode_chunk's output unchecked. morlet_encode_chunk now
  runs the same check, from one function shared with the batch path, so no SQL function changes. The codec was
  and is lossless by design and every chunk keeps its CRC-32C; this closes the gap in the write-time proof.
  Tests: the single-chunk path stores the same bytes with the check on and off, and refuses (single and batch)
  a code that does not decode to its input, which with the check off is stored, so the check is what refuses it.

• File and server transfers verify samples, not chunk bytes (#356): Client.import_file and
  Client.export_file compared a hash of the chunk bytes on both sides after every 256 chunks, and to continue a
  recording. A file whose chunks another encoder version wrote (the same samples as other bytes) would have been
  refused with "chunks differ after the import", although no sample was lost. They now compare a hash of the
  decoded float64 samples, every bit (hashed by the server from morlet_read_interval_raw), and continue a
  recording if its samples equal the other side's first samples. The chunk bytes are still compared when both
  sides report the same encoder version, and reported: report=[...] collects a TransferCheck per recording.
  Tests import, verify and continue a file written with other chunk bytes, export server chunks written with
  other bytes, and refuse a sample changed by one ulp, a changed sign of zero, and a sample changed on the way.

• Sample rates up to 100 MHz (#342): recordings without feature extraction (extractor 'none') accept rates up
  to 100 MHz instead of 200 kHz, so acoustic emission and ultrasound (1–10 MHz) can be stored, read and summarized
  losslessly. Nothing else depended on the rate. L1's rate points at 1, 2.5, 3.125, 10 and 100 MHz: every derived
  time exact, every sample read back identical, about 7 M samples/s per writer at every rate
  (evaluation/limits_l1/results-342). Rates above 100 MHz are refused with a clear message. Search features keep
  their domains (128 Hz–200 kHz, and 1–128 Hz).

0.3.10-alpha.1

SQL version 0.3.8, with an update script from 0.3.7. Python SDK 0.1.0a3. The chunk format is unchanged;
existing collections keep their layout. New: the embedded edition (experimental) and file format 1.

• Embedded edition, experimental (#351): MorletDB in one SQLite file, with no server. Implemented in
  crates/morlet-sqlite, with SQLite included, as a Rust library, a C interface and the morlet-sqlite command line.
  In Python: morletdb.open("x.morlet"), which offers ingest, append, read and stats. Chunks and summaries are byte
  for byte pg_morlet's, so interval statistics are the same bit for bit (cross-checked against pg_morlet). Client.import_file and Client.export_file move recordings between a file and a server
  chunk for chunk, checking the chunk bytes on both sides. The SDK's platform wheels carry the library, and the
  release attaches morlet-sqlite-<platform>.tar.gz (the command line, the library and a C header).
  docs/embedded-edition.md.
• File format 1, a portable format for lossless waveforms (#351): docs/file-format.md
  specifies the file (tables, meaning, rules) with a compatibility promise. docs/chunk-format.md
  now specifies decoding exactly (bit stream, prediction, integer mappings, chunk summaries).
  reference/python/morletfile.py reads the format with the Python standard library alone. CI checks it and the
  library against a committed corpus covering every chunk kind (testdata/morlet-file-1), and against fresh files
  from the library.
• Exact times at slow rates (#343): converting a time to a sample (morlet_time_to_samples) and the time filters,
  before_only cutoffs, hit times of search and acquisition alignment computed with sample_rate_hz::numeric,
  which rounds a float8 to 15 significant digits. At 1/60 Hz (0.0166666666666667, above 1/60) a time exactly at a
  sample instant mapped to the next sample; about half of all whole-second periods were affected. A rate is now read
  as the exact fraction it stands for (morlet_rate_ratio: 1/60, 1/7, 25600/1, or its exact decimal) and sample
  boundaries are computed in integer arithmetic. Found by the L1 limits study; a regression test checks 2,001
  sample instants at periods of 1, 3, 6, 7 and 60 s; checked once over every whole-second period from 1 to 3,600 s.
  Rates such as 25.6 kHz and 64 kHz map exactly as before. morlet_time_to_samples costs about 50 µs per call
  instead of 22; where it runs per row (time filters, before_only) the fraction is computed once per query.
• Interval statistics on very long recordings (#340): on a recording of 10^9 samples, statistics over 10^6
  samples took 5.4–6.1 ms instead of about 0.6 ms. The query itself took 0.1 ms; the rest was PostgreSQL compiling
  it (JIT), because the planner's row estimate, and so its cost, grew with the table until it crossed
  jit_above_cost. The general path (morlet_interval_buckets, behind morlet_interval_stats for long intervals and
  morlet_interval_overview) now runs with jit=off and takes 0.6–0.8 ms there. The fast path (#348, #349) already
  avoided it for intervals below 2^20 samples.
• Faster interval statistics (#346): morlet_interval_stats over an interval shorter than a roll-up block
  (2^20 samples) on a collection without memory levels now runs the chunk aggregate directly, through a new internal
  Rust function (morlet_interval_stats_fast) that resolves the collection's tables through the system caches and
  keeps its prepared query per session: no catalog query and no planning per call. The same rows in the same order,
  so the same result bit for bit (tested over 305 intervals against the general path, and after new samples and
  enabled summaries). Release image, 128,000-sample intervals, alternating rounds on a loaded development machine:
  server time 113 µs against 302 µs in 0.3.9-alpha.1; end to end p50 0.41 ms against 0.62 ms (a SELECT 1 round
  trip was 0.28 ms). Other requests take the general path as before.

0.3.9-alpha.1

SQL version 0.3.7, with an update script from 0.3.6. Python SDK unchanged (0.1.0a2). The chunk format is unchanged;
existing collections keep their layout.

• Extensions on demand (#315): pg_morlet can be loaded from a directory outside the server's installation, such
  as an image volume, on PostgreSQL 18 and later: extension_control_path names its control and SQL files and
  dynamic_library_path its library. The control file's ...

MorletDB technical report 0.23

MorletDB: Recordings, Not Rows — Lag-Tolerant Waveform Retrieval in PostgreSQL, version 0.23.
Technical report / preprint by Zin Nam and Semir Maslo (Sulzer-BDiM). It has not been peer reviewed. 0.22 (Grizzbit naming only) was prepared but never tagged, so its change is listed here too.

What changed in 0.23

• Second author: Semir Maslo (Sulzer-BDiM).
• New section "Industrial use" with the study IU1 on the public Bosch CNC milling data (CC BY 4.0). The protocol, scripts and results are in evaluation/industrial_iu1/, together with two post-hoc addenda whose rules were written before they were computed: A1 (drift between half-years) and A2 (lead-in).
  • Scoring runs: each run is scored against a good history of the same operation. AUROC was 0.925 with bounded lag, 0.875 with zero lag and 0.852 with RMS.
  • Drift (A1): with a history from the same half-year, the AUROC was 0.933, and 0.905 within machine and half-year.
  • Lead-in (A2): without each run's first two seconds, it was 0.900.
  • Limits, reported openly:
    • a history from earlier half-years only did not separate the classes (0.603);
    • run length alone separated better than any signal score (0.982);
    • retrieval of other bad runs was inflated by the half-year.
  • Conclusion: bounded lag beat zero lag in every comparison. As a detector, level values and run length did as well or better.
• Write-time verification: from release 0.3.11-alpha.1, every append path decodes each chunk again and refuses one that does not reproduce its input bit for bit.
• AI statement covers both authors.

What changed in 0.22

• The chunk codec is named Grizzbit (Grizzbit 1 = chunk format versions 2 and 3). No other change.

Files

• morletdb-v0.23.pdf: the report.
• morletdb-supplement-v0.23.pdf: the supplement.
• RELEASE_MANIFEST.json: version, commits and the PDFs' SHA-256 sums. Verify with python3 research/paper/release_manifest.py --check in a checkout of this tag.

Provenance and licence

• Measured software: E1 and D1–D2 ran on 68594d7, library SHA-256 3c786684…. Later studies ran on the implementation states named in research/paper/STUDIES.md. IU1 ran on 088f03b.
• Report sources: commit 8875aa6. The tag points at its merge into main (a5b51d2).
• Licence: text and figures are CC BY 4.0; code is Apache-2.0. Copyright Zin Nam.
• Funding: none.
• AI assistance: developed with Claude (Anthropic) under the authors' direction; the section "Industrial use" and IU1 were also written with AI assistance. The authors are responsible for the content.

MorletDB 0.3.10-alpha.1

Changelog

0.3.10-alpha.1

SQL version 0.3.8, with an update script from 0.3.7. Python SDK 0.1.0a3. The chunk format is unchanged;
existing collections keep their layout. New: the embedded edition (experimental) and file format 1.

• Embedded edition, experimental (#351): MorletDB in one SQLite file, with no server. Implemented in
  crates/morlet-sqlite, with SQLite included, as a Rust library, a C interface and the morlet-sqlite command line.
  In Python: morletdb.open("x.morlet"), which offers ingest, append, read and stats. Chunks and summaries are byte
  for byte pg_morlet's, so interval statistics are the same bit for bit (cross-checked against pg_morlet). Client.import_file and Client.export_file move recordings between a file and a server
  chunk for chunk, checking the chunk bytes on both sides. The SDK's platform wheels carry the library, and the
  release attaches morlet-sqlite-<platform>.tar.gz (the command line, the library and a C header).
  docs/embedded-edition.md.
• File format 1, a portable format for lossless waveforms (#351): docs/file-format.md
  specifies the file (tables, meaning, rules) with a compatibility promise. docs/chunk-format.md
  now specifies decoding exactly (bit stream, prediction, integer mappings, chunk summaries).
  reference/python/morletfile.py reads the format with the Python standard library alone. CI checks it and the
  library against a committed corpus covering every chunk kind (testdata/morlet-file-1), and against fresh files
  from the library.
• Exact times at slow rates (#343): converting a time to a sample (morlet_time_to_samples) and the time filters,
  before_only cutoffs, hit times of search and acquisition alignment computed with sample_rate_hz::numeric,
  which rounds a float8 to 15 significant digits. At 1/60 Hz (0.0166666666666667, above 1/60) a time exactly at a
  sample instant mapped to the next sample; about half of all whole-second periods were affected. A rate is now read
  as the exact fraction it stands for (morlet_rate_ratio: 1/60, 1/7, 25600/1, or its exact decimal) and sample
  boundaries are computed in integer arithmetic. Found by the L1 limits study; a regression test checks 2,001
  sample instants at periods of 1, 3, 6, 7 and 60 s; checked once over every whole-second period from 1 to 3,600 s.
  Rates such as 25.6 kHz and 64 kHz map exactly as before. morlet_time_to_samples costs about 50 µs per call
  instead of 22; where it runs per row (time filters, before_only) the fraction is computed once per query.
• Interval statistics on very long recordings (#340): on a recording of 10^9 samples, statistics over 10^6
  samples took 5.4–6.1 ms instead of about 0.6 ms. The query itself took 0.1 ms; the rest was PostgreSQL compiling
  it (JIT), because the planner's row estimate, and so its cost, grew with the table until it crossed
  jit_above_cost. The general path (morlet_interval_buckets, behind morlet_interval_stats for long intervals and
  morlet_interval_overview) now runs with jit=off and takes 0.6–0.8 ms there. The fast path (#348, #349) already
  avoided it for intervals below 2^20 samples.
• Faster interval statistics (#346): morlet_interval_stats over an interval shorter than a roll-up block
  (2^20 samples) on a collection without memory levels now runs the chunk aggregate directly, through a new internal
  Rust function (morlet_interval_stats_fast) that resolves the collection's tables through the system caches and
  keeps its prepared query per session: no catalog query and no planning per call. The same rows in the same order,
  so the same result bit for bit (tested over 305 intervals against the general path, and after new samples and
  enabled summaries). Release image, 128,000-sample intervals, alternating rounds on a loaded development machine:
  server time 113 µs against 302 µs in 0.3.9-alpha.1; end to end p50 0.41 ms against 0.62 ms (a SELECT 1 round
  trip was 0.28 ms). Other requests take the general path as before.

0.3.9-alpha.1

SQL version 0.3.7, with an update script from 0.3.6. Python SDK unchanged (0.1.0a2). The chunk format is unchanged;
existing collections keep their layout.

• Extensions on demand (#315): pg_morlet can be loaded from a directory outside the server's installation, such
  as an image volume, on PostgreSQL 18 and later: extension_control_path names its control and SQL files and
  dynamic_library_path its library. The control file's module_pathname is now pg_morlet instead of
  $libdir/pg_morlet, so the C functions find the library through dynamic_library_path (default $libdir:
  ordinary installations are unchanged). The update from 0.3.6 redefines the existing C functions with the new path
  (CREATE OR REPLACE, same OIDs; no data or table changes). Tested in CI on PostgreSQL 18 and 19
  (tests/on_demand.sh): the server's own copy is moved aside, and CREATE EXTENSION, an append, a read and
  interval statistics work from the separate directory.
• Extension-only image (#315): each release publishes ghcr.io/merkhet-io/pg_morlet:<version>-pg18 (amd64,
  arm64): pg_morlet's control, SQL and library files and nothing else, under /share/extension and /lib, with build
  provenance. Mount it as an image volume (or copy its files) and set the two paths above. The release workflow
  builds it from the Debian package's build stage and trials it mounted into a clean postgres:18 image
  (deploy/extension_trial.sh: install, append, read, statistics, restart) before pushing. PostgreSQL 19 follows
  19.0.

0.3.8-alpha.1

SQL version 0.3.6, with an update script from 0.3.5. Python SDK 0.1.0a2. The chunk format is unchanged; existing
collections keep their layout (the update pins their bulk columns to pglz, catalog only).

• Ingest hold and the steward (#287): tools/steward keeps an edge database healthy without a DBA.

  • It runs the rebuildable upkeep (envelopes, summaries, roll-ups, context values, verify, analyze) in one budgeted
    loop, gated by load and the disk.
  • It records every action and shows one plain status.
  • At the disk reserve it sets morlet_ingest_hold: appends that would store new samples then fail with SQLSTATE
    53100 and the reason, before PostgreSQL itself runs out of disk. It releases the hold when space is free again.
  • Its role cannot remove stored samples.
  • Changes to stored samples and extension updates are plans: proposed by the steward or a person, approved by
    another person with morlet_approver, and applied by a person in one transaction. Rules: no self-approval, the
    payload tied to the approved hash, 24-hour expiry, applied once; enforced in SQL.

• Synchronous channel groups (#292): an acquisition sampled at the same instants (one DAQ, one rate) can be
  declared synchronous (morlet_enable_sync, morlet_declare_sync). Its members are then:

  • appended together in one transaction (morlet_append_acquisition_raw);
  • read by sample index, failing closed (morlet_read_sync_raw);
  • summarised exactly across channels (morlet_acquisition_cross_stats: mean(a·b), such as real power, bitwise the
    same for any chunking).

  Single-member appends, partial memory moves and new channels are refused, and morlet_verify reports misaligned
  groups. See docs/context-workflow.md.

• Exact search is 10x faster on small machines: prepared query matrices (exact scoring and screening) are
  used only when they fit a quarter of the CPU's last-level cache; otherwise windows are scored directly, with
  bitwise the same distances.

  • On a Raspberry Pi 3B+, an N1 query (1,341 windows of 16,384 samples) drops from 15.1 s to 1.5 s.
  • Desktops are unchanged.
  • Override with MORLET_SCORE_CACHE_BYTES.
  • See evaluation/edge_latency.

• Commit notifications (#205): morlet_enable_notifications(collection, channel, detail) sends one
  pg_notify per committed transaction that stored new chunks, with the stored sample ranges per recording
  (payload version 1). Rolled-back transactions, savepoints and identical replays send nothing.

  • detail => false sends counts only; row security requires it.
  • Python SDK: Client.enable_notifications, Client.notifications. The SDK now needs psycopg 3.2.
  • Tested with a real listener session (tests/notifications.sh).

• PostgreSQL 19 (#196, #280): pg_morlet builds and passes its full test suite on PostgreSQL 19 (release
  candidate 19rc1; CI with and without pgvector 0.8.7), with pgrx 0.19.3 and Rust 1.99.0 (from 0.16.1 and 1.91.1).

  • PostgreSQL 19 defaults to lz4 TOAST compression. Every column holding samples, or values derived from them, now
    names pglz, and the update pins the columns of existing collections. morlet_pin_toast_compression(collection)
    does the same for one collection (catalog only).
  • On the B3 set, PostgreSQL 19 stores exactly the bytes PostgreSQL 17 does
    (evaluation/toast_compression). Unpinned, lz4 would change encoded
    chunks by 0.016% and legacy float8[] chunks by −19%.
  • Release images and Debian packages for 19 follow 19.0.

• History counts independent observations (#224): morlet_compare_to_history, morlet_compare_summary,
  morlet_history_neighbors, morlet_level_track and the pooled spreads take history only from segments built with
  the query segment's segmentation spec (keys, occurrence keys, min_samples, trim_samples), and never from a
  segment that overlaps the query segment's own samples. Before, building one interval's segments with several
  settings made the interval count as its own history, so a comparison could report status = ok without any
  independent observation. History built with oth...

MorletDB 0.3.9-alpha.1

Changelog

0.3.9-alpha.1

SQL version 0.3.7, with an update script from 0.3.6. Python SDK unchanged (0.1.0a2). The chunk format is unchanged;
existing collections keep their layout.

• Extensions on demand (#315): pg_morlet can be loaded from a directory outside the server's installation, such
  as an image volume, on PostgreSQL 18 and later: extension_control_path names its control and SQL files and
  dynamic_library_path its library. The control file's module_pathname is now pg_morlet instead of
  $libdir/pg_morlet, so the C functions find the library through dynamic_library_path (default $libdir:
  ordinary installations are unchanged). The update from 0.3.6 redefines the existing C functions with the new path
  (CREATE OR REPLACE, same OIDs; no data or table changes). Tested in CI on PostgreSQL 18 and 19
  (tests/on_demand.sh): the server's own copy is moved aside, and CREATE EXTENSION, an append, a read and
  interval statistics work from the separate directory.
• Extension-only image (#315): each release publishes ghcr.io/merkhet-io/pg_morlet:<version>-pg18 (amd64,
  arm64): pg_morlet's control, SQL and library files and nothing else, under /share/extension and /lib, with build
  provenance. Mount it as an image volume (or copy its files) and set the two paths above. The release workflow
  builds it from the Debian package's build stage and trials it mounted into a clean postgres:18 image
  (deploy/extension_trial.sh: install, append, read, statistics, restart) before pushing. PostgreSQL 19 follows
  19.0.

0.3.8-alpha.1

SQL version 0.3.6, with an update script from 0.3.5. Python SDK 0.1.0a2. The chunk format is unchanged; existing
collections keep their layout (the update pins their bulk columns to pglz, catalog only).

• Ingest hold and the steward (#287): tools/steward keeps an edge database healthy without a DBA.

  • It runs the rebuildable upkeep (envelopes, summaries, roll-ups, context values, verify, analyze) in one budgeted
    loop, gated by load and the disk.
  • It records every action and shows one plain status.
  • At the disk reserve it sets morlet_ingest_hold: appends that would store new samples then fail with SQLSTATE
    53100 and the reason, before PostgreSQL itself runs out of disk. It releases the hold when space is free again.
  • Its role cannot remove stored samples.
  • Changes to stored samples and extension updates are plans: proposed by the steward or a person, approved by
    another person with morlet_approver, and applied by a person in one transaction. Rules: no self-approval, the
    payload tied to the approved hash, 24-hour expiry, applied once; enforced in SQL.

• Synchronous channel groups (#292): an acquisition sampled at the same instants (one DAQ, one rate) can be
  declared synchronous (morlet_enable_sync, morlet_declare_sync). Its members are then:

  • appended together in one transaction (morlet_append_acquisition_raw);
  • read by sample index, failing closed (morlet_read_sync_raw);
  • summarised exactly across channels (morlet_acquisition_cross_stats: mean(a·b), such as real power, bitwise the
    same for any chunking).

  Single-member appends, partial memory moves and new channels are refused, and morlet_verify reports misaligned
  groups. See docs/context-workflow.md.

• Exact search is 10x faster on small machines: prepared query matrices (exact scoring and screening) are
  used only when they fit a quarter of the CPU's last-level cache; otherwise windows are scored directly, with
  bitwise the same distances.

  • On a Raspberry Pi 3B+, an N1 query (1,341 windows of 16,384 samples) drops from 15.1 s to 1.5 s.
  • Desktops are unchanged.
  • Override with MORLET_SCORE_CACHE_BYTES.
  • See evaluation/edge_latency.

• Commit notifications (#205): morlet_enable_notifications(collection, channel, detail) sends one
  pg_notify per committed transaction that stored new chunks, with the stored sample ranges per recording
  (payload version 1). Rolled-back transactions, savepoints and identical replays send nothing.

  • detail => false sends counts only; row security requires it.
  • Python SDK: Client.enable_notifications, Client.notifications. The SDK now needs psycopg 3.2.
  • Tested with a real listener session (tests/notifications.sh).

• PostgreSQL 19 (#196, #280): pg_morlet builds and passes its full test suite on PostgreSQL 19 (release
  candidate 19rc1; CI with and without pgvector 0.8.7), with pgrx 0.19.3 and Rust 1.99.0 (from 0.16.1 and 1.91.1).

  • PostgreSQL 19 defaults to lz4 TOAST compression. Every column holding samples, or values derived from them, now
    names pglz, and the update pins the columns of existing collections. morlet_pin_toast_compression(collection)
    does the same for one collection (catalog only).
  • On the B3 set, PostgreSQL 19 stores exactly the bytes PostgreSQL 17 does
    (evaluation/toast_compression). Unpinned, lz4 would change encoded
    chunks by 0.016% and legacy float8[] chunks by −19%.
  • Release images and Debian packages for 19 follow 19.0.

• History counts independent observations (#224): morlet_compare_to_history, morlet_compare_summary,
  morlet_history_neighbors, morlet_level_track and the pooled spreads take history only from segments built with
  the query segment's segmentation spec (keys, occurrence keys, min_samples, trim_samples), and never from a
  segment that overlaps the query segment's own samples. Before, building one interval's segments with several
  settings made the interval count as its own history, so a comparison could report status = ok without any
  independent observation. History built with other settings now gives insufficient_history until it is rebuilt
  with the query's settings.

• Newest data (#206): morlet_latest(collection, n, metadata_filter, max_recordings) returns every matching
  recording's last n samples (1..32,768) with their statistics, or with n = 0 the statistics of its newest
  stored chunk, and the end time, at a fixed number of index lookups per recording (0.4–1.6 ms per recording on
  a laptop, the same for 10 and 10,000 chunks). Unreadable newest samples (compacted, or a gap) are reported in
  note without failing the call. MCP tool latest and SDK method Client.latest.

• Not found versus not searched (#215): morlet_coverage(collection, recording_ids, window_size) splits
  every recording into ranges that are searchable, not indexed, out of search by a memory level, represented,
  summarized, gone or missing. The MCP server adds a scope to every search result (recordings and windows
  searched, the periods not searched and why, event-library entries in range), coverage reports the same per
  group, and the server instructions tell agents to state the scope with every negative result.

• Search refuses over-budget queries in seconds (#279): screened exact search counts the eligible windows,
  refuses above max_candidates and only then scores the kept windows, and the screen stays linear when many
  windows tie (constant windows all have distance 1). On 2 million all-zero windows the refusal took 2 min 48 s
  and now takes 2.7 s; results and latency on E7's development gallery are unchanged.

• Idle stretches no longer slow exact search (#284): the screen gives an exactly constant window its known
  distance 1 with no error bound, instead of an infinite bound that sent every such window to exact scoring. On a
  dev gallery where half the windows are idle, the median screened search drops from 253 ms to 65 ms; results
  are bit-identical, and galleries without idle stretches are unchanged. Nearly constant windows keep the
  infinite bound. E7's frozen protocol describes the earlier rule.

• Exact roll-ups (#219): morlet_enable_rollups and the resumable morlet_build_rollups store exact merges of
  chunk summaries in blocks of 2^20, 2^25 and 2^30 samples. morlet_interval_stats and morlet_interval_overview
  take the coarsest roll-ups that fit each part of the interval (and each bucket), then finer ones and chunks at
  the edges, with bitwise the same results as from the chunks alone. On a year of one 64 kHz channel (61.6M chunks; dev machine, load about 3.5) year statistics take 17 ms instead of 173 s, 365 daily buckets 137 ms instead of 170 s, one day 9 ms instead of 0.4 s, all bit for bit equal; building the roll-ups took 11 minutes. Roll-ups survive memory levels
  2 and 3, go with the samples forgotten at level 4, follow row security, and are skipped for a reader who may not
  read them.

• Interval statistics merge exactly (#219): each chunk's part is still summed with compensation, but the parts
  are now merged in exact arithmetic and rounded once (morlet_core::exactstats), so results no longer depend on
  how an interval is split or in which order pieces are merged. Mean and RMS over more than one chunk can differ
  from 0.3.5 in the last bits; the mean of a single chunk is unchanged. Stored chunk summaries stay valid.

• Tiny signals (#228): normalization, distances, RMS and envelopes keep their shape for amplitudes down to
  1e-300; ordinary inputs give bit-identical results.

0.3.7-alpha.1

SQL version 0.3.5, with an update script from 0.3.4.

• Integrity scrub (#193): morlet_verify(collection, after_recording, max_recordings) re-reads every chunk of a
  batch of recordings and reports, per recording, every problem it finds without changing data: a chunk that does
  not decode (structure, CRC-32C) to its sample_count, a missing or wrong chunk summary, gaps or overlaps between
  chunks, and chunks that do not end at the checkpoint. Helper morlet_check_chunk. Command-line wrapper
  tools/verify/morlet_verify.py (batches in read-o...

MorletDB 0.3.8-alpha.1

Changelog

0.3.8-alpha.1

SQL version 0.3.6, with an update script from 0.3.5. Python SDK 0.1.0a2. The chunk format is unchanged; existing
collections keep their layout (the update pins their bulk columns to pglz, catalog only).

• Ingest hold and the steward (#287): tools/steward keeps an edge database healthy without a DBA.

  • It runs the rebuildable upkeep (envelopes, summaries, roll-ups, context values, verify, analyze) in one budgeted
    loop, gated by load and the disk.
  • It records every action and shows one plain status.
  • At the disk reserve it sets morlet_ingest_hold: appends that would store new samples then fail with SQLSTATE
    53100 and the reason, before PostgreSQL itself runs out of disk. It releases the hold when space is free again.
  • Its role cannot remove stored samples.
  • Changes to stored samples and extension updates are plans: proposed by the steward or a person, approved by
    another person with morlet_approver, and applied by a person in one transaction. Rules: no self-approval, the
    payload tied to the approved hash, 24-hour expiry, applied once; enforced in SQL.

• Synchronous channel groups (#292): an acquisition sampled at the same instants (one DAQ, one rate) can be
  declared synchronous (morlet_enable_sync, morlet_declare_sync). Its members are then:

  • appended together in one transaction (morlet_append_acquisition_raw);
  • read by sample index, failing closed (morlet_read_sync_raw);
  • summarised exactly across channels (morlet_acquisition_cross_stats: mean(a·b), such as real power, bitwise the
    same for any chunking).

  Single-member appends, partial memory moves and new channels are refused, and morlet_verify reports misaligned
  groups. See docs/context-workflow.md.

• Exact search is 10x faster on small machines: prepared query matrices (exact scoring and screening) are
  used only when they fit a quarter of the CPU's last-level cache; otherwise windows are scored directly, with
  bitwise the same distances.

  • On a Raspberry Pi 3B+, an N1 query (1,341 windows of 16,384 samples) drops from 15.1 s to 1.5 s.
  • Desktops are unchanged.
  • Override with MORLET_SCORE_CACHE_BYTES.
  • See evaluation/edge_latency.

• Commit notifications (#205): morlet_enable_notifications(collection, channel, detail) sends one
  pg_notify per committed transaction that stored new chunks, with the stored sample ranges per recording
  (payload version 1). Rolled-back transactions, savepoints and identical replays send nothing.

  • detail => false sends counts only; row security requires it.
  • Python SDK: Client.enable_notifications, Client.notifications. The SDK now needs psycopg 3.2.
  • Tested with a real listener session (tests/notifications.sh).

• PostgreSQL 19 (#196, #280): pg_morlet builds and passes its full test suite on PostgreSQL 19 (release
  candidate 19rc1; CI with and without pgvector 0.8.7), with pgrx 0.19.3 and Rust 1.99.0 (from 0.16.1 and 1.91.1).

  • PostgreSQL 19 defaults to lz4 TOAST compression. Every column holding samples, or values derived from them, now
    names pglz, and the update pins the columns of existing collections. morlet_pin_toast_compression(collection)
    does the same for one collection (catalog only).
  • On the B3 set, PostgreSQL 19 stores exactly the bytes PostgreSQL 17 does
    (evaluation/toast_compression). Unpinned, lz4 would change encoded
    chunks by 0.016% and legacy float8[] chunks by −19%.
  • Release images and Debian packages for 19 follow 19.0.

• History counts independent observations (#224): morlet_compare_to_history, morlet_compare_summary,
  morlet_history_neighbors, morlet_level_track and the pooled spreads take history only from segments built with
  the query segment's segmentation spec (keys, occurrence keys, min_samples, trim_samples), and never from a
  segment that overlaps the query segment's own samples. Before, building one interval's segments with several
  settings made the interval count as its own history, so a comparison could report status = ok without any
  independent observation. History built with other settings now gives insufficient_history until it is rebuilt
  with the query's settings.

• Newest data (#206): morlet_latest(collection, n, metadata_filter, max_recordings) returns every matching
  recording's last n samples (1..32,768) with their statistics, or with n = 0 the statistics of its newest
  stored chunk, and the end time, at a fixed number of index lookups per recording (0.4–1.6 ms per recording on
  a laptop, the same for 10 and 10,000 chunks). Unreadable newest samples (compacted, or a gap) are reported in
  note without failing the call. MCP tool latest and SDK method Client.latest.

• Not found versus not searched (#215): morlet_coverage(collection, recording_ids, window_size) splits
  every recording into ranges that are searchable, not indexed, out of search by a memory level, represented,
  summarized, gone or missing. The MCP server adds a scope to every search result (recordings and windows
  searched, the periods not searched and why, event-library entries in range), coverage reports the same per
  group, and the server instructions tell agents to state the scope with every negative result.

• Search refuses over-budget queries in seconds (#279): screened exact search counts the eligible windows,
  refuses above max_candidates and only then scores the kept windows, and the screen stays linear when many
  windows tie (constant windows all have distance 1). On 2 million all-zero windows the refusal took 2 min 48 s
  and now takes 2.7 s; results and latency on E7's development gallery are unchanged.

• Idle stretches no longer slow exact search (#284): the screen gives an exactly constant window its known
  distance 1 with no error bound, instead of an infinite bound that sent every such window to exact scoring. On a
  dev gallery where half the windows are idle, the median screened search drops from 253 ms to 65 ms; results
  are bit-identical, and galleries without idle stretches are unchanged. Nearly constant windows keep the
  infinite bound. E7's frozen protocol describes the earlier rule.

• Exact roll-ups (#219): morlet_enable_rollups and the resumable morlet_build_rollups store exact merges of
  chunk summaries in blocks of 2^20, 2^25 and 2^30 samples. morlet_interval_stats and morlet_interval_overview
  take the coarsest roll-ups that fit each part of the interval (and each bucket), then finer ones and chunks at
  the edges, with bitwise the same results as from the chunks alone. On a year of one 64 kHz channel (61.6M chunks; dev machine, load about 3.5) year statistics take 17 ms instead of 173 s, 365 daily buckets 137 ms instead of 170 s, one day 9 ms instead of 0.4 s, all bit for bit equal; building the roll-ups took 11 minutes. Roll-ups survive memory levels
  2 and 3, go with the samples forgotten at level 4, follow row security, and are skipped for a reader who may not
  read them.

• Interval statistics merge exactly (#219): each chunk's part is still summed with compensation, but the parts
  are now merged in exact arithmetic and rounded once (morlet_core::exactstats), so results no longer depend on
  how an interval is split or in which order pieces are merged. Mean and RMS over more than one chunk can differ
  from 0.3.5 in the last bits; the mean of a single chunk is unchanged. Stored chunk summaries stay valid.

• Tiny signals (#228): normalization, distances, RMS and envelopes keep their shape for amplitudes down to
  1e-300; ordinary inputs give bit-identical results.

0.3.7-alpha.1

SQL version 0.3.5, with an update script from 0.3.4.

• Integrity scrub (#193): morlet_verify(collection, after_recording, max_recordings) re-reads every chunk of a
  batch of recordings and reports, per recording, every problem it finds without changing data: a chunk that does
  not decode (structure, CRC-32C) to its sample_count, a missing or wrong chunk summary, gaps or overlaps between
  chunks, and chunks that do not end at the checkpoint. Helper morlet_check_chunk. Command-line wrapper
  tools/verify/morlet_verify.py (batches in read-only transactions, optional pause, exit status 1 on problems).
  Operations docs explain how it complements PostgreSQL's data checksums.
• Episodic memory, phase 1 (#221, design): morlet_enable_memory adds episodes, runs, a
  ledger and kept summaries to a collection; morlet_add_episodes registers sample intervals with context,
  importance and an exemption flag; morlet_set_level moves episodes down the compaction ladder (1 not searchable,
  2 represented by another episode, 3 summarized, 4 gone) in one transaction, with a dry run, removing only rows
  entirely inside the episode that no overlapping episode at a lower level needs, and writing one ledger row per
  move (what was removed, md5 of the removed chunks). Reads of compacted samples fail with the level and ledger
  entry; interval statistics use kept summaries and stay bitwise exact on chunk boundaries; the envelope builder
  skips episodes taken out of search; morlet_verify accepts compacted data. morlet_memory_status reports
  episodes per level. Policies that decide the moves come in phase 2.
• Compressed transfer (design): morlet_read_interval_encoded returns an
  interval's stored chunks unchanged in one frame, under the same visibility, checkpoint and coverage checks as
  morlet_read_interval_raw, without decoding them (helper morlet_frame_chunks); chunks of a format version the
  client does not decode are sent as samples instead. New crate crates/morlet-ffi: the chunk decoder behind a C
  interface...