Skip to content

v0.13.0

Latest

Choose a tag to compare

@mrDlef mrDlef released this 27 Aug 04:41
· 2 commits to main since this release

the parameters beside body, a record that may leave the building, an index
name only you can read and one no longer half-collapsed, a third way in at the
transport that is now checked against the clients themselves, one file to
download, and the examples nothing read

Fingerprints: q4:q5: — a search whose size, from or sort sits
beside body rather than inside it now says so. Only the prefix moved: all
eighteen fixtures kept their twelve hex characters.

The search parameters an envelope carries beside body

['index' => …, 'body' => …] is how both clients take a search, and size,
from and sort are as legitimate beside body as inside it: they are three of
the fifty-three parameters the search endpoint whitelists, where they travel as
query string rather than as JSON. The parser descended into body and dropped
everything next to it, so these two produced the same fingerprint:

$f->describe(['index' => 'members', 'body' => $body, 'size' => 20, 'from' => 40]);
$f->describe(['index' => 'members', 'body' => $body]);

They are not two spellings of one search. One pages forty documents deep, the
other takes the default ten — and sharing a hash meant "which shape is hurting
us" answered with a shape that has no size in it, while the deep-paging one,
usually the expensive one, was invisible.

Both are read now, and the conflict rules are the cluster's rather than the
plausible ones:

  • size and from: the envelope wins. The cluster parses the body and
    then applies the query string, so the outer value overrides the inner one —
    the opposite of what the nesting suggests.
  • sort: the envelope is appended. The query string sorts are added after
    the body's rather than replacing them, so the body keeps the primary key.

Neither rule came from the clients' documentation. Both were read off a live
node — a thirty-document index, size and from and sort set to different
values in the two places, and the hits counted — and
tests/fixtures/19-envelope-search-params pins the answer.

One consequence worth naming: an envelope sort is a query string parameter, so
it carries the URI syntax — last_name:desc, comma-joined — not the body's
structural form. Read as a body sort it would have minted a field called
last_name:desc and claimed ascending.

A digest that carries no search input

Digest::toArray() always emitted four fields, and q is the readable line —
which is user-typed search input: names, addresses, e-mail addresses. The moment
those records leave for a hosted log collector or a third-party SIEM, that is the
difference between "we may ship these logs" and "we may not", and it is decided
per field.

Formatter::create(Options::create()->withText(false));
// {"idx": "logs-*", "sig": "logs-* | q=(@timestamp >= ? and service:?) | size=0", "hash": "q5:…"}

Everything the library is for reads sig and hash, which were already
value-free: which shape got slow, which one the deploy added, which one to group
a dashboard by. q is the convenience of pasting into Dashboards, and it is the
first thing a regulated deployment gives up. It is also the longest of the four
fields, so this is the cheapest log-volume win on offer.

Four things it was worth being careful about:

  • The line is never rendered, not rendered and dropped. A blanket redactor
    came close before this — withRedactor(fn ($field, $value) => '?') — but it
    renders the same line twice to throw one away, and a per-field redactor is
    one forgotten field away from a leak. There is now no literal anywhere in the
    digest, and Digest::text() returns the signature rather than an empty string
    so that nothing reading the wrong accessor, including __toString(), can find
    one either.
  • q is omitted, not emptied. A q that duplicated sig would still have
    to be inspected before those logs could ship, which is not what "decided per
    field" means.
  • The hash does not move. What is emitted beside a fingerprint does not
    change what the shape is called, so a dashboard built before the switch keeps
    matching after it. Asserted, not assumed.
  • It is not on its own a promise that no literal is emitted, and the
    docblock, the guide and a test all say so: under Normalization::none() the
    signature is the readable line. The pair that emits none is withText(false)
    with any normalization above none — which is the default.

One processor and one observer, unchanged: both emit toArray(), so both follow.
The shipped dashboard pack needs no change either — its panels group on
os.hash and read os.sig, and os.q appears only in the index pattern's field
list.

text is a fromArray() key like the rest, since this is a decision a
deployment makes in configuration rather than in code, and the playground gained
the toggle — where turning it on removes the text row from the result rather
than blanking it, which is what the record does.

A numeric index segment no longer collapses halfway

datePatterns() collapsed a standalone numeric segment up to eight digits and
then, from nine, only part of it:

 8 digits  logs-99999999     →  logs-*
 9 digits  logs-999999999    →  logs-*9
10 digits  logs-9999999999   →  logs-*99

The leftover is the digits past the first eight, so it varied with the value.
An epoch-seconds suffix is ten digits, which made every rollover its own
fingerprint:

logs-1756259400  →  logs-*00     q4:ebd17811af9b
logs-1756259412  →  logs-*12     q4:228ebfe1e5b7
logs-1756345837  →  logs-*37     q4:a908898fa64a

Three names of one query shape, three fingerprints — precisely the failure this
class exists to prevent, while the dated form right beside it did the right
thing. And logs-*99 reads as a pattern, so nothing in a dashboard suggested
the grouping was broken. A name that looks collapsed is worse than one that
plainly is not.

The cause is two rules and their order. The date rule matches eight bare digits,
because logs-20260813 is a real index name; it ran first, ate the leading eight
of a longer run, and left *99 — which the numeric-segment rule can no longer
match, being anchored to segment boundaries.

The fix is two lookarounds, and the obvious guard was the wrong one. Anchoring
the date rule to segment boundaries fixes the digit run and breaks a date with a
time on it: orders-2026.08.13T00 would collapse to orders-*.13T00, worse than
before. Bounding it by digits instead — (?<!\d)…(?!\d) — refuses the front of
a longer run while still accepting eight digits followed by anything that is not
one:

logs-999999999        →  logs-*        (was logs-*9)
logs-1756259400       →  logs-*        (was logs-*00)
logs-000000001        →  logs-*        (was logs-*1)
logs-20260813T00      →  logs-*T00     unchanged
orders-2026.08.13T00  →  orders-*T00   unchanged

No fixture moved. All eighteen keep their hashes, because no pinned example
carried a segment of nine digits or more. Two things did move and had to: the
block in the options guide that documented the mangling, and the test that pinned
it — both now say the suffix is left alone, which was always the argument for
custom() being a callable.

This is why the fix is in this release rather than the next one: v0.13.0 already
moves the prefix, so it rides along inside that bump. After the tag it would have
cost a q6: of its own, for two lookarounds.

An index name only you can read

IndexNormalizer::datePatterns() collapses what any cluster does — dates, and
standalone numeric segments, which covers rolling indices and, pleasantly,
multi-tenant numeric prefixes. What it cannot collapse is a suffix whose meaning
is yours: a content-versioned index, where the physical name carries a hash
of the mapping and the alias moves over it on reindex. Every mapping change then
minted a fresh fingerprint for every query shape, and every dashboard built on
the hash reset on the next deploy — the thing the class exists to prevent.

Options::create()->withIndexNormalizer(IndexNormalizer::custom(
    fn (string $index): string => preg_replace('/_[0-9a-f]{32}$/', '', $index),
));

// tenant_0178_members_4f171971a955af948fae1c7a964c49b8  →  tenant_*_members

A callable, not a third mode. A mode that collapsed long hex runs would have to
decide what a hash is — how long, which alphabet — and would move the
fingerprint of every index name with hex anywhere in it. The shipped rules stop
where the cluster's own conventions stop: the tenant number is a number and
collapses, and the suffix is left alone whatever it is made of. Only the
application knows where its own suffix begins.

Three decisions inside the hook:

  • Your rule runs first, then the shipped one. So the example lands on
    tenant_*_members rather than tenant_0178_members: the hook strips what this library
    cannot know is meaningless, and dates and numbers are collapsed afterwards as
    always. Nobody reimplements what already works.
  • It is called once per name. normalize() also splits a comma-separated
    list, deduplicates it and sorts it — machinery, not policy — so a request
    against a,b gets two calls and that part stays where it is. A rule may return
    '' to drop one name from the list.
  • It is not trusted to return a string. Anything non-scalar reads as an
    erased name rather than throwing, the same trade the redactor makes: this runs
    in a logging path, where a TypeError out of a closure would cost the log line
    and not just the digest.

Like the redactor, it has no fromArray() key and no MODES entry — a callable
cannot come out of a configuration file — so fromMode('custom') throws, and the
playground's mode list is unchanged. Nothing moves for anyone who does not ask:
the default is the same rule it was.

Which clients can be captured is a test now, not a paragraph

Every other claim this library makes about the outside world is checked against
it: which OpenSearch versions accept what we render is a committed matrix
replayed on real nodes, which query types exist is a snapshot of the official
specification, and that took sits at the front of a response is asserted rather
than trusted. Which clients the transport integrations can capture was prose —
and it was wrong in one direction and then wrong in the other inside this same
release, without anything failing either time.

tests/Integration/ClientCaptureTest.php installs the clients themselves and
sends a search through each one against a live node:

  • opensearch-php ≥ 2.4 through TransportFactory, via the decorator — a
    _search and an _msearch, three observations, took on the single one and
    null on both batch lines.
  • The same client through GuzzleClientFactory, via the middleware, using
    its middleware option.
  • elasticsearch-php 7.17 over ringphp, via the ring handler — plus an
    indices()->stats() in the same run, which must not be counted.
  • opensearch-php's own deprecated ClientBuilder, which skips itself with a
    note when the class finally goes in 3.0.0 — the notice the coverage table needs.
  • A ringphp client still offers neither seam: defaultHandler() is a
    Closure and the client is not a Psr\Http\Client\ClientInterface. If either
    changes, the reason the ring handler exists changes with it.

And the one no single integration's own tests can show: the same search sent by
different clients through different integrations is one fingerprint
, equal to
the one Formatter::describe() gives for the same body and index. Two
integrations disagreeing about one query would otherwise be invisible.

The clients are in tools/clients/composer.json rather than the root
require-dev, on the pattern tools/infection already set: opensearch-php
requires PHP 8.2 and this library's floor is 7.4, so the root tree would stop
resolving on three of the seven versions in the matrix. The test skips without
that tree, so no ordinary job has to know it exists — the certification workflow
installs it, and make clients does locally.

That manifest is unlocked on purpose, unlike Infection's. The point of the
harness is to notice when a client changes what it transports over, and a pinned
lock would hide exactly that.

One row cannot be checked and now says so: elasticsearch-php 8 cannot talk to
an OpenSearch cluster at all.
It sends
Content-Type: application/vnd.elasticsearch+json; compatible-with=8 and the
node answers 406 Not Acceptable — which is why opensearch-php was forked. The
row describes the shape of its transport, not a combination anyone can run, and
the page carries a footnote saying so.

A third way in at the transport, for the clients the other two cannot reach

Both transport integrations needed a PSR-18 client or a Guzzle handler stack.
elasticsearch-php 7.x and opensearch-php ≤ 2.3 offer neither: they transport
over ezimuel/ringphp, which predates PSR-7. So "wrap the client your OpenSearch
library already uses" meant "unless it is the version you are probably running".

Http\Ring\DigestingHandler is a ring handler that wraps a ring handler:

$client = ClientBuilder::create()
    ->setHandler(new DigestingHandler(
        ClientBuilder::defaultHandler(),
        new LoggingObserver($logger),
    ))
    ->build();

The request array was already the shape the capture wanted — http_method,
uri, and a body that is a JSON string on both of those clients — so the
adapter is thin. What was not thin is everything the format allows around it,
and four things had to be read off a running node rather than assumed:

  • Every handler returns a future, the synchronous one included.
    CurlHandler hands back a CompletedFutureArray, not an array. So the future
    path is the normal path, not the edge case.
  • Reading a key off an unresolved future blocks on it. A middleware that
    read the status to report it would turn a pool of asynchronous searches into a
    serial one. The response is wrapped with Core::proxy() and read inside the
    callback, which is also the honest place to read took: after the answer has
    landed. A test asserts that the future is not dereferenced, against a real
    CurlMultiHandler.
  • A ringphp failure is a fulfilled response, not an exception. A transport
    error comes back as an array carrying error and no status. Reported the way
    a failed PSR-18 request is — counted, with a null status — because the shape
    that times out is the shape worth finding.
  • The response body is a resource. Core::body() would have read it, and
    left the pointer at the end: fine for ringphp, which rewinds afterwards, fatal
    for a middleware, where the client reads next. So took is peeked with the
    position restored, and the integration test reads the body after the handler
    did and asserts the hits are still there.

Recorder grew a ring-shaped pair of edges beside its PSR-7 ones; the pairing,
the timing and the reporting are still written once. SearchExtractor stays
@internal — the issue offered making it public as the smaller alternative, and
that is no longer the trade.

ezimuel/ringphp is a fourth suggested dependency, required by nothing. It
installs on PHP 7.4, so the ring path is tested on the whole matrix rather than
behind a version gate.

The public surface is twenty classes now, and
docs/guides/transport.md no longer tells a ringphp reader that nothing covers
them — which it correctly did, for one release.

The transport guide sent the official client to the wrong integration

Under Which one to use it read: "opensearch-php, or your own Guzzle
client → the middleware." The reasoning above it was that opensearch-php sends
asynchronously and never calls sendRequest(), so the PSR-18 decorator cannot
see it. Both halves were wrong, and the conclusion did not follow from either:
the Guzzle middleware cannot see a ringphp handler any more than the decorator
can — there is no stack to push onto. A reader on the official client followed
the table, pushed the middleware, saw no digests, and had no way to tell whether
they had wired it wrong or the library did not cover them.

What is actually true, read off the packages and then run against a live node:

  • opensearch-php ≥ 2.4 is a PSR-18 client. HttpTransport::sendRequest()
    calls $client->sendRequest($request) synchronously. The decorator sees it,
    and so does the middleware — GuzzleClientFactory takes a middleware option.
    Both were run on 2.6.0 against a 2.x node and mint the same fingerprint for the
    same search, _msearch split per line included.
  • The ringphp transport is the thing neither can reach, and asynchrony is not
    why. ezimuel/ringphp predates PSR-7: a handler is a
    callable(array): array|FutureArrayInterface, so there is no request object to
    intercept. That covers opensearch-php ≤ 2.3, opensearch-php built through
    ClientBuilder — deprecated in 2.4.0, removed in 3.0.0 — and
    elasticsearch-php 7.x, which is still a common way to reach an OpenSearch
    cluster. #42 tracks
    closing it.

The guide now names what each integration attaches to rather than which
packages to use it with, carries a worked example for opensearch-php both ways,
and says plainly that a ringphp client is covered by neither — with the two ways
in that need no transport at all, the Monolog processor and the slow log.

docs/reference/coverage.md gains the same table and loses the "Query type" in
its title: "which clients can be captured" is the second question that page is
for, and it was answered nowhere.

Both runs above were done by hand, which is what let this page be wrong in one
direction and then wrong in the other without anything failing.
#46 is the harness
that would check them — it needs a sub-composer, because opensearch-php
requires PHP 8.2 and this library's floor is 7.4.

The CLI without the package

slowlog answers "which query shape is costing us" from a file every cluster
already writes. The people holding that file are as often SREs as they are PHP
developers, and until now the answer was behind composer require — a language
toolchain to install on a log host, for one report.

Two artefacts per release now, built from the same source:

cat *_index_search_slowlog.log \
  | docker run -i --rm ghcr.io/mrdlef/os-query-digest slowlog

and os-query-digest.phar, one file with a .sha256 beside it, attached to the
release. Neither is a repackaging: tools/build-phar.php finds every file under
src/ rather than listing them, the stub is a nine-line PSR-4 autoloader, and
there is no vendor/ inside because there is nothing to put in it.

  • PharTest builds it and runs it on every PHP version in the matrix, 7.4
    to 8.5. A file anyone can download has to run on the oldest interpreter this
    library claims, not on the one a release was cut from — and the stub is the one
    piece of code a checkout never exercises.
  • The image is compared with the library it was built from. CI digests a
    query both ways and cmps the answers, so "the fingerprints are the same" is
    a check rather than a claim.
  • --version names the build. An installed copy is identified by the
    composer.lock that installed it; a file copied onto a jump host is identified
    by nothing, so the phar carries its release and says so. Nothing was added to
    src/ for this: the build is passed in, and git describe supplies it locally.

docker/Dockerfile is unchanged and still the development image. The
distributable one is docker/cli.Dockerfile, which runs as nobody and reads
standard input, so the common case mounts nothing at all.

Every fingerprint in the docs is recomputed now

The Use cases pages run their aggregations against a live node, the playground
presets are generated and the fixtures pin every hash. The examples in the
guides were the ones nothing read: written by hand, and read only by people.

That cost something in v0.12.0. A digest block copied from the logging guide into
the transport guide and edited by hand had its hash and its clause order wrong
— the canonicaliser reorders, so not status:200 sorts ahead of service:api
and the hand-edited line described a query nobody could send. It was caught by
recomputing it by hand before the tag, which is not a check.

DocExampleTest recomputes them instead. Each block carries the
<!-- verified: name --> marker the Use cases pages already use, and:

  • the two landing pages print a request and its digest, so the digest is
    checked against a run of the request beside it — neither is copied into the
    test;
  • the slowlog table and the --json block are compared, byte for byte, against
    a real run of a real 60-line log the test builds;
  • the CLI transcript pipes the request out of its own echo;
  • the explain block is checked line by line, including which rules fired.

A stray fingerprint in a sentence fails too: every q5:-prefixed hash on the
pages has to be one these examples mint, with a single documented exception —
the page about hash stability prints the shape of a hash rather than one.

Two hashes in the --ndjson example were not real. Nothing in this
repository produced q4:2e2169e22798 or q4:33a434d95576, and nothing ever
had. That block now counts fingerprints from a file the test digests, and one of
them is the hash the same page prints thirty lines above — which is the point the
paragraph under it was already making.