TREX 0.3.0

Every surface

Breaking

• The unit above the token is called a supertoken everywhere: a template's
  ${@construct} is ${@super}, an explanation reads call supertoken at depth 0, and the axes a refused template lists name super.
• Bare @seam and @seam:super do not hold at the input's first unit, as
  @seam:byte and every written cut do not; they held there before.
• The pair field's bound of a cut is the mean log2 g over the pairs of
  units straddling it, in bits per pair, where it was their sum. A cut near
  either end of the input, which fewer pairs straddle, reads on the scale
  every other cut reads on, so @bound and --record bind:Q no longer
  favor the input's ends; on a short log the weakest cuts are its odd lines
  where they were its first and last tokens. A cut more than eight units
  from both ends keeps its order, and a bound written in bits
  (@bound<=-1b) reads per pair.
• A run of more than 32 hex digits at a length no digest has lexes as one
  hex token; it lexed as a word, a number and a word, or base64. hex is
  a built-in atom name, so a declaration named hex is refused. A declared
  kind's code is 34 and up, and an index an earlier trex wrote reads as
  absent until trex index writes it again.

Added

• A hex token kind, \{hex}: a run of more than 32 hex digits holding a
  hex letter at a length no digest has, as a key, a dump or a digest of
  another size is written; 32, 40 and 64 digits stay \{hash}. It is
  painted red and underlined with the other alarm kinds, a scan for
  \{hex} over an input with no such run answers without lexing, and the
  GPU matches it as the CPU does.
• A written cut on @seam and @ambiguous: @seam>k, @seam:byte>=k and
  @seam:super>k keep the cuts whose strength is more than k
  standard deviations above the input's mean, or at least k;
  @ambiguous>k and @ambiguous:token>=k the points whose disagreement
  passes k, from 0 to 1. An explanation repeats the cut as written (on at a token cut >0.5) and holds it in a cut piece, ${@seam.cut} and
  ${@ambiguous.cut}.
• TREX_REQUIRE_CUDA=1 fails a build for Windows or Linux whose CUDA kernel
  did not compile, naming why; without it such a build warns and runs on the
  CPU. trex's own release builds set it.

Changed

• A number written with an exponent (6.02e23, 2.5e-4), a 0x, 0b
  or 0o prefix (0x3e8), or _ before each further group of three
  digits (1_000_000, 0xFFFF_FFFF) lexes as one number token on every
  surface and the device; it lexed as a number and the word after it.
  Magnitude reads it at its value's scale, so 6.02e23 reads 23.78 where
  it read 0.78, and \N predicates, typed values, aggregates, the
  numeric orbit group and a tree index read its value. A comma still
  separates, so 1,000 is three tokens, and a form with a letter, digit
  or underscore right after it (1e3a, 2024_01_02) lexes as before.
  Readings of any input holding these forms change.
• A number's exact value is held as its digits and a power of ten, so a
  huge one costs a comparison, a sort or a key what its written digits
  cost, and prints in that form where writing it out would put more than
  21 zeros past its digits: 1e999999 prints as 1e999999 in a --json
  value, an aggregate and an orbit or echo key, as does a number written
  out with 22 or more trailing zeros.
• A declaration takes every line a pattern file holds, fields and a
  rule's fix, meta, files, unless, record, record-start and
  record-span among them, and removing an atom takes the lines declared
  that add to it. A test line declared on its own is no longer listed as
  an atom.
• SARIF output declares its columns as "columnKind": "unicodeCodePoints",
  which SARIF 2.1.0 requires of a run holding results.
• Of two contested points closer than the observation's minimum gap, the
  stronger is kept, a tie keeping the earlier; the first was kept before.
  The strongest point is always kept, and a higher threshold removes only
  points below it.
• A shape template region ends at its last token whose kind matches the
  kind one period before.
• Region classification (trex spectral --classify, scan --files --texture and the --texture filter, Measure-TrexSpectral -Classify
  and trex.axes.spectral(classify=True)): a region is a blob where base64,
  hash and hex tokens and high-entropy runs cover more than half of it, and
  otherwise a table only where template runs holding a kind other than a
  word cover more than half of it. A run repeating every one or two tokens
  counts toward a table only where its lines hold about as many tokens as
  each other, so a column of numbers one to a line is a table and a
  paragraph of clauses and commas is not. A paragraph whose words repeat
  reads as prose, and base64 of English text as a blob, on one line or
  wrapped.
• The shape axis reads a word's case and length from its characters, so
  Επειδή is Pascal as Whereas is and a Korean word of two syllables is
  short, and a punctuation token's glyph is its whole character, so 、 and
  。 are two classes and echo's structural-rhyme keys print them; it read
  ASCII case, length in bytes and the glyph's first byte. Shape classes,
  periods, templates and change points, the context axis and echo's keys
  change on text outside ASCII; ASCII text reads as it did.
• A spectral change point also marks where a span's texture gives way to
  the texture before it, so a line of code in the middle of a sentence has
  one at each edge (trex spectral --segment, \F{onset}, the regions of
  --classify, Measure-TrexSpectral and trex.axes.spectral). The exit
  is at the first of twelve bytes in a row on which the recent reading
  stays close to the texture the span interrupted; before, a span shorter
  than the reader's slow clock had an entry and no exit, and the regions
  after it read as one with the span.
• The spectral change-point threshold's running mean and deviation take
  each byte's departure only up to three deviations, so a divergence that
  rises over many bytes, as code inside Chinese, Japanese or Korean prose
  does, no longer lifts the threshold ahead of itself; such a span was
  often given no change point at all. Change points move on most inputs,
  and plain text holds a few more of them.
• The CUDA kernel compiles with an nvcc of CUDA 12.0 or later; with an
  older one the build warns, as it does with none, and runs on the CPU.
  trex's releases are built with CUDA 12.0, and their kernel loads on an
  NVIDIA driver of the 525 series or later.
• Release builds of the trex binary and the Python extension use no
  link-time optimization, and the PowerShell module uses thin link-time
  optimization. Measured against fat link-time optimization under Windows
  on an AMD Ryzen 9 7900X, the binary is 3% smaller and scans in the same
  time, and a scan takes 11% less time in the extension and 6% less in the
  module.
• Error messages, help text and --explain readings use plain wording.
  Messages a script may match read: a ~k group's count "must be less than
  the number of atoms"; "an orbit scope cannot be nested inside another"; in
  a test line, "in goes between the span ... and the text it is read
  from"; an empty mark "holds no text, so nothing places the field"; a
  template field "is one value and takes no .PIECE after it" or "is not a
  reading AXIS gives; it gives ..."; and @order on the first timestamp
  "the first timestamp, with nothing to compare with".

Fixed

• The pair field's first unit, with nothing before it, reads no strain and
  no bound: @strain and @bound hold nowhere there at any grain,
  --explain says there is nothing before it to strain against, a
  percentile is taken over the units that have a reading, and --record bind:Q cuts at the weakest Q% of those. Both read zero there before, so
  @bound<10 held at the first token of most code files.
• A byte-pattern refuses a backslash before a letter other than d, D,
  w, W, s, S, p and P, (? at the start of a group and a
  backslash inside a class, each a parse error naming it, in a shape and
  between backticks. They parsed as other characters before: \x00 was the
  characters x00, (?s:.) the characters ?s: then one more, and
  [\x00-\xff] a class of punctuation, digits and capitals.
• Prose in a script written outside ASCII reads as prose: the spectral
  texture readings (--classify, the --texture filters, -Classify,
  classify= and \F{texture:...}), observation's byte grain and
  resonator::over_bytes read their bytes as UTF-8. A letter outside ASCII
  is a letter on each of its bytes, any other character outside ASCII
  counts once by its class, so ’ weighs what ' does, and only a byte
  that is not part of a well-formed character is read apart. They read
  Cyrillic, Greek, Chinese and Japanese prose as a blob before, and the
  accents inside Latin words as foreign bytes. The texture readings'
  entropy gates rise with the share of bytes inside multi-byte characters,
  whose bytes carry more entropy than the text they spell. Readings of any
  text outside ASCII change; ASCII text reads as it did.
• The lexer's blob gate takes the same share into account, so a Chinese or
  Japanese sentence lexes as words and punctuation; a sentence of a real
  Chinese document collapsed into one opaque other token, and no scan
  could find a word inside it.
• The region classification reads Arabic, Chinese, Greek, Japanese, Korean
  and Thai prose as prose; it read the Universal Declaration of Human Rights
  in each as a table, since their words were in one or two shape classes
  that repeat at any period. Base64 wrapped at 76 columns, digests one to a
  line and PEM certificate bundles read as a blob; they read as a table, one
  token kind repeating line after line.
• The Windows...

trex 0.2.0

Added

• The trex binary for Windows x64, Linux x64, FreeBSD x64 and macOS arm64
  on the GitHub release.
• A fields line in pattern files, which the file trex infer saves
  carries: a saved build read again keeps its casts, record starts, nested
  fields and accessors on every surface. scan --fields prints its records
  table, and Python's Pattern.fields and Pattern.records(text) read it.
• Python: trex.files() lists the files a scan of some paths reads,
  trex.read() gives a file's text decoded as a scan decodes it,
  trex.texture() names what a text reads as mostly, and Pattern.diff()
  and Pattern.rewrite_file() diff a rewrite of a file or write it in the
  file's own encoding.
• PowerShell: Get-TrexFile -Binary.

Changed

• An explanation names the token of a declared shape or kind by its
  declaration (ticket), as a token listing does.
• The shape axis reads a region as a table only where template runs cover
  more than half of its bytes.
• trex seam --recover FILE leaves the word count out of its header.
• Select-TrexMatch, Edit-TrexText and Protect-TrexText write a warning
  naming a file given by -Path or -LiteralPath that holds a NUL byte.

Fixed

• trex scan --files and Get-TrexFile listed files holding a NUL byte that
  a scan leaves unread; they leave them out unless --binary or -Binary
  asks for them.
• rewrite, redact and scan --rules --fix passed over one named file
  holding a NUL byte without a word under --dry-run, --in-place and
  --interactive; they refuse it and exit with failure, as they do when
  writing to the standard output.
• Python Match.explain reports the rung of the scan ladder that answered,
  as scan --explain does.
• The PowerShell module reads a pattern's relative @file paths against
  PowerShell's current location.

Removed

• Python Match.explain takes no route=.