1.1.0 - Thirty-two bits, packages that ask nothing of the IDE, and a defect the accelerator hid from its own tests

Thirty-two bits, a package that asks for nothing from the IDE, and a defect the
accelerator had been hiding from its own tests.

The occasion was not ours. The maintainer of the Online Package Manager built
the catalogue version, could not, and wrote down five reasons. All five are
closed. Looking into them turned up as many of our own, all in the same place:
targets we never built, so nothing there was ever checked.

The packages no longer ask for the LCL

crosspascal_parser and crosspascal_parserjit are built with NOFORMS and
NOGRAPHICS, and neither declares the LCL any more. A console program links
against them with nothing else in its uses clause: no Interfaces, no
widgetset. That was the maintainer's fourth and fifth findings, and before this
release not one of the eight documentation samples linked.

Two features step aside for it. Thread used to route an exception raised in a
worker through Application.HandleException, and BlobManager stores images as
TGraphic. Both stand behind those defines. If you want them under Lazarus,
drop the defines from packages/lazarus/crosspascal_parser.lpk and add the LCL
to its required packages; nothing else about the package changes.

Delphi is untouched by any of this. It builds from the sources, where neither
define is set and TGraphic is the VCL one.

Thirty-two bits

Windows i386 joins win64 and linux64, on both compilers. The whole battery runs
there: sixteen test programs on Free Pascal 3.2.2, the documentation samples,
and the packages themselves.

It is a separate installation of the compiler rather than a switch. Free Pascal
refuses to target i386 from a host whose Extended is a Double, and on Win64
it is.

Getting there took two fixes in the library. Five loop counters in
BlobManager were Int64, which i386 does not accept in that position; they
are NativeInt now, while the fields whose value goes into the stream stayed
Int64 so the blob format does not shift. And the value record came out at
twenty bytes instead of twenty-four, with the payload four bytes in rather than
eight - see below.

The accelerator got the order wrong after a function call

Reading the right operand of an operator consumed the rest of the term instead
of a single step whenever that operand was a function call. So 6 / cos(x) / 16
was evaluated as 6 / (cos(x) / 16), and x / sqr(y) * y as
x / (sqr(y) * y). Only a call on the right was affected: with a variable or a
constant there the fold stayed left to right, which is why it went unnoticed.

On x86-64 the machine code generator takes such formulas and the generator is
right, so the wrong answers appeared only where there is no generator at all -
which, until this release, was a target we did not build.

Checked by comparing three thousand random formulas against the interpreter on
Delphi win32 and win64, Free Pascal 3.2.2 on win64, i386-win32 and linux64, and
Free Pascal 3.3.1: zero disagreements.

The check that should have caught it was comparing nothing

The differential fuzz skipped every formula the accelerator declined. Off
x86-64 it declines every formula there is, so the check reported three thousand
compared and zero disagreements while comparing none of them. It now reads the
level off the interpreter counter, and the floor it guards means something
again.

The value record is the same size everywhere

TValue is twenty-four bytes on every target now, with the payload at offset
eight. The compiled script format carries that record verbatim, so a script
built on 64 bits would not have loaded on 32. The directive meant to keep the
layout identical sets a limit on alignment rather than a size, and on i386
nothing in the record asked for eight; it is padded out explicitly instead.

The package no longer hands you a unit under a name the system uses

Outside Windows the parser needs a stand-in for Messages, which otherwise
comes from the LCL. That stand-in was reaching consumers on Windows too, where
the runtime has a unit of that name - and standing in front of it. A project
that used both this package and the LCL stopped with a message naming a unit it
could see. The stand-in is now added only where the system has no such unit.

Installing the accelerator no longer rebuilds the parser

The accelerator package listed the parser's source directory and rebuilt
forty-one of its units into a second output directory of its own. The two sets
then disagreed, and a sample that used both stopped on a unit that was lying
right there. It uses what the parser package built, as it was meant to: six
units instead of forty-seven.

Projects for the samples

The eight programs under samples/docs now come with .lpi files. Open one in
Lazarus and build it - that is the whole recipe. They were written earlier but
never reached a release: the slicer did not carry the extension, so they existed
and travelled nowhere.

There are deliberately no .lpr copies. A copy of the program text is a second
source, and it drifts.

Compatibility

Delphi	37.0 (Delphi 13), win32 and win64
Free Pascal	3.2.2 and 3.3.1, win32 and win64 and linux64

Every line of that table is a matrix that runs before a release.

1.0.8 - Three ways into an evaluation, and the mask now covers all of them

There are three ways to start an evaluation. The 1.0.7 release put the floating
point exception mask on two of them. This one covers the third, which happens to
be the one the accelerator documentation recommends.

Compiled code ran without the mask

CompileScript returns an object you evaluate directly, and docs/USAGE.md
points at it for evaluating from several threads: it holds no shared state, so
each thread can have its own. That path never installed the mask. Division by
zero raised EZeroDivide from a worker thread instead of answering infinity -
the same defect 1.0.7 fixed, on the entry point most likely to be used from a
thread.

tests/JitContractTest.dpr now drives all three entry points through the same
formula and requires the same answer.

A nested parser ignored its own ExceptionMask

A parser called from inside another parser's evaluation used to inherit whatever
mask the caller had installed. The test for "is this the outermost evaluation"
asked whether any frame existed in the thread, and a foreign frame counts, so the
inner parser concluded it was nested and installed nothing at all. Its
ExceptionMask property could say something different and quietly mean nothing.

The test now compares the mask that is installed against the one this parser
wants. tests/FpuMaskTest.dpr covers it in 17 checks.

ExceptionMask is public

The property is documented as yours to narrow, and was declared protected,
which put it out of reach of the code that was supposed to narrow it. Reaching it
took a descendant class for no reason other than the visibility.

A nested loop guard switched off the outer cancellation

ArmLoopGuard with a budget of its own but no flag of its own wrote nil over
the cancellation flag that was already there. An owner asking the work to stop
went unheard for the whole of the inner run. An inner run may replace the budget,
which is a measure of this run; it may not replace the cancellation, which is
about all of the work. tests/LoopScopeTest.dpr, 19 checks.

Name lookup asked the memory manager twice

Looking a name up in the parser tables converted the string to lower case once
for the hash and once for the comparison - same string, same result, two trips to
the allocator. Deriv went from fourteen allocations per call to eleven, and
from 4.05 to 3.40 microseconds.

Worth knowing where that leaves things. On four threads the allocator is the
limit: an allocate-and-free pair takes 4.69 times as long as on one thread, while
moving the same bytes takes 1.91 and comparing the same strings 1.02. Fewer
allocations is the only lever that moves, and there are still eleven.

Compatibility

Delphi 13, Free Pascal 3.2.2 and 3.3.1. Nothing in the public interface changed
except ExceptionMask becoming visible.

Three pieces of thread state that belonged to somebody else

Three things the library kept in thread storage turned out to belong to a
narrower scope. Each is reproducible in a single thread, and each has a test that
fails on the old code.

The loop guard is armed in a pair

ArmLoopGuard and DisarmLoopGuard replace writing to ParseLoopLeft and
ParseBreak by hand. Disarming restores whatever was there before, so guards
nest: a formula that calls Parse can set a budget of its own without clearing
the outer one.

Writing by hand leaked. A budget that runs out is recorded as a negative number,
and that number outlived the run that spent it. Whatever came next in the same
thread inherited the refusal even though it had never armed a guard: a different
parser, a later button press, a plain ten turn loop that should have finished.
tests/LoopScopeTest.dpr covers this in 16 checks.

Two things stay as they were, and the tests now state them on purpose. A nested
evaluation spends the same budget, because a budget for each evaluation is no
budget at all: a nested loop hangs a page as readily as an outer one. A nested
evaluation watches the same cancellation flag, because an owner asking you to
stop means all of it.

The exception mask belongs to the evaluation

The floating point exception mask was installed by the parser constructor, so it
belonged to whichever thread created the object. That went wrong in both
directions:

• Evaluate on a shared parser from a worker thread and division by zero raised
  EZeroDivide instead of answering infinity. That is the arrangement the
  plotting component uses: one parser, four workers.
• While a parser was alive the mask stood for the whole program, so neighbouring
  code in the same thread quietly stopped getting its own exceptions.

An evaluation now installs the mask and restores the caller's when it
returns. The accelerator does the same around machine code, which runs outside
the interpreter. The mask an evaluation installs is the ExceptionMask
property. Narrow it if you want the exceptions inside formulas too.
tests/FpuMaskTest.dpr, 13 checks.

Two cases make this release worth taking. You share one parser between threads,
or your program narrows the mask for its own arithmetic.

The lock around Deriv and Parse is per parser

It used to be one lock for every parser in the process. Four threads with four
unrelated parsers queued behind each other on any formula holding a derivative.
Parse now holds the lock only while it compiles. Keeping it across the
evaluation meant holding it across arbitrary user code, which is how deadlocks
are made. tests/MethodLockTest.dpr, 8 checks, both of them deterministic.

Compatibility

Delphi 13, Free Pascal 3.2.2 and 3.3.1. The Linux matrix builds and runs the
parser on both, in two locales.

Free Pascal 3.2.2 builds the library again

Free Pascal 3.2.2 builds the library again.

Function references - reference to function - arrived in 3.3.1, and everything
that used them turned out to sit in a single file: seven declarations in
MemoryUtils, none of which any other unit touches. On 3.2.2 those callbacks are
method pointers instead: you pass a method where you would otherwise pass an
anonymous function, and nothing else changes.

Nine functions that 3.2.2 lacks in its Math unit - ArcCot, ArcCotH,
ArcCsc, ArcCscH, ArcSec, ArcSecH, CotH, CscH, SecH - now travel
with the library, taken verbatim from the 3.3.1 runtime, sign handling and all,
so the values agree to the last bit.

The compatibility table in the README is a matrix that runs before a release
rather than a guess. This one was not: the Linux matrix had never been run
against 3.2.2, and against 3.2.2 not one unit compiled.

Added

• tests/MathFamilyTest.dpr guards that whole family by contract rather than by
  a table of numbers: a reciprocal multiplied by its base is one, an inverse
  returns the argument of the direct function. It also pins the branch
  ArcCotan answers in - ArcTan(1/x), within (-pi/2, pi/2), not (0, pi) -
  which is exactly where a hand-written formula quietly picks the other
  convention.

Fixed

• Two accelerator tests and the thread-safety sample died on Linux before
  reaching their first line. On Unix the thread driver has to be the FIRST unit,
  not merely present: units are initialised in the order they are listed, and
  Classes standing ahead of it was enough to break that. The sample had no
  driver at all, so the program that backs the thread-safety section of the
  README had never run on Linux.
• A check compared a bound against Double(High(NativeInt)), which reinterprets
  the bits rather than converting the value: 0x7FFFFFFFFFFFFFFF read as a
  number is NaN. The comparison was against garbage wherever the compiler took
  the cast literally, and stayed green where it did not.
• The accelerator now says why it declined machine code even when the
  interpreter picked the work up, so the contract about a wide Extended can be
  checked at all.
• The Linux test script looks for the widgetset folder instead of naming one, so
  a Lazarus built with gtk2 no longer reports a missing Interfaces unit.
• The README says which two units ask for the LCL - Thread for
  Application.HandleException, BlobManager for TGraphic - and that
  NOFORMS and NOGRAPHICS switch both off. That is what the console matrices
  and the WebAssembly build do.

The Delphi battery guards what the FPC battery guards

No change to the engine: this release exists so that the two test batteries
watch the same contracts.

ExitRoutingTest, ThreadSafetyTest, ThreadShareTest and LoopGuardTest
were listed for FPC only. What they guard - who owns an Exit in a chain of
parsers, and the thread-safe subset - is written in code both compilers share,
so a regression there would have gone unnoticed on Delphi. They were run by
hand instead, which is the definition of a regression waiting to happen: it
holds only while somebody remembers.

The Delphi run is now 18 targets and 581 checks.

The Lazarus package descriptions carry the product version.

An Exit reaches the evaluation it belongs to

A formula can call out to a second parser, and that parser can call back into
the first. The frame chain of one thread then reads A, B, A, and an exit
raised in the innermost A has to travel past B untouched.

Fixed: a parser standing in the way took the Exit. Version 1.0.2 asked one
question in the handler - "is there another frame of MY parser below me?" - and
for the middle B the answer was no. B took itself for the root, swallowed an
exit addressed to A and returned its value as its own. The outer A never saw
it and quietly finished a different sum: 49 where 42 was due. The exception now
carries its owner, so the handler asks two questions: is this exception mine,
and am I the outermost evaluation of my parser. The second alone was never
enough.

The defect was not a regression in 1.0.2 - the code before that release behaved
the same way. We fixed the ownership of exit and closed two cases out of
three.

Cheaper, too. Looking for the enclosing evaluation moved off the path of an
ordinary formula: it happens only when an exception actually appears.

Fixed: the README promised more than the code delivers. "Evaluation is
thread-safe after that" was not true. A TScript is a mutable execution image,
not immutable bytecode: evaluation writes intermediate values and the result
into the script headers. So one buffer cannot carry two active evaluations -
not in two threads, and not reentrantly in one. Measured: a formula that
re-enters the same buffer answers 302 where 301 is due; four threads on one
script instance answer wrongly several thousand times out of four thousand
each, without raising anything.

The Thread safety section is written from scratch: what to freeze before
starting workers, why Copy(Script) is required and how it differs from a plain
assignment, what happens to registered variables and callbacks, and how long the
returned PValue stays valid. The copy rule comes with a working example,
samples/docs/threadsafe.dpr, which the build matrix compiles and runs.

Proof. tests/ExitRoutingTest.dpr - 32 checks covering recursion, a chain
through one foreign parser, a chain through two, an exit owned by the parser
in the middle, the legacy constructor inside and outside an evaluation, the
reentrancy that is allowed, and the semantics of the formula-level TryExcept
and TryFinally. On 1.0.2 the suite fails 7 of 31; here it is green, on Delphi
13 and FPC 3.3.1 alike.

tests/SharedBufferRepro.dpr reproduces the forbidden case on purpose. It is
kept out of the green suite: it demonstrates what the documentation now
prohibits, and it will be the starting measurement on the day the execution
image becomes read-only.

Packaging. The Lazarus package descriptions said "Copyright Yuriy Pisarev"
where the repository is MIT, and carried a version unrelated to the product.
Both are corrected in all package files.

Exit belongs to its own evaluation

One parser is routinely shared by several threads - the plotting component
hands the same object to four workers at once. Evaluation always survived
that; Exit did not.

Fixed: Exit answered to the thread scheduler. The nesting depth lived
in a field of the parser, shared by every thread using it. Two threads
evaluating at the same time read a depth of 2, so the one whose formula hit
Exit took itself for a nested call and threw the value out as an exception
instead of returning it. The counter also drifted: Inc and Dec are
read-modify-write, a lost update parked the field below zero, and Exit
stayed broken for that parser until another race happened to repair it. The
depth now lives in a frame on the stack of the call, linked through a
thread-local pointer, with a reference to the nearest frame of the same
parser. No locks, no atomics, no allocation on the path of an ordinary
formula. Parallel roots are independent, two parsers nested in one thread are
independent, and only true recursion into the same parser passes the Exit
upward.

One declared change of behaviour. A bracketed group in a formula is a
script of its own, and the evaluation mode decides when it runs. Evaluating
groups up front used to happen outside the handler, so the group kept its own
Exit: 99 + (Exit(42)) answered 141 with ExecuteKindSet = [], and 42
in the default mode. One public call now means one scope for Exit, and both
modes answer 42. The pair of tests R4/G4 in the new suite records both
the new value and the fact that the default mode did not move.

Proof. tests/ThreadSafetyTest.dpr - 29 checks. On 1.0.1 it fails 5 of
them, exactly the defects above; on this release it is green, along with the
rest of the battery on Delphi 13 and FPC 3.3.1, Windows and Linux.

Documented. The loop guards from 1.0.1 (ParseBreak, ParseLoopLeft)
now have a README section of their own with a compiled-and-run example,
samples/docs/loopguard.dpr. The scope is stated exactly: guards belong to
the thread, every nested evaluation in it shares the budget and the flag, and
they are set at the boundary of the outermost call.

v1.0.0

First public release.

An expression parser and virtual machine for Object Pascal. A formula is
compiled to flat bytecode and run with a linear pass over memory; on x86-64 the
bytecode can be compiled further, to machine code.

• Delphi and Free Pascal, Windows and Linux, one source
• runtime and design-time packages for both IDEs
• a test matrix that builds and runs everything with both compilers
• documented samples that the matrix builds and runs as part of the checks

Try it without installing anything:
https://pisarev.github.io/mathparser-live/demo/

There is no binary here on purpose: this is a library, and what you want from it
is the source. The archives below are the sources of this tag.