Pipsqueak

ChrysaLisp 7.0 "Pipsqueak"

The Lisp engine squeezed till the pips squeak, a network that goes quiet when
it has nothing to say, and a machine that sizes its own network. Small, and
much quicker than it looks.

Every machine on a network must run this release. The routing ping and the
shared memory link status have both changed. An install from before this
cannot build its way forward, update with make install, which starts from
the new snapshot.zip, and rebuild the host programs with make.

The Lisp engine

• 40% fewer instructions over the test suite, and a full build of the system
  on one node in 0.32 seconds, from 0.54, on an Apple M4. A Raspberry Pi 4
  builds in 1.3 to 1.5 seconds, the best seen before was 2.3.

• The recursive core holds a fraction of what it did on the stack. A special
  form and the body of a lambda hold nothing at all, a call holds 16 bytes
  while its args evaluate.

• The args list and the environment of a call are reused, not made and freed.
  A prebound function is not evaluated. The Lisp object is fetched from the
  task control block, not kept in a stack slot at every level.

• The boot environment is spread over buckets once loaded, a lookup of an
  unbound symbol was a scan of all 828 of its symbols.

• The :lisp class, all but its init and deinit, is now written with
  registers.

• New doc, docs/ai_digest/till_the_pips_squeak.md, has every step, what it
  saved, how it was measured, and what was tried that did not pay.

The network

• The routing ping no longer carries a node's services, only a hash of them,
  and backs off from 1 second to 64, so a settled network goes quiet. A link
  coming up or going down kicks it awake.

• A shared memory link knows the process at the other end and asks the host if
  it still runs. A dead node is known for certain within a second, and a busy
  one is never taken for dead. A link that times out is marked down, not torn
  down, and comes back.

• An idle network burns far less, a fully connected 27 nodes went from 3.76%
  of a core each to 0.85%.

• A node tells its neighbours its load, not just its task count, and a new
  task goes to the least loaded.

• Every node knows which machine every other node is on, (lisp-nodes :t),
  (lisp-nodes system) and (lisp-systems).

• Mail for a node that is still known is no longer freed as undeliverable.

• New topology launch scripts, ring, cube, tree, mesh and star.

The host

• New host calls, (pii-spawn) (pii-pid) (pii-alive) (pii-cpus) and
  (pii-memory), on Darwin, Linux and Windows.

• A network can size itself. ./run_tui.sh -n 0, and run.sh, start one node
  for each processor, fully connected. make install uses it.
  (node-spawn), (node-auto) and nodes -a add nodes to a running network.

• RISC-V 64 is running again, and LoongArch 64 is running, both tested under
  QEMU with byte identical self hosted boot images.

The assembler and its checks

• Call fusion on the link register targets, boot images 2.4% smaller on ARM64
  and 4.2% on RISC-V and LoongArch.

• The generic VP optimiser is one sweep, and a bug in it is fixed, a push or a
  pop between two reads of a stack slot was not seen to move the stack.

• A validate build, make it validate, now fills cells on alloc and on free,
  guards the end of each cell, and checks every ref and deref, to catch use
  before set, use after free, double free and overruns. It found two real
  faults at once, both fixed.

• trace -i -l lints every function, with no filters.

Tests

• A new test framework, tests/suite.inc, and the suite has grown from 1,629
  tests to over 3,300. The tests command prints only failures and a summary,
  runs modules in parallel over the nodes, and takes -m, -v, -l, -j
  and -f.

• The new edge case modules found and fixed faults in (nlz), (trim),
  (join), (swap), (slice), the shifts, (nums-sum), (json-stringify),
  (map!) and (filter!), (sort), (replace-regex), (tree-load), regexp
  matching at the end of text, whole word search, editor replace, and the
  Document and Buffer classes.

Services, commands and apps

• The lock service, @Lock, has shared read and exclusive write locks on
  hierarchical paths, scoped (with-lock) macros, a history, and a locks
  command. The file commands, libraries and all the GUI apps use it for their
  files and configs.

• The task farms no longer lose a job on a slow machine.

• lisp -r runs code given on the command line. New brackets command. The
  reader reports an unbalanced form, it used to crash the node.

• mail-read-timeout cancels its timer when mail arrives first, 40,000 timed
  reads went from 5.4 seconds to 15ms.

• Onslaught is feature complete against the C++ original, with the campaign
  map, load and save, and panned sound. It can be played remotely over its
  @Onslaught service, and the onslaught command has a bot that plays it.

• New lib/streams/hex.inc, (ui-save), View :add_before and :add_after,
  large strings in tree-save and tree-load, and tool tips on six more
  desktop apps.

Where it came from

• New docs/history/history.md, "How We Got Here". The Virtual Processor
  began in 1989 as a set of assembler macros under two Atari ST and Amiga
  games, became the Taos operating system, which won the UK's Computing
  gold award for innovation in 1994, then Tao Group's intent, and is the
  same idea this system is built on. The press cuttings of the time, from
  The Independent, BYTE, Edge, the IEE Review and others, are in
  docs/history/press/, each credited to its author.

Windows

• Installed, built and tested on Windows for this release, with thanks to
  Martyn Blyss. The TUI and GUI run, the system rebuilds itself to byte
  identical boot images, a full build takes 0.19 seconds, and the test
  suite passes.

• The first Windows run found that starting a node took down the node that
  asked. A host call runs on the stack of the task that makes it, and on
  Windows CreateProcess needs far more than a task has. (pii-spawn) is
  now made by the kernel task, on the host's own stack, on every host.
  nodes -a works there.

Not ready

• The fmt command has a new engine, it is an experiment and is parked. Do
  not use it on the tree.

• On Windows, -n 0 in the PowerShell scripts has not been run, and the
  batch files do not take it.

The full notes for every change are in STATUS.md.