Rosetta

ChrysaLisp 7.5 "Rosetta"

Code made once, as VP, and translated where it runs. A build compiles
each file to VP one time and gives every CPU its native code from that,
by translators that are themselves VP. A machine that lacks a function's
native code makes it from the VP as the function is bound, and a task sent
to another machine reads its code from the system that sent it.

This is the first GitHub release since 7.0, "Pipsqueak". Four tags came
between, and this page has each of them in short: the shader language and
the GPU, triangles, a file system of our own, machines that find each
other and sync, the Storage system, Films with sound of our own decoding,
sealed links over the internet, and voice calls between desktops.

Update every machine together, with make install. The link has a
version now, and a machine of 7.4 or before does not link to one of 7.5.
The translators are VP, the Lisp ones are gone, so a tree from before
cannot build its way forward: make install starts from the new
snapshot.zip, whose boot image carries every translator.

VP, made once

• The VP is the same on every platform. A host call is the same VP
  everywhere and the translator puts its args where the ABI wants them;
  a task's state is saved and loaded by VP instructions the translator
  expands; nothing in the VP knows an ABI, and the stack has no
  alignment in it, each host call lines it up at the call.

• A build has two stages. Each file is compiled to VP once for a build
  mode and kept, obj/vp/<mode>/; each platform's code is translated
  from it, only what has changed. A build of every platform walks the
  tree once.

• The translators are VP, lib/trans/<cpu>.vp: VP binary in, native
  code out, with their peephole passes. They give the same code to the
  byte as the Lisp translators did, the whole tree in 7 ms in place of
  162. ARM64, x86_64 for AMD64 and WIN64, RISC-V 64, LoongArch 64, and
  VP64 for the emulator.

• The VP spec, vp_vm.md, says what each instruction
  does at its edges: sets give 0 or 1, float branches follow IEEE on a
  NaN, constants and offsets of any size, loads and stores at their
  natural alignment only. tests/system/test_vp_spec.lisp holds every
  target to it. Writing it down mended RISC-V's long branches on equal
  operands, x86's logical and, and the optimizer forwarding a narrow
  store as the whole register.

• run_tui.sh -V, the VP64 emulator in validation mode, says each
  unaligned load or store.

Code that travels

• A function's VP binary is kept beside its native code, <name>.vpb.
  When a node binds a function that has no native code here, the loader,
  which holds its CPU's translator, makes it from the VP, with each
  function it calls, and keeps it as a build would have.

• @Source, one to a system. A task sent to another machine reads its
  Lisp from the system that sent it, so NetMon's and Mandelbrot's
  children need none of their files where they run.

• Function servers. A task from another system binds that system's
  natives: the hashes of a function and all it calls are asked for once,
  only VP this node does not have is sent, as VP binary, and its
  translations are cached by hash. They live in a server of their own
  that goes with its last task.

• The boot images are smaller. The VP64 one holds what an install needs
  before it has built anything, and carries all the translators; a
  native one holds the loader, the kernel and what they call, 44KB, and
  the rest is bound from obj/ as it is asked for. snapshot.zip is
  161KB. A load goes on in the block the last one ended in, and a block
  that has run is never written again.

Films

• AVI, lib/media/avi.inc, read as an mp4's tracks are: idx1, OpenDML,
  or its chunks one after another where there is no index. Sound of MP3,
  AC-3, PCM, and Microsoft and IMA ADPCM, lib/media/adpcm.inc.
  Pictures of Motion JPEG, ours or the host's, H.264, and MPEG-4 Part 2
  where the host decodes it.

• An MP3's picture is drawn from the frames its sound decoded, not
  decoded again, and the decoder takes a third less time a frame, the
  same samples bit for bit. HE-AAC decodes 13% quicker by SBR's filter
  banks done by the DCT-IV, and 15% quicker again by its hot loops
  native, the same samples.

• A film resized keeps its decoder and does not pause.

Storage

• A store on a flash stick is written 2.3 times as fast: blocks go in
  folders by when they were put, so a run of them writes one folder, not
  256. A copy into a store is four times as fast, its whole blocks
  written in order at the end of a run; 63 MB/s onto Flash.

• A store's stream keeps its last block and what it asked for ahead
  across a near seek: an MP3 read from a store plays with no gaps,
  frames 116 times faster.

• A member that steps aside for a desktop gives up its store's open
  part way, and one told to go while it starts is stopped, not left
  running and claimed. The Windows host seeks and stats in 64 bits, so
  a store image past 2GB is made and read there.

Files

• A file picker shared by the apps, one call to open it. A drawer: a
  folder open, its folders and files of the kinds asked for, a filter,
  the path above. Down the side the places: the tree, the user's
  folder, and the app's own folders. A preview
  of what is picked, a film's first frame and length, a font's glyphs.
  Objects of the store are picked as files are, store:path. Films,
  Fonts, PCB and Images open by it.

• Whiteboard: Insert adds a board file's things to the board. A .cwb
  keeps the keys it does not know and saves them back, and :script is
  kept for the scripts shapes and documents are to carry.

Networks

• The link has a version: the handshake is CLDOOR3 with
  +door_version 1, a node of another closes at once, and beacons say
  it. mesh says in one sentence what is not right of a machine it
  hears, and names the machines a door was closed on. The link protocol
  is written down, link_protocol.md, and
  the TCP link is tested against it byte by byte.

• A message is made with its src and offset 0: a message of one
  fragment had carried a reused block's old header onto the wire.

Tests and tools

• Tests wait for what they test, never a time, so a slow machine fails
  nothing. A run always says how it goes, and a module long past its
  usual time is said to be late as it runs.

• check, forward, includes and imports find more: forward
  finds a function named in (const) above its definition, and
  includes reads an .inc that is not a class's as part of the file.
  The lint is clean, relies with it.

• On a slow machine no app takes a whole node: Eyes takes at most a
  quarter of its node, Molecule draws its new atom images between
  frames, and the GUI waits as long as its last frame took.

• The eraser asks (poly-hit?) of two hulls, the GJK is gone, 25 to 55
  times quicker. A sync -d takes the folders its removes leave empty,
  and a file changed twice in one second, to the same size, is no
  longer taken for unchanged by sync's cache of hashes.

• Gone: the profiler, whose figures were its own cost as much as the
  code's, use (time-it); debug mode and its app, the stack frames are
  kept; an app's (jit) of its lisp.vp, make install makes the apps.

Since 7.0, the tags between

Each has its own notes.

• 7.1. A shader language of our own, written as
  s-expressions, and the GPU to run it on. The GUI on SDL3, a Raspberry
  Pi running the desktop with nothing under it, and a session that keeps
  itself to itself.

• 7.2. Vertex and pixel shaders drawing triangles on the
  nodes or the GPU, and typed functions Lisp calls as native code. An
  exFAT file system of our own, SHA-256 and ChaCha20-Poly1305. Tests
  that run only what a change could have broken. Machines that find each
  other: the Net services make a mesh, and sync makes one machine's
  files another's.

• 7.3. The Storage system: the machines of a mesh keep one
  store between them, each block on three of them, read and written as
  a file is, or a function sent out to where the blocks are. A band of
  the scheduler for tasks that work for others, textures in the shader
  language, PNG and JPEG, deflate, HTTPS by TLS 1.3, and the Whiteboard
  made over.

• 7.4. Films with their sound: mp4, Matroska and WebM, the
  pictures by the host's decoders, the sound by our own, AAC with
  HE-AAC, AC-3 and E-AC-3, MP3 and FLAC. Links over the internet sealed,
  voice calls between desktops, TLS 1.2 beside 1.3.

The release check

Every stage passes on every platform, from cold, whatever the cache of
test results had.

• An Apple M4 Max, a 2018 x86_64 MacBook Pro and a Raspberry Pi 4.
  make install from snapshot.zip, 4 s, 10 s and 42 s; every test
  with tests -a, 195 modules and 8,006 tests on each; the boot images
  of all six CPUs; the lint, the forward check and a restore, clean.

• The VP64 emulator's release image, 7,734 tests, those of the error
  checks a release build does not have left out.

• RISC-V and LoongArch under QEMU, each installed from the snapshot as a
  new machine is, its boot image the same to the byte as the M4's cross
  build, and 7,943 tests.

• The Windows host under Wine, make install and 7,936 tests.

Thanks

• To Andy Stout, for the issues that found a store left running and
  claimed, a test that hung with no user, an MP3's pictures decoded
  twice, the Linux host's warnings, and a big store's slow open; for
  input -f taking text==, and for putting the history right on the
  PlayStation assembler; and for offering VA-API on the hardware he
  has.

• To Martyn Blyss, for the Windows runs and the GPU triangles' first
  start there, and to Darren Pegg, for the shader language's names, and
  for the iOS and Android hosts he is building on the VP64 emulator.

Not ready

• Echo cancelling in voice calls has not been heard live; half duplex
  stays on.

• Opus is not here, and WebM's pictures are mostly VP8, VP9 or AV1,
  which the host is not given.

• Pictures by Media Foundation on Windows are built and not yet run on
  Windows itself; the Windows scripts are not run under Wine.

• A store of 256GB takes about 70 s to open, nearly all of it reading
  its folders of blocks.

Enjoy

Chris