slate 0.0.31 — five new modules, a server on the JavaScript host, and Unicode where ASCII used to be

Five new built-in modules, a server on the JavaScript host, and Unicode where ASCII used to be.
The largest release since the language got a name, and every item of it is something a consumer —
sluice, lath, mortar, the board — asked for by not being able to write something.

slate:sqlite, the database that needs no server

Eight natives are the floor — the connection, the statement and the row being C's — and everything a
program touches is written in slate. One export, because a database is one object: exec,
query, run, transaction, compiledWith and close are methods on it. They could not have been
exports whatever the design — close is slate:net's name, query is slate:dom's and run is
slate:process's, and every built-in module declares into one scope — so it reads the way
slate-language/pg does and the two databases a program
might reach for are not learned twice.

Statements are compiled once per piece of SQL and kept, bounded so a program building SQL in a loop
does not accumulate one per string. transaction commits on a return and rolls back on a fault.

Under slate js this is node's node:sqlite, so the module is whole on both hosts and a browser
refuses naming it. node exposes no sqlite3_bind_parameter_count, so the runtime works the count out
from the SQL itself — without that the two back ends would disagree about the commonest mistake there
is, node binding SQL NULL for a ? nobody gave. Eight shapes of SQL are compared against SQLite's own
answer; the ninth is select "hello", a string literal in the build macOS ships and no such column
in node's, which is two builds of one library rather than anything slate can settle.

slate:lmdb, a store a server can restart over

Every server slate is aimed at wants the same three things and had nowhere to put them: a session
store, a rate-limit bucket per client, and a replay ring for the events a reconnecting browser missed.
A read takes no lock and blocks nothing, which is the shape a request handler needs.

MDB_NOTLS is always on and is not an option. LMDB's default gives each thread one reader slot,
so a second read transaction on a thread that has one is refused — and the message, "Invalid reuse of
reader locktable slot", reads as a damaged lock file rather than the design decision it is. slate is
one thread with an event loop on it, so two handlers each holding a reader is the ordinary case.

A key that is not there is null and deleting one is false; everything else faults with a sentence,
and the two worth catching name the knob — a full map names mapSize, a reader asked to write names
lmdbWrite. No JavaScript host has LMDB and all nineteen names refuse there.

slate:image — PNG, JPEG, GIF and WebP

An image is a record — { width, height, channels, pixels } — and not an opaque handle, because
the pixels are the point: a program wants to read them, store them, send them, or build one itself. A
thumbnail is encodeJPEG(resizeImage(readImage(upload).value, 200, 200), 80) with nothing in the
middle to give back.

readImage and imageShape answer results and the other three fault, which is the library's rule:
bytes from an upload are a 400 to send, and a record the program built that lies about its own
dimensions is a defect. imageShape exists for the size guard — the pixels are
width * height * channels bytes however small the file was, so a 4 KB PNG claiming 20,000 square is
1.2 GB the moment anything decodes it.

WebP is what a browser writes when a page re-encodes a photograph before uploading, and stb has
never read one — so it is a second binding, over libwebp, routed by the twelve-byte RIFF header rather
than by trying stb and catching the refusal. encodeWebP(image, quality) and
encodeWebP(image, { lossless: true }) are the writing half. libwebp has no greyscale, so one and two
channels are converted here with stb_image's own luminance weights: readImage(upload, 1) means one
thing whatever somebody uploaded. An animated WebP is refused by both readers, naming the demuxer that
reading one would take.

slate:zstd, and Content-Encoding: zstd on a response

Zstandard fills the gap the other two encodings leave between them: brotli at quality 5 is the slow end
of what a request handler can afford and deflate is the fast end and compresses worse than either,
where zstd at level 3 is several times faster than deflate and compresses better. slate:http writes
Content-Encoding: zstd for a client that asks for it and brotli otherwise, and Accept-Encoding is
now read once per name — so zstd;q=0, br gets brotli rather than nothing.

Under slate js this is node's zlib. node's decompressor is silent about a frame that ends in the
middle — an empty buffer and no error, where libzstd says srcSize_wrong — so the runtime reads the
frame header and measures what came back against what it claimed.

slate:net and slate:llhttp on the JavaScript back end

The whole of slate:net was owed under slate js, and what that cost is not slate:net:
slate:http is written in slate over listen, onBytes, send and close, so no server could be
started there at all — and lath, whose router is checked by rendering a page through a real request,
had two tests skipping on every run for want of a listener.

slate:llhttp is a parser here rather than a binding — node's own is behind
internalBinding('http_parser'), which is why undici carries llhttp compiled to wasm — so the HTTP/1.1
request grammar is written out, refusals and all: 400 for bytes that were not HTTP, 431 for a head over
the limit, 413 for a body over it, and the two smuggling shapes llhttp refuses by construction. TLS is
still owed and says so.

Case, whitespace and the normal forms are Unicode's

upper, lower and trim were the ASCII range and are the whole database now; normalize(s, form)
and casefold(s) are new. What decided every detail is that a JavaScript host answers all of this
natively, so the back end carries a table of the hundred and two characters that uppercase to more than
one, writes out the final-sigma condition, and spells trim out on both hosts because ECMAScript's
whitespace is not the database's. Every code point there is was run through both back ends and compared.

Placeholder lambdas

map(xs, _.name)
filter(ns, _ > 3)
sorted(ps, _.age < _.age)

A _ where a value goes becomes a parameter of a function nobody wrote, whose body is the smallest
scope around it: a call's argument, a bracketed group, or the value of a binding, an assignment or a
return. Every _ is a parameter of its own, left to right, which is Scala's rule and the
notation's one surprise — _ > 3 && _ < 9 is a function of two parameters, and the checker says so
where it is written. A lone _ standing as the whole of a scope is handed outward, so f(_) names f
and add(_, 1) is the partial application it reads as. A _ no scope took in is refused with a
sentence pointing at it.

Desugared in the parser into the same Lambda node a written lambda makes, so the checker, the machine
and the JavaScript back end learn nothing.

Test hooks, assertFaults and --only

@setup and @teardown run before and after each @test in a file; @setupAll and @teardownAll
run once, either side of the whole file. A fault in a setup fails the tests it guards and the failure
names the setup; a skip in one leaves them out with its reason; a teardown runs however the test
went and its own fault is a verdict of its own.

assertFaults(fn) and assertFaults(fn, message) are the assertion that cannot be written as a
condition — an async call's fault arrives a turn later, so it answers a promise the test awaits.
slate test --only <substring> runs the tests whose name contains it; a filter matching nothing says
so and exits non-zero.

slate:dom can read a page a server rendered

nodeKind, property, createComment and splitText, and before them dispatch, observe and
events. tagName answers null for a text node and for a comment alike, so a reconciler walking a
server's markup read a comment as a piece of text; attribute goes on saying what the markup said
however much has been typed into the field since, so nothing could read back what setProperty writes;
and one run of server text has to become the two children a component rendered. dispatch(node, event)
sends what on reads, observe is a MutationObserver, and events(url, options) is the reading end
of slate:http's sse.

Smaller things

• slate.sum records the whole dependency graph. A package's own manifest arrives with the
  package, so the first resolution of a cold cache saw only what the project itself declared, and
  slate deps then reported a transitive dependency as not recorded for ever. fetch_graph resolves,
  fetches, and resolves again until a round arrives with nothing new.
• slate --help and -h print the usage. They fell through to the last arm of the command line and
  answered cannot read --help: no such file or directory.
• writeBytes and appendBytes in slate:fs, with their Sync twins.
• A third position on indexOf and lastIndexOf, over both arrays and strings, with a
  Boyer–Moore–Horspool search under the string forms.
• stat's missing modified in the JavaScript host, and onSignal/offSignal there.
• close(server) cuts a stream that never ends, rather than waiting for it.
• A JavaScript back-end defect the placeholder work turned up: js_pat.sysl emitted ["_"] for
  both Wild and Bind, so val [_, second] = [1, 2] bound second to 1 under slate js and to 2
  under the interpreter.

Installing

brew update
brew upgrade slate

The formula now names zstd, lmdb and webp beside the six libraries it already did — a dependency
missing from it installs cleanly and then fails to start with a dyld error naming a path nobody typed.
SQLite is /usr/lib's rather than Homebrew's, and stb, miniz, monocypher, llhttp and QOI are vendored
C, so none of them is a line here.

The compiler floor is sysl 0.0.105.