Skip to content

Install it anywhere, with the graph already in it - #47

Merged
tamnd merged 3 commits into
mainfrom
m20-packaging
Aug 7, 2026
Merged

Install it anywhere, with the graph already in it#47
tamnd merged 3 commits into
mainfrom
m20-packaging

Conversation

@tamnd

@tamnd tamnd commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Closes #46.

The pipeline has been finished for a while and the graph has been good for a while, and getting to either of them still meant having a Go toolchain, driving a container runtime by hand, and spending days of model calls.
This makes it two commands on any of the three platforms.

One deployment implementation instead of three

deploy/neo4j/neo4j.sh was POSIX only, so Windows was going to get either nothing or a second copy of the same argument lists.
Three copies drift.
The first time a flag changes two of them are wrong, and nobody finds out until somebody on that platform tries.

The steps moved into a Go package as data rather than as calls:

func (c Config) Load() []Step {
	return []Step{
		{Args: []string{"volume", "create", c.Volume}, Ignore: true, Quiet: true},
		{Args: []string{"run", "--rm", "-v", c.mount(), ...}, Why: "importing ..."},
	}
}

That is worth more than tidiness.
What goes wrong with a container invocation is the argument list, and an argument list that is built and immediately executed can only be checked by running it, which needs a runtime, a network and two minutes.
Returned as a slice it is checked in a unit test on a machine with no runtime installed, which is what a CI runner is.
The tests assert the things that have actually gone wrong before: that the offline importer is never handed server arguments, that up sets the database name through configuration because Community edition will not create a second database and a server started without it comes up healthy and holds nothing, that down does not remove the volume, and that the four environment variable names match the ones graph.TargetFromEnv reads.

The runtime half is tested by re-executing the test binary as a fake container runtime, so Run, Capture, IsRunning and Ready all get a real fork, real pipes and a real exit status on whichever platform the test is running on.
That caught one bug worth having: IsRunning matched a container name by substring, so a stale luatdo-neo4j-old read as a running luatdo-neo4j and the load step refused for a reason that was not true.

The graph, published

Running the pipeline over the whole corpus takes days and several hundred dollars of model calls and produces the same 8,175,346 nodes for everybody.
So it is published: open-index/luatdo-graph, 550MB gzipped, with a dataset card that says what is in it and which layers are still thin.

It is versioned apart from the code, as luatdo-graph-YYYY.MM.tar.gz.
A corpus grows when somebody runs the pipeline over more of it and the tool changes when somebody changes the tool, and tying them together means either re-uploading half a gigabyte to publish a one line fix or shipping a tag whose data is a copy of the last one.
The code carries the version of the data it was built against and says so.

The checksum is pinned and not optional.
This is a file downloaded over the network, unpacked, and then handed to a container that runs a shell script out of it.
Fetch writes to a temporary file, hashes while downloading, verifies, and only then unpacks, so a corrupt download has not already written most of a graph into the store by the time the checksum says so.
Unpacking is archive/tar rather than a call to tar, because Windows has none, and it refuses entries that are not regular files or directories and entries that resolve outside the destination.

Two commands

curl -fsSL https://raw.githubusercontent.com/tamnd/luatdo/main/install.sh | sh
luatdo neo4j install
irm https://raw.githubusercontent.com/tamnd/luatdo/main/install.ps1 | iex
luatdo neo4j install

install.ps1 is written for Windows PowerShell 5.1, which is what a Windows machine has before anybody installs anything.
No ternary, no null coalescing, and TLS 1.2 set explicitly, because 5.1 still defaults to 1.0 on older installs and GitHub has not accepted that for years.
Both scripts check the download against the release checksums and install somewhere that needs no administrator.

luatdo neo4j install fetches, imports offline and then waits until the database will answer RETURN 1 before saying it is up.
Started and ready are minutes apart on a graph this size, and telling somebody to open a browser at the moment the container forks sends them to a page that does not load and reads as a failed install.
When it finishes it prints the four environment variables in the syntax of the shell they are standing in, and on Windows it prints the PowerShell form with the cmd.exe form named underneath rather than guessed at, because PSModulePath is set in cmd.exe too and there is nothing in the environment that reliably tells the two apart.

status counts nodes and relationships rather than reporting that the container is up.
A wiped volume and a volume nobody imported into both produce a server that starts perfectly and holds nothing, and asking is the only way to tell that from a working install.

Also

.goreleaser.yaml now writes its archive name template out rather than leaving it to the default, because the install scripts build that name themselves and a default that changes between goreleaser versions would break every install with a 404.

tamnd added 3 commits August 7, 2026 10:26
The tool was reachable to people with a Go toolchain, a container runtime they
drove by hand, and days of model calls to spend. This makes it two commands on
any of the three platforms.

The container steps move out of deploy/neo4j/neo4j.sh and into a Go package,
as data rather than as calls. A shell script is half an answer when the project
has to run on Windows too, and the alternative to one implementation is three
that drift: the first time a flag changes, two of them are wrong and nobody
finds out until somebody on that platform tries. Steps as values also means the
argument lists can be asserted on a machine with no container runtime, which is
what CI is and what most of these tests are.

The graph itself is published rather than rebuilt. Running the pipeline over the
corpus takes days and several hundred dollars of model calls and produces the
same 8.2 million nodes for everybody, so it goes to Hugging Face as a dataset,
versioned apart from the code so that a one line fix does not mean re-uploading
half a gigabyte. The archive is checked against a pinned checksum before
anything is unpacked, because the tool unpacks it and then runs a shell script
out of it, and it is unpacked in process rather than by calling tar, because
Windows has no tar.

The two install scripts take a release binary, check it against the release
checksums, and put it somewhere on PATH that needs no administrator.
The first macOS run failed at the import with "statfs
/tmp/luatdo-verify/export/neo4j: no such file or directory", about a directory
that was plainly there and held 3.4GB.

The runtime on macOS is a virtual machine, and the mount source is read inside
it rather than on the host. The machine shares /Users, /private and /var/folders.
It does not share /tmp, because /tmp on macOS is a symlink to /private/tmp, so
the path handed over resolved to the machine's own empty /tmp.

Resolving the source through its symlinks before building the mount argument
turns that into a mount that works, and changes nothing on the hosts where the
path was never a symlink. A path that will not resolve is passed through as it
was given, because the runtime's own complaint about a missing directory says
more than one from here about the step before it.
The Windows CI run failed on a case the Linux and macOS runs passed: an entry
named /etc/escaped.csv was unpacked rather than refused.

filepath.IsAbs says that name is not absolute on Windows, it carries no volume,
and Join then rewrote it to a file inside the destination. Nothing escaped, so
this was not a hole, but an archive that Linux refuses and Windows accepts is
worse than either answer on its own. The same reasoning covers drive letters and
backslashes, which are ordinary filename characters on Linux and separators on
Windows, so those are refused in both places too.

The outside test also compared the whole first path element rather than a
prefix, so that a file honestly named ..stray.csv is not refused for looking
like an escape. There is a test for that as well, because a rule that rejects
real files fails an install for a reason nobody can act on.
@tamnd
tamnd merged commit 5b625ae into main Aug 7, 2026
5 checks passed
@tamnd
tamnd deleted the m20-packaging branch August 7, 2026 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

M20: Install it anywhere, with the graph already in it

1 participant