Install it anywhere, with the graph already in it - #47
Merged
Conversation
The tool was reachable to people with a Go toolchain, a container runtime they drove by hand, and days of model calls to spend. This makes it two commands on any of the three platforms. The container steps move out of deploy/neo4j/neo4j.sh and into a Go package, as data rather than as calls. A shell script is half an answer when the project has to run on Windows too, and the alternative to one implementation is three that drift: the first time a flag changes, two of them are wrong and nobody finds out until somebody on that platform tries. Steps as values also means the argument lists can be asserted on a machine with no container runtime, which is what CI is and what most of these tests are. The graph itself is published rather than rebuilt. Running the pipeline over the corpus takes days and several hundred dollars of model calls and produces the same 8.2 million nodes for everybody, so it goes to Hugging Face as a dataset, versioned apart from the code so that a one line fix does not mean re-uploading half a gigabyte. The archive is checked against a pinned checksum before anything is unpacked, because the tool unpacks it and then runs a shell script out of it, and it is unpacked in process rather than by calling tar, because Windows has no tar. The two install scripts take a release binary, check it against the release checksums, and put it somewhere on PATH that needs no administrator.
The first macOS run failed at the import with "statfs /tmp/luatdo-verify/export/neo4j: no such file or directory", about a directory that was plainly there and held 3.4GB. The runtime on macOS is a virtual machine, and the mount source is read inside it rather than on the host. The machine shares /Users, /private and /var/folders. It does not share /tmp, because /tmp on macOS is a symlink to /private/tmp, so the path handed over resolved to the machine's own empty /tmp. Resolving the source through its symlinks before building the mount argument turns that into a mount that works, and changes nothing on the hosts where the path was never a symlink. A path that will not resolve is passed through as it was given, because the runtime's own complaint about a missing directory says more than one from here about the step before it.
The Windows CI run failed on a case the Linux and macOS runs passed: an entry named /etc/escaped.csv was unpacked rather than refused. filepath.IsAbs says that name is not absolute on Windows, it carries no volume, and Join then rewrote it to a file inside the destination. Nothing escaped, so this was not a hole, but an archive that Linux refuses and Windows accepts is worse than either answer on its own. The same reasoning covers drive letters and backslashes, which are ordinary filename characters on Linux and separators on Windows, so those are refused in both places too. The outside test also compared the whole first path element rather than a prefix, so that a file honestly named ..stray.csv is not refused for looking like an escape. There is a test for that as well, because a rule that rejects real files fails an install for a reason nobody can act on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #46.
The pipeline has been finished for a while and the graph has been good for a while, and getting to either of them still meant having a Go toolchain, driving a container runtime by hand, and spending days of model calls.
This makes it two commands on any of the three platforms.
One deployment implementation instead of three
deploy/neo4j/neo4j.shwas POSIX only, so Windows was going to get either nothing or a second copy of the same argument lists.Three copies drift.
The first time a flag changes two of them are wrong, and nobody finds out until somebody on that platform tries.
The steps moved into a Go package as data rather than as calls:
That is worth more than tidiness.
What goes wrong with a container invocation is the argument list, and an argument list that is built and immediately executed can only be checked by running it, which needs a runtime, a network and two minutes.
Returned as a slice it is checked in a unit test on a machine with no runtime installed, which is what a CI runner is.
The tests assert the things that have actually gone wrong before: that the offline importer is never handed server arguments, that
upsets the database name through configuration because Community edition will not create a second database and a server started without it comes up healthy and holds nothing, thatdowndoes not remove the volume, and that the four environment variable names match the onesgraph.TargetFromEnvreads.The runtime half is tested by re-executing the test binary as a fake container runtime, so
Run,Capture,IsRunningandReadyall get a real fork, real pipes and a real exit status on whichever platform the test is running on.That caught one bug worth having:
IsRunningmatched a container name by substring, so a staleluatdo-neo4j-oldread as a runningluatdo-neo4jand the load step refused for a reason that was not true.The graph, published
Running the pipeline over the whole corpus takes days and several hundred dollars of model calls and produces the same 8,175,346 nodes for everybody.
So it is published: open-index/luatdo-graph, 550MB gzipped, with a dataset card that says what is in it and which layers are still thin.
It is versioned apart from the code, as
luatdo-graph-YYYY.MM.tar.gz.A corpus grows when somebody runs the pipeline over more of it and the tool changes when somebody changes the tool, and tying them together means either re-uploading half a gigabyte to publish a one line fix or shipping a tag whose data is a copy of the last one.
The code carries the version of the data it was built against and says so.
The checksum is pinned and not optional.
This is a file downloaded over the network, unpacked, and then handed to a container that runs a shell script out of it.
Fetchwrites to a temporary file, hashes while downloading, verifies, and only then unpacks, so a corrupt download has not already written most of a graph into the store by the time the checksum says so.Unpacking is
archive/tarrather than a call totar, because Windows has none, and it refuses entries that are not regular files or directories and entries that resolve outside the destination.Two commands
curl -fsSL https://raw.githubusercontent.com/tamnd/luatdo/main/install.sh | sh luatdo neo4j installinstall.ps1is written for Windows PowerShell 5.1, which is what a Windows machine has before anybody installs anything.No ternary, no null coalescing, and TLS 1.2 set explicitly, because 5.1 still defaults to 1.0 on older installs and GitHub has not accepted that for years.
Both scripts check the download against the release checksums and install somewhere that needs no administrator.
luatdo neo4j installfetches, imports offline and then waits until the database will answerRETURN 1before saying it is up.Started and ready are minutes apart on a graph this size, and telling somebody to open a browser at the moment the container forks sends them to a page that does not load and reads as a failed install.
When it finishes it prints the four environment variables in the syntax of the shell they are standing in, and on Windows it prints the PowerShell form with the cmd.exe form named underneath rather than guessed at, because
PSModulePathis set in cmd.exe too and there is nothing in the environment that reliably tells the two apart.statuscounts nodes and relationships rather than reporting that the container is up.A wiped volume and a volume nobody imported into both produce a server that starts perfectly and holds nothing, and asking is the only way to tell that from a working install.
Also
.goreleaser.yamlnow writes its archive name template out rather than leaving it to the default, because the install scripts build that name themselves and a default that changes between goreleaser versions would break every install with a 404.