Rework racket based runtime/compilation #4675

dolio · 2024-02-06T19:30:04Z

This PR includes the changes that get us about as far as we can easily go for our racket evaluation/compilation at the moment. It includes the following aspects.

I renamed the runner file/executable to ucr, so that it's more similar to ucm. It's not something that people should be calling directly, but runner is obviously not a very good name.
I changed the 'compilation' part of ucr to just generate a file. The reason for this is that somehow a math library we are using for the primops causes create-embedding-executable to fail. The easiest solution to this problem is to divide the compilation process into two steps: first generate the file using all the unison infrastructure, then compile that file. At first I was thinking about copying the math operations into the primops, but I realized that dividing the programs up is a better solution.
At the moment, compiling the generated file is just done via raco exe using a racket installation. The reason for this is that create-embedding-executable does not seem to work in a standalone executable. The API needs to find a copy of the racket executable to modify, and the procedure for that doesn't work in a standalone binary (confirmed with racket devs). Peeking at the code, it doesn't seem like this absolutely couldn't work, but making it work would require someone digging into the racket internals to make it happen. So for the time being, compiling requires a racket install.
I've included a short transcript that will download base, @unison/internal, and generate the racket modules alongside the checked-in modules. This is a step necessary in building ucr or getting things set up for compilation, so we'll probably use it when we get this stuff building in CI.

Even for compilation, where we still depend on a racket install, we no longer actually need to make people download @unison/internal themselves. We can run the library generation as part of a build, and just distribute the generated libraries to people. The runtime should be completely self-contained, though. Once ucr is built, ucm just needs to shell out to it to execute run.native. A racket install is only necessary to build ucr.

I think one tweak I have left is a better error message for compile.native if raco doesn't exist. Beyond that, I just need a little advice on how to get this hooked into CI to get everything distributed properly. I might need to make the native runtime calls a bit more configurable with respect to the location of ucr and such, too, but that shouldn't be a big deal.

Switch compile.native to using the native Runtime Tweak the call for compilation to use `racket runner.rkt`, since it seems like _compiling_ runner.rkt precludes the use of create-embedding-executable, at least until something further is figured out. runner and other racket infrastructure additions for producing the right input for racket executable creation

There are, for the moment, multiple problems with producing executables. One is that the API doesn't work when certain math libraries are imported. The other is that the API doesn't work in a generated executable. So, I've turned the 'compile mode' of the standalone runtime program just generate a module to be compiled. Then we can use an installed `raco` to build that module into a standalone program. I don't think there's any advantage currently to not using raco, since the best we can do is write our own separate exe builder that still requires racket to be installed. In any case, the exe builder must apparently be a separate program so that it doesn't need to import the offending math library.

aryairani · 2024-02-06T20:18:43Z

I renamed the runner file/executable to ucr, so that it's more similar to ucm. It's not something that people should be calling directly, but runner is obviously not a very good name.

I'm thinking we could use a more descriptive name and stick it in libexec if nobody needs to call it.

I changed the 'compilation' part of ucr to just generate a file.

What did it do before?

I've included a short transcript that will download base, @unison/internal, and generate the racket modules alongside the checked-in modules. This is a step necessary in building ucr or getting things set up for compilation, so we'll probably use it when we get this stuff building in CI.

Even for compilation, where we still depend on a racket install, we no longer actually need to make people download @unison/internal themselves. We can run the library generation as part of a build, and just distribute the generated libraries to people. The runtime should be completely self-contained, though. Once ucr is built, ucm just needs to shell out to it to execute run.native. A racket install is only necessary to build ucr.

Can we just do all of this in CI? Build ucr there, and nobody needs to install racket outside of CI?

I think one tweak I have left is a better error message for compile.native if raco doesn't exist. Beyond that, I just need a little advice on how to get this hooked into CI to get everything distributed properly.

We can pair on the CI part if you want.

unison-cli/src/Unison/Codebase/Editor/HandleInput.hs

dolio · 2024-02-06T20:28:18Z

I'm thinking we could use a more descriptive name and stick it in libexec if nobody needs to call it.

Any suggestion on the name? I guess it could be something more like unison-runtime or something.

What did it do before?

It generated a file then threw errors. But the command line gave you the expectation that it would produce an executable.

Can we just do all of this in CI? Build ucr there, and nobody needs to install racket outside of CI?

You can do the equivalent of run. Or you can generate a racket source file, but that's useless without racket. If you want to build a standalone executable from some unison code, you need racket installed, because the API for generating executables doesn't work when you build it into a standalone executable.

ChrisPenner · 2024-02-07T19:03:16Z

parser-typechecker/src/Unison/Runtime/Interface.hs

+      ucrError (_ :: IOException) =
+        die
+          "I had trouble calling the unison runtime exectuable.\n\n\
+          \Please check that the `ucr` executable is properly\


Would be great to link to a guide in these errors, as an end user I wouldn't know where to start on ensuring "ucr" is installed and google probably won't help.

Yeah, I'd like to make them better. I'm not exactly sure what the install process will be right now, though. Hopefully I can flesh these out after I work out some details with Arya.

I'm anticipating that either our installers will set it up correctly for them, or they're in big trouble and need to come ask for help, which will help us flesh out better error messages :-\

ChrisPenner · 2024-02-07T20:44:17Z

Looks great! I'd second the idea of changing ucr to something like unison-runtime

All the infrastructure was there already, just needed to enable it. This should break the jit tests in CI until we can get the runner building as a prior step, and coordinate the location the native runtime looks for the build files.

recommend moving support scheme files to another repo if they aren't deeply related to ucm; but maybe they are.

but bash tricks can maybe?

and hope for the best

tests results are cached with the ucm binaries

aryairani

I'm gonna doublecheck CI again in the morning, but I think this is ready to merge even though we may do some additional tweaks later.

aryairani · 2024-03-07T05:15:21Z

Okay, I think the tests in @aryairani/jit-tests should pass with this branch.

I also added compiler.generateSchemeBoot to @unison/internal in the new release 0.0.11 that this depends on. I think you were planning on doing something with that in the transcripts?

Thanks for the reminder, got it in 20a0e73.

…nto topic/native-compiler

existing ucm commands can be used instead, i.e. `gen-racket-libs.md`

aryairani · 2024-03-08T19:10:46Z

~~The default runtime location is set in fixNativeRuntimePath, currently as $XDG_DATA/unisonlanguage/libexec/unison-runtime, and can be overridden by nativeRuntimePathFlag.~~

It should probably default to a system-wide location though, like $0/unison-runtime?

The default runtime location is set in fixNativeRuntimePath, as $0/runtime/unison-runtime[.exe], and can be overridden by nativeRuntimePathFlag.

aryairani · 2024-03-08T19:25:53Z

A racket install is only necessary to build ucr.

The racket install is still needed to use compile.native, right?

dolio added 4 commits January 16, 2024 15:24

Rename runner, add docs, delete dead code

958b8b9

Add a transcript to create generated racket libs

2e98e1a

dolio requested review from pchiusano, aryairani and ChrisPenner February 6, 2024 19:30

aryairani approved these changes Feb 6, 2024

View reviewed changes

unison-cli/src/Unison/Codebase/Editor/HandleInput.hs Outdated Show resolved Hide resolved

unison-cli/src/Unison/Codebase/Editor/HandleInput.hs Outdated Show resolved Hide resolved

dolio added 4 commits February 6, 2024 16:28

Give a nicer looking error for ucr/raco problems

c3c62c4

Eliminate commented dead code

72743c5

Add an improved error message for unison native runtime failure

788e082

Add information on further building to gen-racket-libs.md

7ce4ad2

ChrisPenner reviewed Feb 7, 2024

View reviewed changes

dolio added 5 commits February 7, 2024 16:10

Actually make run.native use the standalone runner

a678f55

All the infrastructure was there already, just needed to enable it. This should break the jit tests in CI until we can get the runner building as a prior step, and coordinate the location the native runtime looks for the build files.

Merge remote-tracking branch 'origin/trunk' into topic/native-compiler

5f946e2

Rename ucr to unison-runtime

6703193

Add info file for unison racket package

9a253e8

Native runtime location argument, and an improved error msg

038550c

This comment was marked as outdated.

Sign in to view

Arya Irani added 8 commits February 11, 2024 21:02

refactor ci.yaml with jit setup steps (attempt 1)

b3cad2c

recommend moving support scheme files to another repo if they aren't deeply related to ucm; but maybe they are.

yaml

61d63a6

unyaml

221ebb8

yaml home

eb845f1

yaml unhome

296333d

yaml move env

f96d8ce

yaml matrix

cf91e87

yaml can't define vars in terms of vars

5dde584

but bash tricks can maybe?

dolio and others added 12 commits March 4, 2024 11:52

Bump gen-racket-libs version for @unison/internal

aa53784

Update ci.yaml with jit version 0.0.11

197ce9a

disable windows jit integration tests until #4705

3433362

yaml: temporarily print what's in the cached system stack dir

28dd96b

yaml: add test caching and change a bunch of caching stuff

8926e19

and hope for the best

yaml: forgot a plugin name

eb33604

yaml: temporarily print what's in the stack cache, better

4e22249

yaml: fix test results caching?

62eab49

tests results are cached with the ucm binaries

yaml: bikeshed variable names and hope nothing broke

47c1826

yaml: these were supposed to be part of a previous commit

4a08bed

yaml: call generateSchemeBoot directly from @unison/internal

20a0e73

yaml: cache key vars for invalidation

e026d13

aryairani approved these changes Mar 7, 2024

View reviewed changes

Arya Irani and others added 7 commits March 8, 2024 11:52

CI docs

4dc1023

Update ci.md

98eb284

Update ci.md

b0e8c76

Update ci.md

236d895

yaml: clarify that you shouldn't have to invalidate build caches

db3e48f

Merge branch 'topic/native-compiler' of github.com:unisonweb/unison i…

8df8744

…nto topic/native-compiler

remove compile.native.{fetch,genlibs} commands

b7530a5

existing ucm commands can be used instead, i.e. `gen-racket-libs.md`

Arya Irani added 5 commits March 8, 2024 14:50

yaml: update stack caching

021677d

do we need base.output.md?

e8e610d

change default ucr location to $0/runtime/unison-runtime[.exe]

5034bf4

yaml: maybe fix stack caching condition again?

bf3a99f

yaml: predicate requires British spelling

9f2645d

aryairani merged commit 27b29c1 into trunk Mar 8, 2024
12 checks passed

aryairani deleted the topic/native-compiler branch March 8, 2024 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework racket based runtime/compilation #4675

Rework racket based runtime/compilation #4675

dolio commented Feb 6, 2024

aryairani commented Feb 6, 2024

dolio commented Feb 6, 2024

ChrisPenner Feb 7, 2024

dolio Feb 7, 2024

aryairani Feb 17, 2024

ChrisPenner commented Feb 7, 2024

This comment was marked as outdated.

aryairani left a comment

aryairani commented Mar 7, 2024 •

edited

Loading

aryairani commented Mar 8, 2024 •

edited

Loading

aryairani commented Mar 8, 2024

Rework racket based runtime/compilation #4675

Rework racket based runtime/compilation #4675

Conversation

dolio commented Feb 6, 2024

aryairani commented Feb 6, 2024

dolio commented Feb 6, 2024

ChrisPenner Feb 7, 2024

Choose a reason for hiding this comment

dolio Feb 7, 2024

Choose a reason for hiding this comment

aryairani Feb 17, 2024

Choose a reason for hiding this comment

ChrisPenner commented Feb 7, 2024

This comment was marked as outdated.

aryairani left a comment

Choose a reason for hiding this comment

aryairani commented Mar 7, 2024 • edited Loading

aryairani commented Mar 8, 2024 • edited Loading

aryairani commented Mar 8, 2024

aryairani commented Mar 7, 2024 •

edited

Loading

aryairani commented Mar 8, 2024 •

edited

Loading