Improving Support for Dependency Versions #1264

pyrmont · 2023-08-20T13:13:23Z

pyrmont
Aug 20, 2023

One of the limitations of Janet's code importing system is dependency versioning. Although a user can specify in a project.janet file a Git tag/commit to use, there is no guarantee that the Janet interpreter will import the file from this revision of the repository. This discussion proposes a backwards-compatible way in which Janet could allow users to specify a dependency at a particular revision.

Background

Janet source files can import other Janet source files (each, a 'dependency'). When a dependency is imported, the source file is read, parsed and compiled into values that are stored in an environment table. During importation, the public values of this table are merged into the environment table of the importing file (with the symbols typically prepended with a name so as to avoid conflicts). A Janet source file can import dependencies that are located relative to the importing file or it can import dependencies from the system library (often /usr/local/lib/janet).

Janet files are typically added to the system library using JPM, the Janet Project Manager. By listing the URLs of Git repositories in a project.janet file, a user can import the dependencies into the system library using jpm deps. JPM allows a user to import the files at a particular revision of each repository. Users do this by specifying a :tag (despite the name, the value can be a Git tag or a commit SHA).

Problem

Unfortunately, while a user can specify a revision in the project.janet file, there's no guarantee that the dependency imported at runtime is the version from the revision specified. If the user subsequently installs a different version of the file, this will simply overwrite the version in the system library.

Solution

This post proposes a two-part solution. The proposal is detailed below but the way this would be called by the user is demonstrated first.

Rather than this:

(import foo)

(foo/bar)

a user could write this:

(import foo :tag "1.2.3")

(foo/bar)

In other words, the user would specify at the call site of import the revision desired. Janet would then import a version of the file from that revision using the tag as part of the filename (e.g. foo.1.2.3.janet). If the file with that name could not be located, an error would be thrown. Existing code would work exactly the same as now; specifying a tag would be optional.

Implementation

As noted above, the solution involves two parts.

Janet interpreter

The first part is a change to the private require-1 function in Janet's boot.janet.

(defn- require-1
   [path args kargs]
+   (when (and (string/has-prefix? "." path) (dyn :tag))
+     (put kargs :tag (dyn :tag)))
+   (def apath (if (kargs :tag) (string path "." (kargs :tag)) path))
+   (def [fullpath mod-kind] (module/find apath))
-   (def [fullpath mod-kind] (module/find path))
   (unless fullpath (error mod-kind))
   (if-let [check (if-not (kargs :fresh) (in module/cache fullpath))]
     check
    (if (module/loading fullpath)
      (error (string "circular dependency " fullpath " detected"))
       (do
         (def loader (if (keyword? mod-kind) (module/loaders mod-kind) mod-kind))
         (unless loader (error (string "module type " mod-kind " unknown")))
+         (def newenv (merge-into (make-env) {:tag (kargs :tag)}))
+         (def env (loader fullpath (array/concat @[:env newenv] args)))
-         (def env (loader fullpath args))
         (put module/cache fullpath env)
         env))))

The changes here:

Set the (kargs :tag) to the value of a dynamic binding if the import is for a relative path (this dynamic binding is set up later).
Append . and the value of (kargs :tag) to the path if a value exists.
Search for the modified path.
Set the initial environment for the imported file to include the (kargs :tag) value as a dynamic binding (this is necessary for the relative paths described at 1).

No other changes are required to the interpreter.

JPM

The second part involves changes to JPM. These would cause JPM to append the tag specified in a project.janet to the Janet source files installed into the system library.

A symlink (without the tag) could also be created to point to the source file. This would allow the source file to be used by import statements that don't include a tag.

Objections

Inconsistent with other languages

Some may object that specifying the tag at the import call site is inconsistent with other programming languages. While I agree this is true, I argue that this is the preferable approach.

The author of the source file is in the best position to know which revision of the dependency they wish to use and they should be able to specify this explicitly. It should not be the case that a user of the source file can change the revision of the dependencies used (without any change to the source code). We should avoid unnecessary coupling but the revision of a dependency is necessarily coupled to the file importing it. A different revision of a dependency may change the implementation of a function used or it may change the function signature, both of which may be incompatible. By allowing an author to specify the revision, that author can know the code will not run unless that specific revision is available.

Painful to update dependency versions

When updating a dependency, a user would need to change both the project.janet file as well as the import call sites. I agree this is painful but argue that it is a necessary consequence of being explicit about the revision used.

Increased resources

This increases the disk space used, particularly as more and more Janet source files are installed in the system library. My argument would be that, in 2023, this is fine.

Non-source dependencies

This proposal has only discussed Janet source files but Janet also supports native dependencies and Janet image files. The way the code has been suggested above would allow a user to specify a tag for a native dependency.

While I don't think that would cause an issue if only one version of that dependency is imported, I don't think multiple revisions of native dependencies can be imported into the same process (but maybe they can—I'm not sure). I'm also unsure what the consequences of multiple Janet image files would be.

`init.janet` files

The above proposal doesn't work cleanly for source files that are found by appending /init.janet to the path. I don't have a great answer for this at the moment.

Thoughts

For those who read through all that, I'm curious what you think. Any thoughts? :)

ianthehenry · 2023-08-20T15:27:02Z

ianthehenry
Aug 20, 2023

I think that the problem you're describing is already solved pretty well by project-local dependency directories? If I'm understanding correctly, you only have this clobbering issue if you install dependencies to the system path, and if you don't use the system path the issue goes away for free.

The author of the source file is in the best position to know which revision of the dependency they wish to use and they should be able to specify this explicitly. It should not be the case that a user of the source file can change the revision of the dependencies used (without any change to the source code). We should avoid unnecessary coupling but the revision of a dependency is necessarily coupled to the file importing it. A different revision of a dependency may change the implementation of a function used or it may change the function signature, both of which may be incompatible. By allowing an author to specify the revision, that author can know the code will not run unless that specific revision is available.

If I'm distributing an application, I want to specify exactly the version of all of my dependencies that I expect. But if I'm distributing a library, I don't want to set exact constraints on what will become transitive dependencies for applications or libraries that use my library.

If library foo depends on some particular version of spork, and library bar depends on a different version of spork, then I can't write an application that depends on both foo and bar, and I'm sad (assuming that the version differences are trivial and the two versions are not actually incompatible).

If I'm the author of foo and bar, I really want to set constraints on the range of versions of spork that I need. (Which assumes that spork were using something like semver, which, you know, it's not even versioned, so that's maybe not the best example.)

I know that JPM gives me no way to express this right now, but that makes me sad, and I wouldn't want to get further away from what I see as the right end goal.

Painful to update dependency versions

I would add another part to this, which is that it's a bit fragile: you could update the version in your project.janet file without updating every use and import and require in your codebase, and your code would continue to work fine -- on your machine. But if installed it from scratch, it wouldn't work.

Thoughts

It seems like a heavy solution to a problem that already has a cheap solution. Maybe it's worth saying more about what you're using the system path for?

I think that Janet's packaging story has a lot of shortcomings (dev dependencies! -- I know you're aware of that one :) and there's certainly a lot of room for improvement in JPM. I haven't felt the pain that this particular proposal allays, and I think that you could get the same benefit with no change to user code with a more ambitious change to JPM's library directory structure. Which would be a backwards-incompatible change to existing installations, but would be completely transparent to the user...

11 replies

CosmicToast Aug 27, 2023

Looks like a very good start.
What's the recommended way to use this?
Let's say I want to produce an executable from script.janet which imports jurl (which contains jurl/native, a native module which depends on a pkgconfig package and jurl/* which are janet modules).
As far as I'm aware, you cannot expect a given package to be present when executing jpm as of currently, since jpm is what would normally be used to install it (so a clean janet+jpm install + clone + jpm command wouldn't have, for instance, spork available).

bakpakin Aug 27, 2023
Maintainer

Yeah, this isn't a full build system, just a wrapper around the compiler. Notice also the msvc is a separate interface because it works differently enough. I don't recommend using this right now with jpm - this is something that could/will be used to replace jpm. While you could plumb it together, I don't really see the point.

As far as I'm aware, you cannot expect a given package to be present when executing jpm as of currently, since jpm is what would normally be used to install it (so a clean janet+jpm install + clone + jpm command wouldn't have, for instance, spork available).

This is currently done in jpm with the post-deps macro in project.janet, which is just a block of code that isn't executed for certain subcommands (like jpm deps), allowing some functionality to be used even before dependencies are installed.

bakpakin Aug 27, 2023
Maintainer

As far as how to use this, to create a Janet native module (on posix platforms), the following should work (using the file json.c in spork):

(import spork/cc)
(setdyn cc/*visit* cc/visit-execute-if-stale)
(cc/search-static-libraries "m" "rt" "dl")
(cc/compile-and-link-shared "json.so" "json.c")

A build system could do more, for example cache whether or not all of the runtime libraries are needed, build both a static and dynamic version, but this is sufficient and will work in a REPL. Also, note that this should work on BSDs that don't have -lm or -lrt and will not try to link to them (I haven't actually tested this), but the idea is to auto-detect available libraries, and return any that are missing. The user can then look for dynamic versions of these libraries, or raise and error, or try to install from a system package manager, or do whatever.

sogaiu Aug 28, 2023

The last step didn't work for me. Looks like janet.h is not being picked up.

Below is a log of steps taken with output.

$ jpm install spork
remote: Enumerating objects: 69, done.
remote: Counting objects: 100% (65/65), done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 55 (delta 40), reused 37 (delta 23), pack-reused 0
Unpacking objects: 100% (55/55), 14.77 KiB | 889.00 KiB/s, done.
From https://github.com/janet-lang/spork
   a764ec1..cb4a883  master     -> origin/master
From https://github.com/janet-lang/spork
 * branch            HEAD       -> FETCH_HEAD
HEAD is now at cb4a883 Add *use-rdynamic* and *smart-libs* to control linking.
generating /home/user/.local/lib/janet/.manifests/spork.jdn...
Installed as 'spork'.
copying spork to /home/user/.local/lib/janet...
installing bin/janet-format to /home/user/.local/bin/janet-format
installing bin/janet-netrepl to /home/user/.local/bin/janet-netrepl
copying build/spork/json.so to /home/user/.local/lib/janet/spork...
copying build/spork/json.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/json.a to /home/user/.local/lib/janet/spork...
copying build/spork/rawterm.so to /home/user/.local/lib/janet/spork...
copying build/spork/rawterm.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/rawterm.a to /home/user/.local/lib/janet/spork...
copying build/spork/crc.so to /home/user/.local/lib/janet/spork...
copying build/spork/crc.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/crc.a to /home/user/.local/lib/janet/spork...
copying build/spork/utf8.so to /home/user/.local/lib/janet/spork...
copying build/spork/utf8.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/utf8.a to /home/user/.local/lib/janet/spork...
copying build/spork/tarray.so to /home/user/.local/lib/janet/spork...
copying build/spork/tarray.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/tarray.a to /home/user/.local/lib/janet/spork...
copying src/tarray.h to /home/user/.local/lib/janet/...
copying build/spork/zip.so to /home/user/.local/lib/janet/spork...
copying build/spork/zip.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/zip.a to /home/user/.local/lib/janet/spork...
copying build/spork/cmath.so to /home/user/.local/lib/janet/spork...
copying build/spork/cmath.meta.janet to /home/user/.local/lib/janet/spork...
copying build/spork/cmath.a to /home/user/.local/lib/janet/spork...

$ janet -v
1.30.0-874fd2ab

$ janet
Janet 1.30.0-874fd2ab linux/x64/gcc - '(doc)' for help
repl:1:> (upscope (import spork/cc) :done)
:done
repl:2:> (setdyn :verbose true)
true
repl:3:> (setdyn cc/*visit* cc/visit-execute-if-stale)
<function visit-execute-if-stale>
repl:4:> (cc/search-static-libraries "m" "rt" "dl")
@[]
repl:5:> (cc/compile-and-link-shared "json.so" "src/json.c")
cc -std=c99 -O2 -I/home/user/.local/lib/janet -L/home/user/.local/lib/janet -L/home/user/.local/lib -fPIC -g -DJANET_BUILD_TYPE=develop -c src/json.c -o ./src___json.c.o -pthread
src/json.c:23:10: fatal error: janet.h: No such file or directory
   23 | #include <janet.h>
      |          ^~~~~~~~~
compilation terminated.
error: command failed with non-zero exit code 1
  in os/execute [src/core/os.c] on line 1336
  in visit-execute [/home/user/.local/lib/janet/spork/cc.janet] (tailcall) on line 382, column 7
  in exec [/home/user/.local/lib/janet/spork/cc.janet] (tailcall) on line 74, column 3
  in compile-many [/home/user/.local/lib/janet/spork/cc.janet] on line 210, column 31
  in compile-and-link-shared [/home/user/.local/lib/janet/spork/cc.janet] on line 225, column 26
  in _thunk [repl] (tailcall) on line 5, column 1

$ ls ~/.local/include/janet/janet.h ~/.local/include/janet.h 
/home/user/.local/include/janet.h  /home/user/.local/include/janet/janet.h

$ ls ~/.local/lib/janet/janet.h
ls: cannot access '/home/user/.local/lib/janet/janet.h': No such file or directory

bakpakin Aug 28, 2023
Maintainer

Setting CPATH for the compiler will fix that, however I have added some more code to try and infer include paths from `(dyn syspath) .

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improving Support for Dependency Versions #1264

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Improving Support for Dependency Versions #1264

pyrmont Aug 20, 2023

Background

Problem

Solution

Implementation

Janet interpreter

JPM

Objections

Inconsistent with other languages

Painful to update dependency versions

Increased resources

Non-source dependencies

init.janet files

Thoughts

Replies: 1 comment · 11 replies

ianthehenry Aug 20, 2023

CosmicToast Aug 27, 2023

bakpakin Aug 27, 2023 Maintainer

bakpakin Aug 27, 2023 Maintainer

sogaiu Aug 28, 2023

bakpakin Aug 28, 2023 Maintainer

pyrmont
Aug 20, 2023

`init.janet` files

Replies: 1 comment 11 replies

ianthehenry
Aug 20, 2023

bakpakin Aug 27, 2023
Maintainer

bakpakin Aug 27, 2023
Maintainer

bakpakin Aug 28, 2023
Maintainer