Allow user to set an install location for modules #55
Conversation
The preferred package install location can be set using the %install-location magic command, which points to the destination directory for compiled Swift modules. This allows the user to compile and install packages in a notebook (using the %install directives), and then import them in a different notebook without recompiling. The compiled packages are built as dependencies of a `jupyterInstalledPackages` target. Note that compilation generates a dylib with that target, that must be `dlopen`ed before pre-built packages can be imported. For this reason, the `.so` file is also copied inside the package install location, and lazily opened when the Swift process is started. Limitations: - Build artifacts are not cached. When restarting a notebook, comment or skip installation directives (but keep `%install-location`) to prevent the recompilation of packages previously built.
Run install_location_install first, then verify that the packages are available for install_location_use.
There was a problem hiding this comment.
Thanks, this seems like it will be very convenient to use!!
One principle that I think is very important is: Notebooks should not depend on external state (as far as possible). This way, people can download and run notebooks without needing to set anything up.
This change doesn't quite follow this principle, because it lets you write notebooks that require packages to be installed in a certain directory.
I think it'll be easy to update this PR to make notebooks self-contained: Just reuse the build artifacts, and call swift build every time. I've plated with swift build a bit, and it seems pretty good at recognizing when dependencies have changed and only rebuilding what is necessary. So we'll still get a large package installation speedup.
Sure. That was actually the approach I first tried in pcuenca@81574b5. I can confirm the speedup. The problem of that revision was the whole concept of shared environment, I like the idea of allowing the user to define the location that we are doing here. |
Yes, I also like allowing the user to define the location. My suggestion is to put the intermediate build artifacts there too, and reuse them if there are already any artifacts in that user-defined location. |
This reverts commit 118dd8b. I could have rebased and force-pushed, but I'm not sure about the PR policies in place.
The Jupyter notebook library target will be rebuilt every time. However, SPM properly determines the dependencies that need to be rebuilt, so no unnecessary download / compilation steps should be necessary. Regenerating the target is much better for consistency, and ensures the notebook does not rely on external state. In order to implement this change, an additional container directory has been created, and that container is symlinked to the user-specified location when appropriate.
|
I think I addressed all the issues in my last three commits. |
…upyter into package-install-location
|
Oh no, test failure. I think it's because this branched off an older version of |
The preferred package install location can be set using the
%install-locationmagic command, which points to the destination directory for compiled Swift modules. This allows the user to compile and install packages in a notebook (using the%installdirectives), and then import them in a different notebook without recompiling.The compiled packages are built as dependencies of a
jupyterInstalledPackagestarget. Note that compilation generates a dylib that must bedlopened before pre-built packages can be imported. For this reason, the.sofile is also copied inside the module install location, and loaded when the Swift process is started.A lock is taken while building and installing modules, to prevent conflicts between notebooks running simultaneously.
Limitations:
%install-location) to prevent the recompilation of packages previously built.jupyterInstalledPackages.somust depend on the modules that are going to be later used, possibly in a different notebook. This means that all packages should be built in the same notebook session.