-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Package mountainlab (+plugins) using conda #15
Comments
@magland please assign this issue to me |
I think this should default to being totally self-contained, using conda to get nodejs and mongodb within the environment, and setting up a mongodb within the env as well. Then if users want to instead use a system-wide mongo instance that can be a configuration option. |
Made some initial attemps here to package up qt-mountainview (to start): https://github.com/tjd2002/qt-mountainview/tree/conda-packaging The built package is at: https://anaconda.org/franklab/qt-mountainview Currently it is only built for linux, just pulls in conda's qt, doesn't depend on mountainlab-js, and hasn't been tested on other machines. For now, this conda package installs the When we conda-package up mountainlab-js proper, we'll need to configure it to look in that location, and not use the ~/.mountainlab/ directory for anything, since this would break the isolation which is the whole reason for this approach. These changes are being tracked at #14 |
Cool! Will check it out later in the week. |
The trickiest part of this is going to be dealing with npm (which is itself a package manager) within conda. It turns out jupyterlab relies on npm (or actually facebook's drop-in replacement for npm, called yarn) for installation of plugins, and the jupyterlab devs have been working out the implications of this for distributing their software using any mechanism other than npm itself (especially conda) for the last year or so). It looks pretty hairy! I don't understand all the nooks and crannies, however most of the complication they are hitting seems to stem from:
I think that we can avoid almost all of this hassle in mountainlab-js, because we don't really have an 'extensible' application with true plugins (requiring compatible versions of shared libraries). Instead, ML-Js and each ML package is a completely separate beast, with
|
Good point, npm will be present for all users, but it adds complexity if we use it as part of the conda-install process (consider: managing uninstallation, npm package updates resulting in different version of js libraries for the same version of our conda package...). There are workaround for these (uninstall hooks, package-lock.json...) but overall it means we have to manage and integrate 2 different package managers instead of just one. |
I did a lot of work towards this in #27. Regarding package size, once I pruned all the devDependencies, the total size of the (zipped) package is now down around 5MB! (This could change if it turns out we do need to ship webpack, or if I otherwise screwed up the packaging). I also did some research and discovered that option 1, is the preferred way to do this (I think for reasons that I laid out above). In particular, conda-forge packages up lots of npm packages this way. You can see how they do it at this GitHub code search link. Note that the Some of those recipes use |
@magland has simplified the npm packaging even further, which makes conda packaging even more straightforward. (In particular it allowed me to circumvent the npm bug I ran into, so I can use the neat trick of using npm pack+install to emulate a true npm install through the npmjs registry). The mountainlab-js conda package is down to about 2MB, so the file-size worries above turn out to be misplaced for now. We'll see what happens when I try to package up ephys-viz using this strategy. Remaining issues before closing:
|
Making good progress on all of these, plus a few more. I've created a new repo tjd2002/mountainlab-conda to contain all the recipes in one place (will help with creating coherent 'releases' of multiple packages at once); once that's ready I'll remove the conda recipes that are already in some of the repos. There's also a spreadsheet at https://docs.google.com/spreadsheets/d/1hFiNEQWope6t_IN-sSGIZa1oQ-_jXVCXwSN3wlYSHmg/edit#gid=0 with information on all the different software components that currently go into mountainlab/mountainsort. Notably, we are planning to package and ship old versions of vis (qt-mountainview, qt-mountaincompare) and processing packages ( |
OK, we now have a working conda install pathway! Missing is documentation, but that's already covered in #37 There is also now a 'flatiron' channel on Anaconda.org: https://anaconda.org/flatiron which contains the latest conda packages, including a metapackage called "mountainsort" that installs mountainlab-js, all the needed processor plugins for sorting, and various dependencies. |
Once we figure out how to run mountainlab in an isolated env (#14), consider whether we can package it using conda.
Goal is to be able to run:
conda install -c flatironinstitute mountainlab-js mountainlab-ephys-plugins
...or similar, and have all dependencies pulled in and configured, binaries on path, etc.
Nodejs and mongo are both already provided as conda packages, and conda claims to support distribution of js apps, so I think this might be doable without heroics.
The text was updated successfully, but these errors were encountered: