Make versioned .julia/v0.x dir by default.

[kmsquire]

This addresses JuliaLang/METADATA.jl#264.

Pkg already had the ability to deal with julia-versioned package directories (.julia/v0.x), but didn't create them by default. This patch causes that to happen. Currently existing .julia directories will continue to work.

The same fix was made in base/pkg2, so the upcoming Pkg update will have the same behavior.

Note that, unfortunately, git submodules don't work well with being moved, so for existing installations, the best solution is to blow away .julia and start afresh.

cc: @StefanKarpinski

[kmsquire]

The build failure shouldn't have anything to do with this request. Tests for the clang version didn't complete for some reason.

[ViralBShah]

I think it is actually a good idea to create versioned package directories by default. Simplifying this will also encourage package developers to support older released versions of julia easily.

[kmsquire]

Bump. This isn't really necessary at this instant, because so few people were using v0.1, but I really think you will be saving yourselves and your users a lot of headaches in the future by making this a policy earlier rather than later. At v0.3, you will have people that will have both versions of julia on their systems, because they want to test out the new version, see if things break, etc., before switching, and sharing an unversioned .julia directory between them isn't going to work well in most cases.

[StefanKarpinski]

I just can't see more than a smallish minority of users ever having multiple versions of Julia installed. For those users, we already support using versioned .julia directories and that seems like enough to handle what I can't picture being a typical scenario. It also does seem to me that having two different Julia versions using the same .julia directory being the end of the world. All you have to do is Pkg.resolve() and you'll be the versions that work for the current Julia. @stevengj – do you have an opinion here? You were very vocal about versioning system directories that go into LOAD_PATH.

[stevengj]

I agree with @kmsquire. Avoiding mysterious breakage in corner cases is worth an extra few characters in the path.

[StefanKarpinski]

Ok, I'm outvoted. I'm going to test this out and merge if there aren't any problems.

[StefanKarpinski]

Ok, this definitely has conflicts now; please rebase.

[StefanKarpinski]

I also wonder about the appropriate level of granularity here. Should we really have separate versioned .julia directories per minor version? Would it make more sense to have minor only when the major is zero and major-only otherwise? I.e. v0.1, v0.2, ... v1, v2, ...

[stevengj]

It needs to include the minor version, because minor versions may add features that are not forwards-compatible. (e.g. there may be package versions that work with 1.3 but not with 1.2.) We went through the same discussion already for /usr/share/julia/VERSDIR ...

[kmsquire]

I'll try to get to this tonight. I'm wondering if the package cache should
be shared or separate. Thoughts?

[ivarne]

Do you suggest sharing .git/ directories and METADATA repository between the folders for the different versions?

[kmsquire]

I think we could share the package cache (which is a set of git repos), but
keep the packages for different Julia versions (which are also necessarily
git repos) separate.

I'm thinking each version should have its own METADATA, because when Julia
v10 comes out, it won't make much sense for it to see versions of
packages which
only worked with v0.2 (and keeping all of those files around will likely
flow things down).

[kmsquire]

Okay, I've rebased the pull request. As of right now, it uses a shared cache dir, but each version v$major.$minor has its own METADATA and set of packages.

Some comments

1. this commit does not force any upgrades to a versioned package dir; existing non-versioned directories will continue to work (and be used through the next upgrade, etc.)
2. this does introduce the complication that, upon any version update (major or minor), all packages will have to be re-added (or there needs to be a mechanism provided to do so manually or automatically)

[StefanKarpinski]

I wonder if we shouldn't set the .cache up as a symlink, which would allow manually separating the caches.

[StefanKarpinski]

What's up with the commented out warning?

[kmsquire]

A symlink for the cache dir sounds good.

The warning is commented out because the function to migrate to a versioned package dir doesn't exist yet (sorry, should have probably just removed that).

I'll update the pull request.

[pao]

Is a symlink going to be a problem on Windows, or is this not going to be used there?

[kmsquire]

Whoops, got simonster's commit mixed up in there...

[kmsquire]

Sorry, this took a little while (real work, life, yadda yadda...). I was trying to figure out the best way to upgrade non-versioned to versioned, add Windows symlinks, etc., but finally realized that all that was holding up this relatively trivial patch, and that stuff can be taken care of in later patches.

Is a symlink going to be a problem on Windows, or is this not going to be used there?

For right now, it's not used, but I submitted #4396 to add support for symlinks on Windows.

[StefanKarpinski]

Right, we should probably do this one too for 0.2. Almost forgot.

[StefanKarpinski]

Unfortunately, probably needs yet another update to deal with recent WINDOWS changes.

[kmsquire]

I rebased, but AFAICT, the recent Windows changed only updated user_homedir() (and associatively, user_prefdir()), and the patch just uses whatever value user_prefdir() provides.

The patch doesn't make any attempt to move old versions of the dir, and on Windows, it doesn't attempt symlinking (#4396 needs a slight update won't be merged until after 0.2 is released, and it's unclear what to do about Windows XP, which doesn't support symlinks).

Otherwise, the patch is pretty minimal. Is there something else it should do or needs to take care of?

[StefanKarpinski]

I'm still reluctant to merge this, for the following reason: I want to add support for per-project package directories. This is like a .git directory in a project that is found by looking in the current directory, then the parent directory, etc. If none of the parent directories have a .julia, then you look at ~/.julia instead. The idea here is that each project can have its own set of installed packages and versions, rather than sharing them between all the projects on your system. This prevents the requirements of one project from affecting the package versions used by another, and lets you upgrade the packages each project uses independently.

Let's say I'm using a single shared set of system packages and I feel like using the latest and greatest for project A and so I upgrade and fix any problems that might result. Now projects B, C, D, etc. might be in a broken states because they were relying on the system packages and I didn't check to see if they were messed up by the upgrade. Obviously, this is not good. With per-project packages, this doesn't happen since I choose the upgrade the packages each project is using independently.

With per-project .julia directories, the shared cache becomes even more important and this current symlink approach probably isn't a great way to go (sorry, I know it was my suggestion – we still should have the symlink functionality in base, so that's not wasted effort). Instead, Pkg should probably just know that .cache doesn't live under .julia but lives in some shared place. Moreover, we need to think about what caching should be keyed on. What if different projects have METADATA directories with different origin servers? Should they share a cache then? It might be ok since git repos are content-addressable and there can't be much in the way of interference even between divergent copies of the same repository. But what if two different packages by the same name occur in different METADATA.jl instance? I'm not sure that's worth handling, but it is something to consider.

Another thing to consider is where binary dependencies live. They probably should not live where they do now, inside of the installed package. That implies that when you have multiple .julia directories, they can't share built binary dependencies. I really don't want to have to rebuild libcairo for every project that wants to be able to plot stuff. So binary dependencies should also live somewhere shared like .cache. But we want to make sure that multiple different versions of binary dependencies can coexist there at the same time, in case we have different versions of Cairo.jl installed that don't want the same version of libcairo.

With per-project .julia directories, it becomes somewhat unclear what to do about .julia/v1 versus .julia/v2. Does that only apply to ~/.julia? Or does it apply to each project .julia? Or can we reasonably assume that each project only aims to support a single version of julia? I'm not sure and I'm afraid that merging this will just make it harder to address all these problems going forward.

[JeffBezanson]

Isn't having multiple .julia directories orthogonal to having version-specific directories inside .julia?

[StefanKarpinski]

Somewhat, but they both entail shared .cache directories, among other design issues. I just don't want to merge a half-designed version only to break everything later. Given that currently there is exactly one version of Julia that we're supporting packages for, this seems like something that can wait a bit.

[JeffBezanson]

OK.

[StefanKarpinski]

@kmsquire – I think we need to go ahead with this now. Could you rebase?

[kmsquire]

Sure, will get to it tomorrow.

[ssfrr]

Re: per-project .julia directories - I think that this is a great idea, and something that in Python you need a separate package (virtualenv) to achieve. It's a common problem and it would be great to have it included in the language.

I'm not necessarily sold on the value of shared binary deps between project-local package stores. I feel like you either want a sandboxed version of a package or you don't. For me I'm willing to wait a little longer for a package's binary dependencies to build for the simplicity and peace of mind that there's no interaction between projects.

For the most part it seems like the trend is to not require users to compile binary dependencies anyways, and instead to provide pre-built binaries, right?

[StefanKarpinski]

I was mostly concerned about very large binary dependencies like libcairo and such. Having a copy per project seems excessively wasteful.

[kmsquire]

Rebased.

[kmsquire]

Two more thoughts:

1. This should probably be backported to the 0.2 branch
2. It would be nice to have a way to migrate from a non-versioned package dir to a versioned package dir, preferably in a package so that the migration code is short lived and doesn't rot in the Julia source.

[StefanKarpinski]

Ok, those can both be taken care of after this is merged. I have a probably-undue amount of dread about merging this, but I'm just going to do it and see what happens...

[kmsquire]

I have a probably-undue amount of dread about merging this, but I'm just going to do it and see what happens...

I recommend breathing. ;-)

[StefanKarpinski]

I have PTSD from developing the package manager.

[johnmyleswhite]

I'm not surprised.