RFC: fix package usages of Pkg.dir()

[MikeInnes]

Many packages use Pkg.dir() to find load static resources that aren't Julia code. However, Pkg.dir() always points to ~/.julia/v0.x, so this breaks for packages which are stored in julia/usr/share/julia/site/v0.x, for example. It looks like we need a better way to handle this if there isn't one already.

Are packages that use Pkg.dir() for this purpose in the wrong, or should Pkg.dir() be smarter? Or should we just only support a single package directory?

[tkelman]

Duplicate of #8679 - generally packages should use @__FILE__ to refer to their own location rather than Pkg.dir, with the possible exception of if they need to ensure a writable location. For referring to the location of other packages, it's less clear what should be assumed.

[MikeInnes]

That may be the cause of this issue but it's not the same thing. What Pkg does or doens't support is one thing, but if use of Pkg.dir() is incorrect then a huge number of packages across the ecosystem – I count 676 uses overall – are broken. It seems valid to talk about solving that particular problem (clearer docs? PkgEval testing?) outside of what happens with #8679.

[tkelman]

Also true. That's sort of where the discussion in #8679 went, but the manual changes that were made based on that discussion only caught a few packages. We could probably try running PackageEvaluator in a non-default-location mode and see what massive breakage ensues. This could be reopened, but probably as a documentation issue (or to track potential changes to the loading mechanism?) as far as what's actionable within base Julia.

[wildart]

BTW, when did we start using julia/usr/share/julia/site/ directory for packages?

[tkelman]

it's been on LOAD_PATH since #2550, though I don't think very many people have been trying to use it yet

[vtjnash]

since this thread is currently short and fairly related, I'm going to try to hijack this for my own purposes now. referencing tkelman's old "common wisdom" from #8679 (comment), #8745 has some pretty significant conflicts with the usage of @__FILE__ and Pkg.dir(), and often source_path() too.

here's my current ramblings, with a hope that someone will have a better idea:

let me start by splitting the current usages of @__FILE__ into 4 categories so that they can be addressed independently:

1. static resources (e.g. julia code, help files, READMEs, templates, images)
2. configuration files (e.g. ini files)
3. external package feature / version / existance detection
4. storing dependencies (e.g. deps/usr/lib)

For (3), I still think the right solution is move forward with #6884, and it doesn't seem like anyone has disagreed.

For (1), this is handled pretty well by source_path() (load path-relative make sense, although it might be useful to abstract FilesytemPath into a separate type so that cross-node loading can be handled transparently)

For (4), I'm starting to wonder if we should introduce a standard ~/.julia/v0.x/usr/ folder that is shared by all packages. But shared state scares me silly, so maybe something more like http://nixos.org/nix/ would be amazing?

• Aside: maybe just use Nix for everything? it seems to have all of the right buzz-words on its homepage to allow it to completely replace BinDeps/Homebrew/WinRPM/AptGet/Etc. with something more reliable (yep, even windows, to the extent that the windows kernel allows: https://invalidmagic.wordpress.com/2011/02/14/bringing-nix-to-its-limits/). It would be cool to hear if anyone has some actual experience trying their system (Or fork it on Github! https://github.com/NixOS/nixpkgs)

For (2), I think this actually has two parts (yay, nested lists!): location and installation/upgrade/change mechanics. If there is a ~/.julia/v0.x/usr/etc and ~/.julia/v.x/usr/share folder (and appropriate corresponding global locations for system packages) from (4), then I think the where question is pretty simple. but the question of installation is really tricky then. It seems like this calls for a way of installing packages that also handles the necessary file moves, dependency tracking, versioning, and and merge questions. Or Nix?

@StefanKarpinski since most of these issues are also highly relevant for your proposal to pull package code straight out of git history, in order to provide the reproducible versioning behavior (similar to Nix, but for a single package).

[ssfrr]

It would also be great to have a standard place for tools/applications written in Julia. If there's going to be a ~/.julia/v.x/usr/share then maybe a ~/.julia/v.x/usr/bin that people can put on their $PATH. Then a user could do Pkg.add("GreatShellTool") and immediately be able to run said tool. I think that pip just puts them into /usr/local, so there's precedent for that as well, though it needs to be user-writable or requires sudo.

I don't want to double-hijack so feel free to ignore if this is OT.

[tkelman]

I'm starting to wonder if we should introduce a standard ~/.julia/v0.x/usr/ folder that is shared by all packages

Maybe a common install prefix. I think there'd also need to be a common parent build-dir that is not contained within each separate Julia package's source, since we either need to keep the build tree around to be able to do reliable uninstalls if we're working in a shared install prefix, or adopt some existing package manager's installed files manifest convention. We probably need uninstall hooks somewhere in the system.

Aside: maybe just use Nix for everything?

I'm gonna say no. NixOS works pretty well as a Linux distribution. Nix as a package source on top of an existing system, especially a non-posix one, is not something I feel remotely comfortable imposing on users right now. The few times I've tried to set up a Nix sub-environment within an Ubuntu system I've given up, and I've heard (from Haskell people, who have even more incentive to like Nix!) that it's still not perfect on Macs either.

What we really need is a hosted system that automates building (and re-building) and distribution of binaries for each major platform. I'm really extremely happy with the opensuse build service as a platform for cross-compiling, in the places that makes sense. Anaconda.org (nee Binstar) only provides an automated build service from Linux (and isn't set up well for cross-compiling) on their open-source plan, last I checked.

[stevengj]

Simple question: why use dirname(@__FILE__) at all, as opposed to just using relative paths? I see lots of packages using e.g. include(joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")), for example, but wouldn't include(joinpath("..", "deps", "deps.jl")) work just as well?

[vtjnash]

they are roughly equivalent, since include is defined in terms of calling source_path(). the general intent of this issue is to elicit ways of locating resource files that are not julia code.

(personally, i would like to provide nix-like declarative configuration as the basic framework, and merge something like #2716 so that the management of any configuration data and resources can be done explicitly by Base.compilecache)

[staticfloat]

@stevengj Those two are not equivalent when not explicitly using include(). When I run using Homebrew and Homebrew.jl attempts to discover the location of the Homebrew/deps/usr folder, if I use joinpath("..", "deps", "usr") from the main homebrew.jl file, it uses the interpreter's current directory, rather than the source file's location. It's only include() that brings in source_path().

Just spelling this out for others reading this as I didn't realize this at first, and Jameson's explanation wasn't in my face enough to really drive it home.

[stevengj]

I like the suggestion from #18329 to add a @__DIR__ macro.

[tkelman]

There are still a few dozen packages that I haven't gotten to yet still using Pkg.dir that need fixing, but that's not an action for this repo.