don't install into package directory

[stevengj]

Currently, BinaryProvider downloads into MyPackage/deps/downloads and installs into MyPackage/deps/usr. With Julia 1.0, every version of MyPackage installs into its own directory, which means that every update installs a new copy of the binary dependencies even if they didn't change.

I don't think this is acceptable in the long run. Proposal:

• BinaryProvider should install binary dependencies into a global SHA-addressed directory, e.g. ~/.julia/BinaryProvider/{MyPackage-UUID}/sha, so that they are shared between versions.

• Some kind of hook should be added to Pkg.gc() to allow binary dependencies that are no longer needed to be deleted.

[fredrikekre]

I think that the idea is that each builder will be a separate tiny package, that installs the binary dependency in its directory. Those small packages should have much fewer releases than the actual packages that uses them, so I think that will help, if not completely fix, this issue.

[stevengj]

I thought "builder" referred to the repo using BinaryBuilder, which is usually not a Julia package at all?

But I get your point — one way to resolve this is to split every package using BinaryProvider into two packages, one that provides the binary and one that uses it. (Note that this doesn't seem to be documented as a best practice.)

This is likely to be a slow process since there are ~50 packages that use BinaryProvider at the moment, and hardly any of them seem to split their dependencies off like this. Is there a way to automate it?

[fredrikekre]

@staticfloat is working on this new infrastructure, see e.g. https://github.com/JuliaPackaging/Yggdrasil

[stevengj]

@fredrikekre, that seems totally unrelated — those are the builder repos, i.e. the repos used for BinaryBuilder. They are not Julia packages that use BinaryProvider. Am I missing something?

[staticfloat]

@fredrikekre is correct, but his link doesn't show it. :)

What he's referring to is the .jll packages idea, which is outlined in the BB roadmap. Essentially, what I'm working on is to have builder repos output small, autogenerated packages that themselves have a lifecycle completely locked to builder repository changes; they are automatically generated by BB, automatically published to the General registry and do nice things like export handles to libraries, wrapper functions for calling executables, etc... It's not documented as a best practice because I hate adding more maintenance work onto people, so it's just going to become "something that happens" rather than "something you have to do."

[SimonDanisch]

Where are we with this? is there a project for this? It seems like there are lots of small work items anyone could work on to make this happen! So maybe we could create a github project and make a few seizable issues to make it easier for outsiders to work on this!

[SimonDanisch]

at least we could start with something like making run not mess with the LD_LIBRARY_PATH for executable binary products :P
Currently, loading a package with an executable bindeps destroys download for some unix system...
I feel like I could easily tackle this issue, but without a more precise roadmap & issues, it doesn't seem easy to make a PR that conforms to the current plans of the maintainers.

[fredrikekre]

Elliot has been thinking about this recently, and we discussed it on the Pkg triage call the other week, see JuliaLang/Pkg.jl#841.