Please make dub find & use packages installed to system locations

[ximion]

This is a follow-up of issue #811
At time, it is impossible to properly package dub packages in Linux distributions.
In Linux distros, we need to have all stuff required to build a software present in the distribution archives, this includes dub packages which other software depends on.
Unfortunately, dub does not search for matching packages in /usr/include/d.
So, ideally dub would allow some kind of dub -v --allow-registries=local build command to build using only locally installed packages.
So, I am proposing:

• Install dub packages locally in the file system under a standard path
• The path could e.g. be /usr/include/d/{package}/{version}/ or /usr/share/dlang/{package}/{version}/ (the former if the dub package can also be used without dub without changes, since that directory seems to be used by non-dub D software too)
• When resolving dependencies, have dub prefer these locally installed packages. This requires to stat once to see if the expected package name is present, and then one quick search to find the highest matching version. This should be really quick, so it won't hurt performance.
• Build with the packages found locally
• Optional, but ideal: Use the shared/static libraries built from the sources which have been installed into a library path (e.g. /usr/lib/<triplet>/libfoobar.so) to link against, instead of recompiling the package that has already been pre-built.

Not having a mechanism like this creates real problems when packaging any dub-using D software for Linux distributions, see gnunn1/tilix#304 for an example.
Would be great to have this fixed, in order to be able to build D code in distributions without crude hacks.

[ximion]

This is really annoying for users, it would be awesome if this bug received some feedback & attention...
For a real-world example of problems this causes for distributions and people wanting to use dub packages, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830412

[s-ludwig]

I'm having issues understanding the root problem here. So the situation is that DUB will search /var/lib/dub/packages/ before falling back to the registry*. If all required packages are present there with the latest versions (or those in dub.selections.json), using dub build --skip-registry=all should use them and not connect to the internet.

#811 mentions pre-compiled binaries, which are problematic in the DUB ecosystem, because packages can be built in many different forms to produce different binaries (optional dependencies, different build configurations, different build modes, upwards-inherited version identifiers etc.). This is why the build results are stored in a directory that contains a hash of all build parameters that went into the build.

So using a vanilla build would be an option, but generally I don't think that it makes sense to distribute pre-compiled binaries for library packages. Executables are a different matter, of course.

* It's also possible to add search paths or to register individual package directories in /var/lib/dub/packages/local-packages.json. The only caveat is that in all cases, the package version must be identifiable, either using a .git sub directory, or using an explicit "version" field in dub.json/dub.sdl.

[ximion]

The problem here is that /var is for system state data that distributions should not write to directly.
So, when we package a dub application, we can not store stuff in /var, but have to put it into /usr/share/<subdir> (for arch-independent data) or /usr/lib/<subdir> (arch-dependent stuff).

Using pre-compiled binaries would be really awesome, but getting the first path issue fixed is really essential.

We also can not write to an index file, so ideally I would just like to have packages installed into soe directory in /usr/(lib|share) and have dub figure out the versioning etc. at build time.
Greetings from Debconf!

[mihails-strasuns]

@s-ludwig to put it very short : issue @ximion is trying to solve is to be able to do a Debian source package from dub package so that dub will recognize its presence in the system and not try fetching again to own cache.

@ximion as a work around, have you considered post-install hook to call dub add-local /usr/include/d/<pkgname> (for all D source packages and dub itself) as a workaround?

[s-ludwig]

There is one other issue to solve, which is to separate the build output into a different directory, maybe somewhere in ~/.dub/. That may be a good idea to do anyway, at least as the default behavior.

Regarding the /usr/include or /usr/share folder, one thing to keep in mind is that we just added an extra directory layer for packages in DUB, so that a certain directory structure that was mainly somewhat popular during D 1.0 times can be supported. The idea is that the repository gets merged into a single source tree using a GIT submodule. For this to work, the folder which contains the package contents needs to have the same name as the package. So DUB currently uses .../name-version/name/*.

[ximion]

@Dicebot Yes, I thought about adding such a hook, but that would mean every package shipping dub stuff would need that (or we would need to handle this via triggers), and it is a really annoying hack (I also don't know if dub behaves correctly when this gets called a lot of times - it probably will be fine).
I would preferably resolve this in dub upstream, before we add more D things to Debian (which is highly likely to happen soon, when this is resolved).

@s-ludwig Please don't use something in HOME for build-output. Tests and builds should not modify contents in user's home directory, that's why Debian enforces this policy on our autobuilders (by setting home to an non-existing directory). Placing it in the source build tree is fine, so is adding some config option or autodetection to change the behavior of this.

Hmm, name-version/name/* is a pretty odd scheme... Why wasn't something like name-version/* or name/version/* chosen?

[mihails-strasuns]

Hmm, name-version/name/* is a pretty odd scheme... Why wasn't something like name-version/* or name/version/* chosen?

Oh, I can actually explain this, because we use same naming scheme at work (with git submodules instead of dub packages).

D module system works best when it has close mapping to filesystem layout, for example, module pkg.foo.bar can be found via path pkg/foo/bar.d. It is possible to define custom module name if one manually supplies all required sources to the compiler (like dub does right now) but it breaks tools like rdmd which use module names to automatically find imported files.

This pretty much limits one to either <projdir>/<sources> and using -I.. (extremely unhygienic) or the somewhat redundant <projdir>/src/<projname>/<sources> in one of its forms. Extra nesting is visually annoying but technical benefits of direct file to module mapping outweight it.

[wilzbach]

@s-ludwig and I just quickly talked about this at DConf and we think that this should be as easy as adding a read-only SystemPackageRepository to packagemanager.d and keeping the first two existing ones as the only writable ones.

[wilzbach]

See #1459 for @s-ludwig's PR created during the DConf hackathon.

[ximion]

@wilzbach This fix doesn't work (yet) because dub never reads /etc/dub/settings.json and only the user settings in /home or in /var, which is unsuitable for Linux distributions.

[ximion]

Oh, and additionally dub doesn't seem to fetch the right version number for local packages installed in /usr/include/d/, instead it always assumes master as version number, unless a version is set in dub.json. This is probably not a bug (projects should explicitly define a version number), but since the version number is encoded in the directory name, it would be neat if dub could infer it from that.

In other news, with this issue fixed I am working on some integration of dub into the Meson build system, the result will likely be pretty neat and might allow for seamlessly mixing the two build systems.
On the distro side, dub wanting to write into $HOME for simple building, even with network access disabled, is still a bit of a pain, but with the local package cache, the biggest roadblock is removed.