New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please make dub find & use packages installed to system locations #838

Closed
ximion opened this Issue May 8, 2016 · 11 comments

Comments

Projects
None yet
4 participants
@ximion
Contributor

ximion commented May 8, 2016

This is a follow-up of issue #811
At time, it is impossible to properly package dub packages in Linux distributions.
In Linux distros, we need to have all stuff required to build a software present in the distribution archives, this includes dub packages which other software depends on.
Unfortunately, dub does not search for matching packages in /usr/include/d.
So, ideally dub would allow some kind of dub -v --allow-registries=local build command to build using only locally installed packages.
So, I am proposing:

  • Install dub packages locally in the file system under a standard path
  • The path could e.g. be /usr/include/d/{package}/{version}/ or /usr/share/dlang/{package}/{version}/ (the former if the dub package can also be used without dub without changes, since that directory seems to be used by non-dub D software too)
  • When resolving dependencies, have dub prefer these locally installed packages. This requires to stat once to see if the expected package name is present, and then one quick search to find the highest matching version. This should be really quick, so it won't hurt performance.
  • Build with the packages found locally
  • Optional, but ideal: Use the shared/static libraries built from the sources which have been installed into a library path (e.g. /usr/lib/<triplet>/libfoobar.so) to link against, instead of recompiling the package that has already been pre-built.

Not having a mechanism like this creates real problems when packaging any dub-using D software for Linux distributions, see gnunn1/tilix#304 for an example.
Would be great to have this fixed, in order to be able to build D code in distributions without crude hacks.

@ximion

This comment has been minimized.

Show comment
Hide comment
@ximion

ximion Jul 8, 2016

Contributor

This is really annoying for users, it would be awesome if this bug received some feedback & attention...
For a real-world example of problems this causes for distributions and people wanting to use dub packages, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830412

Contributor

ximion commented Jul 8, 2016

This is really annoying for users, it would be awesome if this bug received some feedback & attention...
For a real-world example of problems this causes for distributions and people wanting to use dub packages, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830412

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jul 8, 2016

Member

I'm having issues understanding the root problem here. So the situation is that DUB will search /var/lib/dub/packages/ before falling back to the registry*. If all required packages are present there with the latest versions (or those in dub.selections.json), using dub build --skip-registry=all should use them and not connect to the internet.

#811 mentions pre-compiled binaries, which are problematic in the DUB ecosystem, because packages can be built in many different forms to produce different binaries (optional dependencies, different build configurations, different build modes, upwards-inherited version identifiers etc.). This is why the build results are stored in a directory that contains a hash of all build parameters that went into the build.

So using a vanilla build would be an option, but generally I don't think that it makes sense to distribute pre-compiled binaries for library packages. Executables are a different matter, of course.

* It's also possible to add search paths or to register individual package directories in /var/lib/dub/packages/local-packages.json. The only caveat is that in all cases, the package version must be identifiable, either using a .git sub directory, or using an explicit "version" field in dub.json/dub.sdl.

Member

s-ludwig commented Jul 8, 2016

I'm having issues understanding the root problem here. So the situation is that DUB will search /var/lib/dub/packages/ before falling back to the registry*. If all required packages are present there with the latest versions (or those in dub.selections.json), using dub build --skip-registry=all should use them and not connect to the internet.

#811 mentions pre-compiled binaries, which are problematic in the DUB ecosystem, because packages can be built in many different forms to produce different binaries (optional dependencies, different build configurations, different build modes, upwards-inherited version identifiers etc.). This is why the build results are stored in a directory that contains a hash of all build parameters that went into the build.

So using a vanilla build would be an option, but generally I don't think that it makes sense to distribute pre-compiled binaries for library packages. Executables are a different matter, of course.

* It's also possible to add search paths or to register individual package directories in /var/lib/dub/packages/local-packages.json. The only caveat is that in all cases, the package version must be identifiable, either using a .git sub directory, or using an explicit "version" field in dub.json/dub.sdl.

@ximion

This comment has been minimized.

Show comment
Hide comment
@ximion

ximion Jul 8, 2016

Contributor

The problem here is that /var is for system state data that distributions should not write to directly.
So, when we package a dub application, we can not store stuff in /var, but have to put it into /usr/share/<subdir> (for arch-independent data) or /usr/lib/<subdir> (arch-dependent stuff).

Using pre-compiled binaries would be really awesome, but getting the first path issue fixed is really essential.

We also can not write to an index file, so ideally I would just like to have packages installed into soe directory in /usr/(lib|share) and have dub figure out the versioning etc. at build time.
Greetings from Debconf!

Contributor

ximion commented Jul 8, 2016

The problem here is that /var is for system state data that distributions should not write to directly.
So, when we package a dub application, we can not store stuff in /var, but have to put it into /usr/share/<subdir> (for arch-independent data) or /usr/lib/<subdir> (arch-dependent stuff).

Using pre-compiled binaries would be really awesome, but getting the first path issue fixed is really essential.

We also can not write to an index file, so ideally I would just like to have packages installed into soe directory in /usr/(lib|share) and have dub figure out the versioning etc. at build time.
Greetings from Debconf!

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jul 8, 2016

@s-ludwig to put it very short : issue @ximion is trying to solve is to be able to do a Debian source package from dub package so that dub will recognize its presence in the system and not try fetching again to own cache.

@ximion as a work around, have you considered post-install hook to call dub add-local /usr/include/d/<pkgname> (for all D source packages and dub itself) as a workaround?

mihails-strasuns commented Jul 8, 2016

@s-ludwig to put it very short : issue @ximion is trying to solve is to be able to do a Debian source package from dub package so that dub will recognize its presence in the system and not try fetching again to own cache.

@ximion as a work around, have you considered post-install hook to call dub add-local /usr/include/d/<pkgname> (for all D source packages and dub itself) as a workaround?

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jul 9, 2016

Member

There is one other issue to solve, which is to separate the build output into a different directory, maybe somewhere in ~/.dub/. That may be a good idea to do anyway, at least as the default behavior.

Regarding the /usr/include or /usr/share folder, one thing to keep in mind is that we just added an extra directory layer for packages in DUB, so that a certain directory structure that was mainly somewhat popular during D 1.0 times can be supported. The idea is that the repository gets merged into a single source tree using a GIT submodule. For this to work, the folder which contains the package contents needs to have the same name as the package. So DUB currently uses .../name-version/name/*.

Member

s-ludwig commented Jul 9, 2016

There is one other issue to solve, which is to separate the build output into a different directory, maybe somewhere in ~/.dub/. That may be a good idea to do anyway, at least as the default behavior.

Regarding the /usr/include or /usr/share folder, one thing to keep in mind is that we just added an extra directory layer for packages in DUB, so that a certain directory structure that was mainly somewhat popular during D 1.0 times can be supported. The idea is that the repository gets merged into a single source tree using a GIT submodule. For this to work, the folder which contains the package contents needs to have the same name as the package. So DUB currently uses .../name-version/name/*.

@ximion

This comment has been minimized.

Show comment
Hide comment
@ximion

ximion Jul 11, 2016

Contributor

@Dicebot Yes, I thought about adding such a hook, but that would mean every package shipping dub stuff would need that (or we would need to handle this via triggers), and it is a really annoying hack (I also don't know if dub behaves correctly when this gets called a lot of times - it probably will be fine).
I would preferably resolve this in dub upstream, before we add more D things to Debian (which is highly likely to happen soon, when this is resolved).

@s-ludwig Please don't use something in HOME for build-output. Tests and builds should not modify contents in user's home directory, that's why Debian enforces this policy on our autobuilders (by setting home to an non-existing directory). Placing it in the source build tree is fine, so is adding some config option or autodetection to change the behavior of this.

Hmm, name-version/name/* is a pretty odd scheme... Why wasn't something like name-version/* or name/version/* chosen?

Contributor

ximion commented Jul 11, 2016

@Dicebot Yes, I thought about adding such a hook, but that would mean every package shipping dub stuff would need that (or we would need to handle this via triggers), and it is a really annoying hack (I also don't know if dub behaves correctly when this gets called a lot of times - it probably will be fine).
I would preferably resolve this in dub upstream, before we add more D things to Debian (which is highly likely to happen soon, when this is resolved).

@s-ludwig Please don't use something in HOME for build-output. Tests and builds should not modify contents in user's home directory, that's why Debian enforces this policy on our autobuilders (by setting home to an non-existing directory). Placing it in the source build tree is fine, so is adding some config option or autodetection to change the behavior of this.

Hmm, name-version/name/* is a pretty odd scheme... Why wasn't something like name-version/* or name/version/* chosen?

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jul 11, 2016

Hmm, name-version/name/* is a pretty odd scheme... Why wasn't something like name-version/* or name/version/* chosen?

Oh, I can actually explain this, because we use same naming scheme at work (with git submodules instead of dub packages).

D module system works best when it has close mapping to filesystem layout, for example, module pkg.foo.bar can be found via path pkg/foo/bar.d. It is possible to define custom module name if one manually supplies all required sources to the compiler (like dub does right now) but it breaks tools like rdmd which use module names to automatically find imported files.

This pretty much limits one to either <projdir>/<sources> and using -I.. (extremely unhygienic) or the somewhat redundant <projdir>/src/<projname>/<sources> in one of its forms. Extra nesting is visually annoying but technical benefits of direct file to module mapping outweight it.

mihails-strasuns commented Jul 11, 2016

Hmm, name-version/name/* is a pretty odd scheme... Why wasn't something like name-version/* or name/version/* chosen?

Oh, I can actually explain this, because we use same naming scheme at work (with git submodules instead of dub packages).

D module system works best when it has close mapping to filesystem layout, for example, module pkg.foo.bar can be found via path pkg/foo/bar.d. It is possible to define custom module name if one manually supplies all required sources to the compiler (like dub does right now) but it breaks tools like rdmd which use module names to automatically find imported files.

This pretty much limits one to either <projdir>/<sources> and using -I.. (extremely unhygienic) or the somewhat redundant <projdir>/src/<projname>/<sources> in one of its forms. Extra nesting is visually annoying but technical benefits of direct file to module mapping outweight it.

@ximion ximion changed the title from Please make dub find & use locally installed packages to Please make dub find & use packages installed to system locations Jul 13, 2016

@wilzbach wilzbach referenced this issue Sep 13, 2017

Closed

lib64 dir #93

@wilzbach

This comment has been minimized.

Show comment
Hide comment
@wilzbach

wilzbach May 5, 2018

Member

@s-ludwig and I just quickly talked about this at DConf and we think that this should be as easy as adding a read-only SystemPackageRepository to packagemanager.d and keeping the first two existing ones as the only writable ones.

Member

wilzbach commented May 5, 2018

@s-ludwig and I just quickly talked about this at DConf and we think that this should be as easy as adding a read-only SystemPackageRepository to packagemanager.d and keeping the first two existing ones as the only writable ones.

s-ludwig added a commit that referenced this issue May 5, 2018

Add a "customCachePaths" configuration field. Fixes #838.
Allows defining additional paths that contain packages in subfolders with the pattern "(name)-(version)/(name)/" by defining a "customCachePaths" field in /etc/dub/settings.json or ~/.dub/settings.json.

s-ludwig added a commit that referenced this issue May 5, 2018

Add a "customCachePaths" configuration field. Fixes #838.
Allows defining additional paths that contain packages in subfolders with the pattern "(name)-(version)/(name)/" by defining a "customCachePaths" field in /etc/dub/settings.json or ~/.dub/settings.json.

s-ludwig added a commit that referenced this issue May 5, 2018

Add a "customCachePaths" configuration field. Fixes #838.
Allows defining additional paths that contain packages in subfolders with the pattern "(name)-(version)/(name)/" by defining a "customCachePaths" field in /etc/dub/settings.json or ~/.dub/settings.json.

s-ludwig added a commit that referenced this issue May 5, 2018

Add a "customCachePaths" configuration field. Fixes #838.
Allows defining additional paths that contain packages in subfolders with the pattern "(name)-(version)/(name)/" by defining a "customCachePaths" field in /etc/dub/settings.json or ~/.dub/settings.json.
@wilzbach

This comment has been minimized.

Show comment
Hide comment
@wilzbach

wilzbach May 6, 2018

Member

See #1459 for @s-ludwig's PR created during the DConf hackathon.

Member

wilzbach commented May 6, 2018

See #1459 for @s-ludwig's PR created during the DConf hackathon.

s-ludwig added a commit that referenced this issue May 6, 2018

Add a "customCachePaths" configuration field. Fixes #838.
Allows defining additional paths that contain packages in sub folders with the pattern "(name)-(version)/(name)/" by defining a "customCachePaths" field in /etc/dub/settings.json or ~/.dub/settings.json.

@wilzbach wilzbach closed this in #1459 May 8, 2018

@ximion

This comment has been minimized.

Show comment
Hide comment
@ximion

ximion May 15, 2018

Contributor

@wilzbach This fix doesn't work (yet) because dub never reads /etc/dub/settings.json and only the user settings in /home or in /var, which is unsuitable for Linux distributions.

Contributor

ximion commented May 15, 2018

@wilzbach This fix doesn't work (yet) because dub never reads /etc/dub/settings.json and only the user settings in /home or in /var, which is unsuitable for Linux distributions.

@ximion

This comment has been minimized.

Show comment
Hide comment
@ximion

ximion May 15, 2018

Contributor

Oh, and additionally dub doesn't seem to fetch the right version number for local packages installed in /usr/include/d/, instead it always assumes master as version number, unless a version is set in dub.json. This is probably not a bug (projects should explicitly define a version number), but since the version number is encoded in the directory name, it would be neat if dub could infer it from that.

In other news, with this issue fixed I am working on some integration of dub into the Meson build system, the result will likely be pretty neat and might allow for seamlessly mixing the two build systems.
On the distro side, dub wanting to write into $HOME for simple building, even with network access disabled, is still a bit of a pain, but with the local package cache, the biggest roadblock is removed.

Contributor

ximion commented May 15, 2018

Oh, and additionally dub doesn't seem to fetch the right version number for local packages installed in /usr/include/d/, instead it always assumes master as version number, unless a version is set in dub.json. This is probably not a bug (projects should explicitly define a version number), but since the version number is encoded in the directory name, it would be neat if dub could infer it from that.

In other news, with this issue fixed I am working on some integration of dub into the Meson build system, the result will likely be pretty neat and might allow for seamlessly mixing the two build systems.
On the distro side, dub wanting to write into $HOME for simple building, even with network access disabled, is still a bit of a pain, but with the local package cache, the biggest roadblock is removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment