Support specification of system dependencies (APT, ports etc.)

[s-ludwig]

This should invoke the platform specific package manager to install non-DUB dependencies and can also be used to automatically generate system specific packages for DUB application projects. @eskimor has posted a basic idea of how to manage such a system in #34.

[eskimor]

Packagekit could probably handle most of the task.

[s-ludwig]

Didn't hear of it so far, that should indeed safe a good amount of work. But it still leaves Windows, OS X (homebrew/macports) and BSD flavors (ports) left to handle. Windows of course is particularly interesting, but maybe is not worth to include (instead bindings packages could distribute the necessary .lib and possible .dll files).

[Llammissar]

I hate to be the bearer of bad news, but this is a bad idea and pretty much impossible to get right. I wish I didn't have to put it that way, but messing with the system is not something a language package manager should EVER do. Security implications aside (and there are many. In the first place just assume you're using dub as root because you need to manage the system's D packages. (Dub does support system-level packages, right?)), the mapping is never clean and you run the risk of completely breaking user systems (you cannot, as a rule, make assumptions about what will or will not cause breakage). The only sane course of action is to test for features and fail cleanly.

If you really, really insist that this simply must happen, it really cannot be default.

But rather a better methodology is to simply make it easy for distros to wrap your PM (historically, CPAN does this best by far). If it's good enough, it can be done automatically, and the system PM can properly track what's installed. (Oh dear, this is getting long. If you'd like, we can discuss this further in a venue of your choosing.)

[eskimor]

I don't really see the problem. There are already applications (like movie players installing needed codecs) that ask the system to install packages. Also installing packages should in general not do any harm to the system. In addition the user would of course be asked for permission before doing any change, also with packagekit dub does not need to have root privileges.

For the mapping, it is possible to search for packages providing a particular file for example. Installing packages on a Linux distribution is hardly a dangerous task, maybe you can elaborate a little more on your concerns.

Wrapping dub with the system package manager is something that might be done in addition for the end user. But for the developer this feature would greatly simplify things -> it would just work, without having the developer keeping track of all the dependencies' dependencies manually.

For being the default or not: The user would be asked for permission, so if the user wants to install the packages on his own, he can just say no and do so.

If a package from an official repository breaks your system, this is a bug in your distribution and has nothing to do with dub. If dub did not install it, the user would do it manually, causing the same problem. dub would just provide a shortcut for an action the user would have to do anyway.

[Llammissar]

Up front, I'm going to offer this
link:ftp://ftp.saix.net/pub/CPAN/misc/ZCAN.html

We have some differing ideas on how laissez faire you can be in the
creation of your language-specific package manager, but my point is you
could do far worse for language PMs to model after. Especially in terms
of reliability and least surprise, CPAN is a model citizen and
exceptionally hard to beat.

I don't really see the problem. There are already applications (like
movie players installing needed codecs) that ask the system to install
packages.

As a rule, we consider that sort of behaviour a bug. Especially if it's
a default. It's not much different from a makefile wgetting a git
snapshot before building.

Also installing packages should in general not do any harm to the
system.

I truly wish I could share in this optimism. But if it were that simple,
our QA teams wouldn't be so overworked.

In addition the user would of course be asked for permission before
doing any change

This is acceptable as an option; for a GUI front-end I might even accept
it as a default on the assumption that anyone using a GUI PM frontend
accepts the risk.

also with packagekit dub does not need to have root privileges.

Packagekit, may work reliably on Red Hat-derived distros; but outside of
that, it's far less certain, and not really on any BSD.

For the mapping, it is possible to search for packages providing a
particular file for example. Installing packages on a Linux
distribution is hardly a dangerous task, maybe you can elaborate a
little more on your concerns.

Even very popular system-level package managers don't tend to have very
robust safeguards against clobbering files that already exist, nor do
they tend to support multiple installed versions of the same library.
Unless it's within scope for dub to deal with library version mismatches
and check for files that might get overwritten, calling the system PM
should be approached in the most cautious way possible. Packagekit is
not cautious; it's a privilege escalation masquerading as a feature.

Wrapping dub with the system package manager is something that might
be done in addition for the end user. But for the developer this
feature would greatly simplify things -> it would just work, without
having the developer keeping track of all the dependencies'
dependencies manually.

If it exports dependency lists in a sane format, it's trivial to wrap
dub with the system PM. We do this with Perl and CPAN to great effect.
As an added benefit, this lets the system PM keep proper track of the
files dub installs. I should be clear that I'm not advocating for
manual dependency tracking; I'm just trying to prevent dub from creating
a QA nightmare the makes distros resistant to packaging D libraries.
The easier it is to guarantee that the process doesn't have any creepy
action at a distance, the better.

I somewhat intimated this above, but wearing my sysadmin hat: it's
really not the job of userspace applications to decide what should be
installed at the system level. Ever. The person responsible for
managing the system software should be extremely aware of what they are
installing and generally have some idea of the meaning behind it.
Things that abstract away privilege escalation tend to short circuit the
consideration given to other packages. (sudo is in a similar situation
for similar reasons.) Please do not underestimate the amount of pain
caused by poorly-considered decisions of language-specific package
managers when it comes to dependencies on system packages. There's a
reason very few sysadmins have anything kind to say about Ruby gems.

(As an aside, it seems odd that there would even BE enough non-standard
deps to necessitate this sort of workaround.)

For being the default or not: The user would be asked for permission,
so if the user wants to install the packages on his own, he can just
say no and do so.

From a QA point of view, asking for permission is obscuring the issue:
It's an insecure default. Distros and sysadmins will thank you to avoid
insecure defaults.

If a package from an official repository breaks your system, this is a
bug in your distribution and has nothing to do with dub. If dub did
not install it, the user would do it manually, causing the same
problem. dub would just provide a shortcut for an action the user
would have to do anyway.

Updates from official repositories can and will break all sorts of
things all the time. Distro QA is a huge undertaking even for
strictly-versioned binary distros. Examples:

• On servers running Plesk, changing the mysql version breaks
  everything.
• Perl updates on cpanel servers break things.
• PHP updates break bindings with things like imagemagick
• Just try running dist-upgrade on a Debian-derived distro.
• udev, recently (predictable interface names, my ass)
• X libraries, drivers, and the server can't be strict dependencies, so
  forgetting one can clobber your display if you restart X.
• Installing new libGL is often tied to drivers. Kernel/driver mismatch
  is a thing that happens.
• Dub can specify versions. Choosing lower versions and installing them
  on a PM that doesn't support multiple library versions will create a
  whole new kind of hell for every application linking that library.

Look, I don't honestly expect I can dissuade you from scratching this
itch at this point; I get the sense you're quite taken with it. But I
do sincerely hope you take the issues I'm raising seriously. If dub
is to fit into the Linux ecosystem, there are certain very basic
expectations we have of it, and stepping on those toes makes D the
subject of derision and anger. I don't want that. I want installation
and maintenance of systems with D packages to be as easy as installing
C packages, and I'm sorry to say I believe this proposal runs counter to
that goal.

Regards,
Wyatt

[eskimor]

Hi! Just wanted to tell you that I take your concerns seriously and I will examine your points thoroughly ( just not now, because I am quite busy and even dub concerned there are some more important things at the moment). I appreciate the sharing of your insights. I certainly have single-user workstations in mind and not servers or something, where the user as developer usually knows how to install packages and knows about its possible consequences. Nevertheless, I will have a closer look at CPAN and I am sure we will find a good solution that works reliably and convenient. It will take some time until I get to this issue, I hope you are still around then.

Thanks for the link!

[etcimon]

I think the proposal in issue #248 should be augmented with a repository and os version. For security reasons, dub should have those repository configurations hard-coded, because RPMs may expose the system to security risks if the repositories are configured through the dub.json file. dub would simply prompt the user running dub if he'd like to install (temporarily?) the corresponding repository if it's not installed, and the corresponding packages ( [yes]/no ) which provide the linker libraries used in the current build configuration.

{
    "systemDependencies": {
        "Libevent 2.0.x": [
            {"os": { "name": "debian", "versions": "=6"}, "repository": { "tool":"apt", "name": "squeeze-backports"}, "packages": ["libevent-dev"], "libs": ["event", "event_threads"]},
            {"os": { "name": "debian", "versions": ">=7"}, "repository": { "tool": "apt", "name": "base"}, "packages": ["libevent-dev"], "libs": ["event", "event_threads"]},
            {"os": { "name": "centos", "versions": ">=6"}, "repository": { "tool": "yum", "name": "remi-test"}, "packages": ["libevent-last-devel"], "libs": ["event", "event_pthreads"]},
            {"os": { "name": "fedora", "versions": ">=17"}, "repository": { "tool": "yum", "name": "base"}, "packages": ["libevent-devel"], "libs": ["event", "event_pthreads"]},
            {"os": { "name": "mac", "versions": ">=10.8"}, "repository": { "tool": "brew", "name": "base"}, "packages": ["libevent"], "libs": ["event", "event_pthreads"]},
            {"os": { "name": "windows", "versions": ">=6"}, "repository": { "tool": "chocolatey", "name": "base"}, "packages": ["libevent"], "libs": ["event", "event_pthreads"]}
        ]
    }
}

It seems to me that dub should know how to install homebrew for mac os x.
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

It should however ask for permission to do so.

Dub should know how to install chocolatey for windows.

[etcimon]

So, the requirements for dub would be to know how to install a package manager on mac and windows (with user permission of course)

https://github.com/Homebrew/homebrew/wiki/Installation
https://github.com/chocolatey/chocolatey/wiki/Installation

And then, it would have to read repo packages vs required libs in dub.json and ask the user for confirmation for the entire installation chain needed to install the library/application needed.

[markos]

As a Debian developer -and one who has started to work on packaging D libraries for Debian, already maintain ldc since recently- I can tell you that wrapping apt behaviour in dub is not -and probably will never be- a distro-endorsed way. At least, even if I were to push for this, it's pretty unlikely that it would ever accepted by policy.

So far what I've been doing is simple, try to package a package from the git repo mentioned in each package's page in the dub registry and install this in a reasonable place (/usr/include/d/common as suggested by the Arch packager sounds simple and could also be made to be used by the gdc package as well).

What I'm having trouble with is creating libraries. I have to follow dub's json rules and see what needs to be built and try to replicate this and then create the corresponding library.

Dub should not try to replicate a package manager's job, but it could help. For example, it would be a great thing if dub had some kind of export to a makefile (of any build system, actually), that way one would just do a dub export-makefile and do a simple make to build the package with a multitude of distro-specific available tools. Otherwise, dub exported packages imported directly into Debian, I don't see it happening, sorry.

[s-ludwig]

Personally, I'm convinced that, at least at this point, Debian packages should be restricted to applications. For libraries to be reasonably packaged, there would need to be a stable ABI, but the ABI differs not only between different compilers and different compiler versions, but also between different builds of the same library when different -version= flags (or many other compiler flags) are used. Because of this it really only makes sense to distribute libraries as pure source packages and build them on demand for the specific combination of each project, configuration and compiler.

Another factor is that I can imagine that building a separate system for D libraries - again, just at this point - could even be a net loss for the ecosystem, because it might lead to fragmentation when part of the developers choose to only support using the system packages. This will certainly not be an issue once there is a healthy amount of packages available.

Regarding an export to makefile, that would indeed be a good thing to have (in the form of dub generate make). It also should be quick to do, but with my current time budget I can't promise when I'll actually manage to implement that. I'll add a ticket for this, though.

Finally, the problem with system dependencies is that there really is nothing better that we can do other than add platform specific hints to DUB packages if we want to have a proper cross-platform solution - or at least the previous discussions haven't revealed any better approach. It wouldn't even be necessary for DUB to actually invoke apt-get, it could simply conveniently output a message to the user with the command that needs to be run to get the missing dependencies. I don't have a strong opinion there, but it would be a big win to have such a functionality available, even if it doesn't fit the philosophy of each single operating system/distribution. But I agree that DUB shouldn't try to compete with the system package manager and thus shouldn't perform any actual installation of packages.

[markos]

One feature that I would suggest is instead of hardcoding the needed packages, it might just list the headers (eg. event2/event.h) and the distro would return the packages the headers belong to (by using dpkg -S "event2/event.h" on Debian and the resp. command on rpm distros, etc). Same goes for libraries. That would be more future proof, as you wouldn't have to follow package names all the time. Just an idea.

[etcimon]

it could simply conveniently output a message to the user with the command that needs to be run to get the missing dependencies

That would be very useful, via attaching to os-specific header searches e.g. dpkg like @markos mentions, so the json would simply list the required headers enforcing it in a way that allows to throw a custom error message (os-specific) when dub does its thing.

[s-ludwig]

Yes, that's indeed an interesting idea, at least for systems that support such a search command.

[mihails-strasuns]

It does not fit well with "simple cross-platform tool" concept as such logic will not only be OS-specific (and impossible for some of those) but also distro-specific.

Why do it need to generate makefile though? It will be just plain command list anyway, generating shell script should be enough (and more portable).

What is more interesting is command to generate source tarball with all dependencies included to avoid necessity to package all code.dlang.org packages in Debian.

[markos]

Unfortunately, Debian specifically "encourages" fine-grained splitting of packages vs big packages that include everything. It's doable, sure, especially if the supplied code is a modified fork versus the package in the archive, but it's going to get some complains definitely :/

In any case, from what I see most packages are really easy to do (eg. especially if they're source-only and need to carry no library). I've already finished openssl-dlang and libevent-dlang (not entirely sure about the -dlang suffix, maybe I could change that, eg. the Go/Python packages are golang-/python- prefixed, anyway, that's a different issue :)
On the makefile/shell script idea, I'm in favour of either, even a perl script would do, or a CMakefile, basically anything that permits to build a DUB-free package.

[mleise]

There are a dozen different package managers on Linux. Give up some control to those who have the respective system installed. The best you can do is provide package maintainers with the information they need and allow full customization of the build process.

As you already correctly identified tools to find *.h files in all packages aren't available on all systems. The ebuild system used on Gentoo for example, builds packages from their sources and until compilation and test installation into a temporary directory is complete, it is not clear which files will make up the installed package.

Further a package may have different names in different distributions or be available as one of several implementations. Two examples: ffmpeg/libav and OpenGL. I think the best you can do is offer us maintainers with a list of package names that we translate into an atom recognized by the package manager. This includes any specific version or range of version of that dependency. The way ebuilds understand it is with a prefixed '<', '>', '=', '<=' or '>=' and to specify any version 2.x you would write =category/package-2*
I can't speak for other package managers, but maybe we can find some minimum granularity that helps every maintainer to write a straight forward translation from dub versions to system package manager versions.

[dhasenan]

The consensus here seems to be that it's a bad idea to get clever with regards to system packages. So can we close this ticket?

[wilzbach]

So can we close this ticket?

Yup. A good way to move forward is to improve the built-in support for compiling C/C++ libraries on the fly (preBuildCommands works, but there's a lot of room for improvement).