Implement dpkg Source

[BuildStream-Migration-Bot]

See original issue on GitLab
In GitLab by [Gitlab user @brejoc] on Apr 13, 2017, 20:40

I'm opening this ticket for discussion and to better understand the goals. What I got from the IRC conversation with [Gitlab user @tristanvb]:

1. We want a way to somehow mirror dpkg packages of a system.
2. Dumping a package list into the dpkg source should produce such a cloned system.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @brejoc] on Apr 13, 2017, 20:50

[Gitlab user @tristanvb] I've looked around and it should be possible to use dpkg --get-selections > the_list to get a pkg list and dpkg --set-selections < the_list && apt-get -y update && apt-get dselect-upgrade to install all the required packages. This should work very well, but doesn't include package versions.

I have one additional question: Am I actually able to simply invoke dpkg commands when DpkgSource ist executed?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 14, 2017, 13:10

Ok so this is a good start! But we're going to need specific versions :)

So, I continue to think this is a breakthrough idea, and have dumped a lot of information here:

• Some tools in BuildStream which will will come in handy in implementation
• Some deeper explanation of BuildStreams expectations and functionality
• Some additional thoughts, which include some "gotchas" which came to mind while writing up the rest of this

So here comes a large markdown formatted brain dump which I hope will be useful for moving forward :)

Utilities that BuildStream provides

The mirror directory

So a Source plugin type is "given" a directory in which to mirror sources, this directory can be obtained with Source.get_mirror_directory(). Calling that as self.get_mirror_directory() will return a directory that belongs to the 'dpkg' Source, and will also result in it's creation if it does not yet exist.

URI specific subdirectories

A specific source will usually cache/mirror it's data in a specific directory, a convenience function utils.url_directory_name() can be used to create a directory name from a url.

So a dpkg Source will mirror packages from a specific URL (either official debian, or some downstream or unrelated debian repo), in a subdirectory named with this utility, inside the dpkg mirror directory.

Project url aliases

For convenience, and also to allow one to mirror all of a projects dependencies in a single location for the sake of guaranteeing repeatability, all Sources must support project url aliases.

When reading a url which may be named foo:bar/baz, a Source uses the Project.translate_url() method to resolve a full uri from the alias read at Plugin.configure() time.

Remember that a Source is subclass of Plugin, so anything on the Plugin is expected of a Source, or can be used by a Source.

Calling Host Tools

Source plugins are the only thing allowed to call out to host tools, we strictly only depend on host tooling for injecting external data into the pipeline.

Obtaining tools at preflight time

Host tooling should be obtained at Plugin.preflight() time, using the utils.get_host_tool() utility function, like so:

  ...
  def preflight(self):

      self.dkpg = utils.get_host_tool('dpkg')

This ensures that an error is raised at preflight time right away, before hours of processing commence. The utility will raise an error so that BuildStream will print an intelligible error in the frontend.

It's worthy to note here that a Project itself brings in additional host dependencies over the base BuildStream dependencies (i.e. you dont need a host git, unless you are using a project which uses git sources).

Running tools at fetch/track/stage time

When you need to actually run the tool, use the tool you previously obtained at preflight time with the Plugin.call() and Plugin.check_output() APIs, which are similar to the corresponding APIs in the subprocess module in the standard libs.

The linked docs contain examples for calling out to host tools.

It's important to use these APIs to call out to host tools because:

• They will automatically ensure process suspend/resume works
• They will automatically ensure the process is terminated with the parent
• They will also ensure that stdout/stderr ends up in the corresponding BuildStream log files

BuildStream expectations

So here are the expectations that BuildStream has, as to how a source should behave.

Isolated configuration

This is not a problem for sources like git or tar, as any host/user configuration pertaining to how these host tools are installed will not have any effect on how the sources are mirrored, or how they are staged into the sandbox area before entering the sandbox.

For Source types which represent distro packaging mechanisms however, this may not be the case. RPM for instance installs a lot of distortion into /etc/rpm (macros and rc files) which may or may not effect how the package is installed.

It is of vital importance here that any and all host configuration for using dpkg which might effect the result of an install into a sysroot location be overridden in such a way that the project has complete control and that there is no host contamination which may cause the staging of a package into a sandbox to have a different result on one host or another.

Network Connectivity

A Source should only require network connectivity in it's Source.track() and Source.fetch() implementations.

This is to say that, you need the network:

• To run bst fetch
• To run bst track
• To run bst build only if the Source refs have not yet been mirrored locally

You dont need the network to run bst build if you have already fetched on the same project/pipeline.

Definitions of url, ref and track properties in a Source configuration

So these could have totally arbitrary names, but by convention we want to keep the same names across the board.

url

This is normally an aliased url (as described above) and for a dpkg source should indicate the debian repository you want to interact with.

ref

This represents an exact Source reference, which will never change, it should be bit-for-bit identical when downloaded on any host.

For dpkg, this should be a fully revisioned package or group of packages.

track

This represents a symbolic name, usually the default is to interact with a VCS so this would be a branch or tag name.

For dpkg, a valid track name, when dealing with upstream debian repository urls, should probably be something like jessie or stretch or sid.

Note that:

• A Project can be fetched and built if no track was ever specified for one or more Sources
• A project cannot be built if there is no ref specified for one or more Sources
• A source will have a ref if it was successfully tracked
• A source which lacks a track configuration, cannot be tracked, but may be built if there is already a ref

Essentially, an end user needs to specify either track or ref, otherwise they are not able to obtain sources in any way.

Fetch Activity

At fetch time, it is assumed that a Source already has a valid ref.

If a Source does not yet have a ref, then BuildStream will error out before fetch is ever called, indicating to the user that they should first call bst track to get a ref for that Source.

A plugin at this time must ensure that local copies are downloaded for the specific ref which is configured in their Source.

Track Activity

At track time, it is the Sources responsibility to check with the remote repository and find the latest version available for the configured track value.

A Source may fetch and download the remote payload at track time if this is convenient but not necessarily. For a git source for instance, it is more convenient to simply run git fetch origin on the git mirror first and then later find out what is the latest commit sha for a tracking branch. For a dpkg source, I suspect that one can obtain the latest versions for jessie without actually downloading and mirroring the said packages, deferring that to a later invocation of bst fetch.

Stage Activity

At stage time, the Source will be handed a directory in which the payload it represents should be dumped.

At this time, Sources are guaranteed to have passed the fetch stage, so the payload is present in the mirror directory and no network activity is required.

For a distro package staging activity, this means to install the extracted package into the staging directory, treating the staging directory as a sysroot (which it actually is).

Additional Thoughts

Single vs multipackage Source

While writing this, it struck me that perhaps a more obvious Source implementation for distro package based sources would be to have one Source represent exactly one package.

This has some pros and cons, though

Pros:

• The Source implementation becomes much simplified when only mirroring and tracking a single package and it's version
• The project which uses an import Element to bring in a bunch of sources, has a finer grained control on what exact packages it imports, this means it's easier to distinguish when we want to import:
  ** A runtime environment and some specific set of libraries
  ** An SDK, which would include a runtime plus some compilers and tooling which we may not want to include in a docker deployment
  ** A BSP, including kernel and drivers for a given arch, which we only need when deploying a bootable image for a VM or bare metal

Cons:

• We lose the ability to introspect dependencies dictated by the package manager in question, this becomes an exercise left to the user, or perhaps some surrounding tooling, or something we havent invented yet.

Package databases

To complete the experience of being able to stage packages, build things, and create new packages for deployment inside a BuildStream pipeline (and perhaps even as a requirement just to get off the ground), it seems required that something take care of initializing a package database for the given tool set.

For instance, when staging RPMs into a sysroot, one first does: rpm --dbpath=${sysroot}/var/lib/rpm --initdb and then calls rpm -ivh --root ${sysroot} rpms-to-install to install them into that root.

Debian packages surely also have some database which needs to be created and maintained, which is a side effect of installing packages from a dpkg source and not part of the payload itself.

How should we address this ?

One idea might be to include an option on a dpkg source configuration which says to initialize and create some base layout directories and database in the staging area to be carried forward in the pipeline (cached as an artifact).

With this approach, perhaps this 'base' source would have to be staged first and would not require any ref, url or track parameters to function (as what it essentially does is just provide a base for further dkpg sources to work with, but not any payload).

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @brejoc] on Apr 15, 2017, 21:49

[Gitlab user @tristanvb] Thanks for the detailed explanations! It's now much clearer to me how BuildStream works!

dpkg indeed allows to install packages to a staging directory: dpkg -i --root=${sysroot} package.deb
I'll see if I can test this this weekend. But I wouldn't be to optimistic, since pre and post install scripts might be a problem here. I couldn't find a source that explains that in detail, so I'll do some test runs.
An other option might be a chroot!?

Regarding single vs multipackage Source. I afraid that a single source approach might get tedious very fast. Managing more than 100 pkg installations could then be a burden instead of a flexibility. And pkg dependencies might drive you over the edge, since you'd have to check this manually or you'd constantly run into build errors.

One additional quest. Where is the sysroot coming from? I assume sysroot would at that point of time be some kind of overlay filesystem with a minimal Debian installation form a previous Source?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 16, 2017, 13:29

[Gitlab user @brejoc] hi... I'll try to reply briefly, on my way to the airport :)

So three things,

pre and post scriptlets

These are indeed a bit annoying, most likely they will remain mostly unneeded for build purposes, needed if one wants to deploy a booting system, and somewhere in between in the case of deploying sdks or docker container images.

The ideal here would be to translate those into integration commands in the bst public data domain, which means "these commands should be run against this dependency in the sandbox"

Currently the puclic domain data cannot be dynamically populated, but i have a couple of other use cases for this so i could make that happen.

But this seems a bit much for an initial implementation so i hope in the first iteration we can ignore install scriptlets.

single vs multiple

Agreed, however this still raises some questions, i think the solution here is that a dpkg source needs a list of unversioned package names the user can provide in the configuration in order to determine what is desired (surely the user does not want the entire content of the distro).

sysroot

Where does the sysroot come from ?

This can be something a bit different depending on the Element, basically the core provides a Sandbox to the elements at Element.assemble() time, and the element is then free to stage dependencies and sources anywhere inside the temporary sandboxes root directory.

For the typical import element case i expect to use dpkg sources with; sources are staged directly at the root of the sandbox and then an artifact is collected directly from that root.

I think more to the point: the sysroot will be empty for any useful purposes, hence my previous comments about package databases and such side effects. It seems to me that a side effect of creating some base directory structure and/or initializing an empty package database at the beginning of the stage operation may be needed.

chroots

Just another word here, we dont want to ever require being root. So we dont want chroots, but can do some things with bubblewrap as that is a hard dependency of the Sandbox (commands in the sandbox are actually run as uid 0 in a uid namespace).

While the dpkg source could use bubblewrap for some things i think its much preferrable that we dont.

in closing

Hope this provides some pointers, while im flying around you might look at the buildstream-tests repo and see how we're using the said integration commands, and what element public data is for.

If we allow the elements to dynamically populate the public data at assemble time and make a statement that one may only read public data from their dependencies at assemble time, then solving scriplets may just be a matter of adding a dpkg specific import element implementation to populate that data, after which the regular integration rules apply.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @brejoc] on Apr 24, 2017, 09:18

[Gitlab user @tristanvb] It seems to be harder than expected to mirror package installations in bulk with exact version numbers. I'm still evaluating options… :\

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 24, 2017, 11:03

[Gitlab user @brejoc] hi!

I just got back to seoul today, nice to hear you are still investigating this :)

From my limited understanding debian packaging scripts are quite exposed (layer upon layer of seldom used apis/scripts) so i wouldnt rule this out just yet... I'll try to do some research on this tomorrow too, maybe i can squeeze the info out of the right irc channel...

The feature of staging something deterministic is however quite important, i hope we can do it this way as the alternatives i think will invariably include extra moving parts (i.e maybe extra mirroring solution either with custom interface or mirroring deterministic snapshots into ostree repos)... Not ideal.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 25, 2017, 12:22

[Gitlab user @brejoc] So I found some interesting info just asking people in #debian irc.

First there is a tool called apt-offline which is used for generating a signature of a specific debian version and generating an installer for that version for another machine that does not have network connectivity.

Then there is also apt-zip and apt-walkabout which are closely related.

Note that I don't expect these tools by themselves to satisfy the requirements out of the box, however I do suspect that examining how these tools work will give us a good idea of how to implement the dpkg source, at least for the purposes of bst track and bst fetch. For the stage() part I think we already have most of what we need with dpkg --install --root=<staging dir> and will later look into pulling the install scriptlets and turning those into BuildStream public data as discussed previously.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @brejoc] on Apr 26, 2017, 21:59

[Gitlab user @tristanvb] Those tools are indeed very interesting! Thanks for the links.
While digging around in python-apt I had an idea. Why don't we simply mirror/download all the packages currently installed on the reference installation. Then we create a local repo (dpkg-scanpackages), which is then the only file resource used in /etc/sources.list.
That way we also wouldn't have to worry about 3rd party repos.
I haven't tested that yet, but I think that could work. What do you think?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @brejoc] on Apr 27, 2017, 19:10

I've been investigating a bit further. I could only test with Ubuntu, since I simply haven't got old enough Debian installations. I assume Debian repositories are mostly organized the same. It seems like downloading the packages from the currently set-up repos only works with a mostly up-to-date package list. As soon as there are old enough packages installed, downloading from the repos fails. The old version simply isn't there anymore. But we might be able to pull packages from snapshot.debian.org. But I'm still not sure, if the overall approach is correct. 🤔

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 28, 2017, 08:39

So you seem to be correct that older package versions are not reliably fetchable via official debian distro repositories. And also correct that snapshot.debian.org is useful for exactly this purpose, I think the tradeoff here is that the official user facing repositories have a lot of mirrors run by contributors and not everyone wants to mirror the entire history of debian.

It seems also quite plausible for an organization to run a snapshot repository of their own in the case they want to be sure that snapshots will exist for them (I dont think debian snapshots are going away, but there is not much of a guarantee they will always be there).

So, I tried asking around in #debian on OFTC a bit more, one promising approach is the possibility of running something like debootstrap inside an unprivileged namespace (bubblewrap) for this, seems there is already a thread about this.

Something like debootstrap would work, but we would want to:

• Be able to run this against a snapshot.debian.org with specific package versions in the debootstrap configuration
• Be able to run it as an unprivileged user
• Still be able to update the configuration to the "latest package versions of release foo" (for tracking)

Sorry if I've sent you on a bit of a wild goose chase, this seems to be very hard ! but the benefits of this would still be enormous... maybe we will have to settle for some middle ground in the short term, though.

Also, I hope you will be on IRC again one day if you have time to chat :)

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on May 3, 2017, 13:34

[Gitlab user @brejoc] Ok so... I'm going to leave this bug open because I still think it would be awesome to have a Source implementation that does something like this for dpkg and then another one for rpm... however...

I have found that running debootstrap as super user, is amazingly simple and gets us really most of the way.

So I just wanted to give you a heads up and I'm not sure how else to reach you... I'm planning to experiment with this this week, and perhaps later setup a service (which seems ridiculously easy) which can do some leg work for us.

What I am thinking is to have a service which:

• Has some configuration for what kinds of images you want to produce, and for what target architectures
• This configuration would only include some toplevel unversioned package names
• Periodically use this configuration to:
• Run debootstrap to create a chrootable runtime based on these packages
• Commit the result to an ostree repository under a target branch name
• Commits can have the date at which the import occurred in their commit messages

This way buildstream can work with the intermediate ostree repos in the regular way, with fully deterministic checkouts (i.e. helping with repeatability and reproducibility of builds) and still be able to track the latest import of a given debian import.

Of course, this approach means there is a moving part, which kind of sucks... but it looks like implementing the dkpg Source for this is going to be a lot of work and in the short term it will be great to experiment with this approach.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @brejoc] on May 3, 2017, 19:49

[Gitlab user @tristanvb] Sorry, for the silence! Last weekend I've been coming back to Germany from half a year of traveling through France, Spain and Portugal with a campervan. Tough switch with some challenges. Hopefully everything will be back to "normal" next week, when we also (hopefully) moved from the campervan into an apartment.

debootstrap is an amazing tool. I also came across it during my researches. Good to see, that this can be leveraged despite the downside. Seems like I have to wrap my brain also around ostree in the near future. I've not worked with it yet.

Although a dpkg source might not be the most easy task, I've got give up yet. I'll hopefully be able to dedicate a bit more time in the next weeks and also show up in IRC again.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @knownexus] on Feb 25, 2018, 22:31

We have been discussing the concept of allowing the usage of pre-built dpkg sources to be used as sources in bst projects.
We have, so far, come up with 2 options:

1. Create a source plugin
   This would import packges and extract the content, allowing them to be built by buildstream and used as a normal source

2. Modifying the produced artifact to make buildstream treat it like a bst generated artifact
   This would allow the use of pre-built artifacts without re-building them,
   However this seems like it would be far more limited in its usability as the artifacts would need to be generated in a known way.
   This would also make it more difficult to update the dependencies, as you would need to replace the artifact whenever you wanted to update a related source

For now, we've decided on option (1), as it is more generalised and more maintainable.

Here's what i've come up with for the creation of the DPKG Source Plugin:

Have the binary fetched from an online repository - stated in the element.bst file
Using _downloadablefilesource.py

Generate a cache key for the binary

Extract the file like a tar into a set dir - either relative or stated in the element.bst file

Remove tmp dir and contents

Limit the contents to the base dir and it's contents - stated in the element.bst file

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @juergbi] on Mar 1, 2018, 08:52

1. [...] allowing them to be built by buildstream

By this I assume you mean creating a BuildStream artifact from the contents of a .deb, not actually building anything from sources. Do I understand this correctly?

I definitely agree on option (1) being the more maintainable choice.

Limit the contents to the base dir and it's contents - stated in the element.bst file

Do you have a use case in mind for this?

Will any metadata from the .deb be used? E.g., runtime dependencies or integration commands? If not, what's the plan to handle dependencies? Require manually maintaining the dependency list in the .bst file? Generate .bst files with external tool?