Build pipeline execution setup

[mih]

[supersedes the respective parts of #1]

The current draft is:

The container runscript implements the pipeline, hence we call singularity run.

Here is an annotated call outlining the setup:

singularity run

• --bind cache/var/lib/apt:/var/lib/apt,cache/var/cache/apt:/var/cache/apt,.:/pkg
  Bind mounts for the container execution. Must exist before singularity is invoked. cache/ brings in a persistent cache to speed up repeated builds. However, this is entirely optional, and maybe not useful for submitted build jobs. An apt-cacher setup would achieve something similar, but with more effort.
  .:/pkg simplifies the runscript, because we can be sure where to look for the source package and where to place the binary packages. Using . for this is YODA-style. All relative to the root of the superdataset. In this case, this is the package dataset, which has the builder dataset as a subdataset.
• --pwd /pkg simplify the logic, PWD inside and outside the container will be the same (albeit with different names), so relative paths can stay as they are
• --fakeroot frees us from having to use fakeroot features of the Debian toolchain, inside the container things will be done by "root"
• --cleanenv brings environment isolation for the build. However, we might want to specifically support passing in DEB_BUILD_OPTIONS to be able to tune the builds (e.g. nocheck, see `man debhelper)
• --containall presently unclear to me, why I included it -- needs verification
• --writable needed because we have to update the system inside the container before the build. However, recent versions have --writable-tmpfs which may be the better approach, needs verification and version dependency checked
• --no-home further isolation, we do not need to talk to user HOME
• <name of the image>.sif
• <path to DSC file>

[aqw]

--writable-tmpfs seems to store all modifications in memory. It makes things fast, but it means that the RAM requirements for building could become sizeable.

The question for me is where the temporary build artifacts will live (e.g. debian/<packagename>/. If in /pkg, then those modifications are outside of the container image, and then the tmpfs requirements are pretty much limited to deps. But if those artifacts will be written in the image, then that is a lot of modifications to store in memory, and I think we are better off not using --writeable-tmpfs.

Data packages and freesurfer/matlab packages can be quite large.

[aqw]

I do recommend that we use --containall. It is still possible to bind mount your homedir into the image while using --no-home if your CWD is your homedir.

--containall does not have such a loop-hole.

It is unclear to me whether --containall obviates the need for --no-home.

[mih]

Here is a complete implementation of the pipeline, using the envisioned structure of the associated DataLad datasets (see #6), but without actually creating/configuring them.

The following needs a Debian-based system with singularity-container and the devscripts package (maybe also build-essentials).

# simulate a future package dataset
mkdir pkg
cd pkg

# populate with a Debian source package
dget -d http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-2.dsc

# simulate a future builder subdataset 
mkdir builder
mkdir -p builder/cache/var/cache/apt
mkdir -p builder/cache/var/lib/apt

cat << EOT > builder/bullseye.def
Bootstrap:docker
From:debian:bullseye

%post
    apt-get -y update
    apt-get -y install --no-install-recommends build-essential devscripts eatmydata equivs
    # remove everything but the lock file from
    # /var/cache/apt/archives/ and /var/cache/apt/archives/partial/
    apt-get clean
    # builder setup relies on this particular directory to exist inside the container
    mkdir /pkg

%runscript
    # we run with -x to get verbose output suitable for a comprehensive
    # log file
    set -e -u -x
    dsc=\$1
    # somehow --containall causes 755 by default
    chmod 777 /tmp
    # update the package list, necessary to get the latest build-dependency
    # version
    eatmydata apt update -y
    # update the base environment. needed for rolling releases to stay
    # up-to-date
    eatmydata apt upgrade -y
    # all building will take place inside the container
    # maybe add option to do it on the host FS?
    mkdir -p /tmp/build
    # ingest the source package into a defined location in the container env
    dcmd cp "\$dsc" /tmp/build
    # extract the source package
    (cd /tmp/build && dpkg-source -x *.dsc source)
    # install the declared build-dependencies
    (cd /tmp && mk-build-deps -t 'eatmydata apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y' -i -r /tmp/build/source/debian/control)
    # build the binary package(s)
    (cd /tmp/build/source && debuild -uc -us -b)
    # deposit the results
    dcmd cp /tmp/build/*changes /pkg
EOT

# bootstrap the build pipeline
sudo singularity build builder/bullseye.sif builder/bullseye.def

# build the binary package(s)
singularity run --bind builder/cache/var/lib/apt:/var/lib/apt,builder/cache/var/cache/apt:/var/cache/apt,.:/pkg --pwd /pkg --fakeroot --cleanenv --containall --writable --no-home builder/bullseye.sif hello_2.10-2.dsc

[mih]

And here is the "same" as the above, but order slightly shuffled to get closer to the final concept. Now with full provenance tracking via datalad:

First the builder dataset. The same dataset is used for all packages

datalad create builder
cd builder


# the next few commands until the `save`will go into a convenience command that can tailor
# a recipe template to a specific distribution release
cat << EOT > bullseye.def
Bootstrap:docker
From:debian:bullseye

%post
    apt-get -y update
    apt-get -y install --no-install-recommends build-essential devscripts eatmydata equivs
    # remove everything but the lock file from
    # /var/cache/apt/archives/ and /var/cache/apt/archives/partial/
    apt-get clean
    # builder setup relies on this particular directory to exist inside the container
    mkdir /pkg

%runscript
    # we run with -x to get verbose output suitable for a comprehensive
    # log file
    set -e -u -x
    dsc=\$1
    # somehow --containall causes 755 by default
    chmod 777 /tmp
    # update the package list, necessary to get the latest build-dependency
    # version
    eatmydata apt update -y
    # update the base environment. needed for rolling releases to stay
    # up-to-date
    eatmydata apt upgrade -y
    # all building will take place inside the container
    # maybe add option to do it on the host FS?
    mkdir -p /tmp/build
    # ingest the source package into a defined location in the container env
    dcmd cp "\$dsc" /tmp/build
    # extract the source package
    (cd /tmp/build && dpkg-source -x *.dsc source)
    # install the declared build-dependencies
    (cd /tmp && mk-build-deps -t 'eatmydata apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y' -i -r /tmp/build/source/debian/control)
    # build the binary package(s)
    (cd /tmp/build/source && debuild -uc -us -b)
    # deposit the results
    dcmd cp /tmp/build/*changes /pkg
EOT
echo 'cache' > .gitignore
datalad save --to-git -r

# build the container, with prov-tracking
# ultimately needs to be done repeatedly, could be just a `rerun`,
# to pick up new package version
datalad run -i bullseye.def -o bullseye.sif 'sudo singularity build {outputs} {inputs}'

# register the image with datalad under a particular name
datalad containers-add -d . -i bullseye.sif --call-fmt 'singularity run --bind builder/cache/var/lib/apt:/var/lib/apt,builder/cache/var/cache/apt:/var/cache/apt,.:/pkg --pwd /pkg --fakeroot --cleanenv --containall --writable --no-home {img} {cmd}' build-bullseye

and now a dataset for a particular Debian package

cd ..
# the will be a configuration procedure to automate some setup steps below
datalad create pkg
cd pkg
# we register the builder dataset
datalad clone -d . ../builder builder

# populate the dataset with a source package version
# this could be any process (here download, but could be git-buildpackage)
# there could also be another subdataset with the actual sources
# whatever is needed for a particular package
datalad run dget -d http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-2.dsc

# setup runtime env for the build pipeline
# this would go into a convenience command, together with the containers-run call
mkdir -p builder/cache/var/cache/apt
mkdir -p builder/cache/var/lib/apt

# it is as simple as calling the container with the source package
datalad containers-run -n builder/build-bullseye hello_2.10-2.dsc

The outcomes are the binary packages, neatly tied to build pipeline version and all the rest.

TODO: capture the logs and save them too

[mih]

I wrote a basic configuration procedure for the builder package (see #13)

datalad create -c debianbuilder builder
cd builder

cat << EOT > recipes/bullseye.def
Bootstrap:docker
From:debian:bullseye

%post
    apt-get -y update
    apt-get -y install --no-install-recommends build-essential devscripts eatmydata equivs
    # remove everything but the lock file from
    # /var/cache/apt/archives/ and /var/cache/apt/archives/partial/
    apt-get clean
    # builder setup relies on this particular directory to exist inside the container
    mkdir /pkg

%runscript
    # we run with -x to get verbose output suitable for a comprehensive
    # log file
    set -e -u -x
    dsc=\$1
    # somehow --containall causes 755 by default
    chmod 777 /tmp
    # update the package list, necessary to get the latest build-dependency
    # version
    eatmydata apt-get update -y
    # update the base environment. needed for rolling releases to stay
    # up-to-date
    eatmydata apt-get upgrade -y
    # all building will take place inside the container
    # maybe add option to do it on the host FS?
    mkdir -p /tmp/build
    # ingest the source package into a defined location in the container env
    dcmd cp "\$dsc" /tmp/build
    # extract the source package
    (cd /tmp/build && dpkg-source -x *.dsc source)
    # install the declared build-dependencies
    (cd /tmp && mk-build-deps -t 'eatmydata apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y' -i -r /tmp/build/source/debian/control)
    # build the binary package(s)
    (cd /tmp/build/source && debuild -uc -us -b)
    # deposit the results
    dcmd cp /tmp/build/*changes /pkg
EOT
datalad save --to-git     
datalad run -i recipes/bullseye.def -o envs/bullseye.sif 'sudo singularity build {outputs} {inputs}'
datalad containers-add -d . -i envs/bullseye.sif --call-fmt 'singularity run --bind builder/cache/var/lib/apt:/var/lib/apt,builder/cache/var/cache/apt:/var/cache/apt,.:/pkg --pwd /pkg --fakeroot --cleanenv --containall --writable --no-home {img} {cmd}' build-bullseye

[aqw]

Overall, this all looks good to me. It makes sense.

One comment: you mix apt-get and apt. You should only use the former for scripts. apt-get's interface is meant to be stable, whereas apt's "feature" is that it can (and does) change to improve ergonomics, etc.

[mih]

Thx @aqw ! I adjusted the last post with this change.

[mih]

Following through on the definition of the builder dataset layout from #8, the setup shown above needs to be adjusted to use architecture specific names for the build pipeline images. The Debian tooling has a dedicated solution for this, like so:

datalad run -i recipes/bullseye.def -o "envs/bullseye_$(dpkg-architecture -q DEB_BUILD_ARCH).sif" 'sudo singularity build {outputs} {inputs}'

(look for dpkg-architecture)

[adswa]

Here is a quick-and-dirty pipeline illustration in terms of datasets. Only the left part is relevant, the text is just code already outlined here in this issue

[mih]

With #29 a full setup+build (from scratch) can look like this now....

Bootstrap a distribution with a builder

# from scratch, a new distribution
datalad deb-new-distribution mystuff
cd mystuff
# give the builder dataset a "globally" accessible URL
datalad subdatasets --set-property url $(pwd)/builder builder

# configure and create a builder
datalad deb-configure-builder -d builder dockerbase=debian:bullseye
datalad -C builder run -i recipes/singularity-any -o "envs/singularity_$(dpkg-architecture -q DEB_BUILD_ARCH).sif" 'sudo singularity build {outputs} {inputs}'
datalad containers-add -d builder -i "envs/singularity_$(dpkg-architecture -q DEB_BUILD_ARCH).sif" --call-fmt 'singularity run --bind builder/cache/var/lib/apt:/var/lib/apt,builder/cache/var/cache/apt:/var/cache/apt,.:/pkg --pwd /pkg --fakeroot --cleanenv --containall --writable --no-home {img} {cmd}' "singularity-$(dpkg-architecture -q DEB_BUILD_ARCH)"
datalad save -d . -m "Update distribution builder" builder

Add a package (to the distribution)

datalad deb-new-package hello

Update a package

The following steps would take place in the context of a single package and do not require the distribution package context. They would also work in a fresh (temporary) clone of the package dataset

# (cd'ing into the existing clone for this example, but could also be a clone call)
cd packages/hello
datalad run -m "Add version 2.10-2 source package" dget -d http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-2.dsc

Build a package

# needed, even when no `update` is intended because we want to establish a cache dir inside the build dataset
# (at least for now)
datalad get -n builder
# optionally pull in the latest builder updates
datalad update --merge -r builder

# setup runtime env for the build pipeline
# this would go into a convenience command, together with the containers-run call
mkdir -p builder/cache/var/cache/apt
mkdir -p builder/cache/var/lib/apt

datalad containers-run -n builder/singularity-amd64 hello_2.10-2.dsc

Update a package within a distribution

Again using the existing dataset hierarchy of this example, but could be done in a lean clone, just pull an update for a package dataset and saving it

cd ../..
datalad save -m "Package update" packages

[mih]

With #31 the "Bootstrap a distribution with a builder" simplifies to

datalad deb-new-distribution mystuff
cd mystuff
# give the builder dataset a "globally" accessible URL
datalad subdatasets --set-property url $(pwd)/builder builder

# configure and create a builder
datalad deb-configure-builder -d builder dockerbase=debian:bullseye
datalad deb-bootstrap-builder -d builder 
datalad save -d . -m "Update distribution builder" builder

[mih]

With #32 this all simplifies to

Bootstrap a distribution with a builder (done once in a distribution's lifetime)

datalad deb-new-distribution mystuff
cd mystuff
# give the builder dataset a "globally" accessible URL for this demo
datalad subdatasets --set-property url $(pwd)/builder builder

# configure and create a builder
datalad deb-configure-builder -d builder dockerbase=debian:bullseye
datalad deb-bootstrap-builder -d builder 
datalad save -d . -m "Update distribution builder" builder

Update a distribution builder (done as necessary)

# will ask to confirm container override ... TODO prevent...
datalad deb-bootstrap-builder -d builder 
datalad save -d . -m "Update distribution builder" builder

Add a package to the distribution (done once in a distribution's lifetime)

datalad deb-new-package hello

Update a package (done as necessary)

The following steps would take place in the context of a single package and do not require the distribution package context. They would also work in a fresh (temporary) clone of the package dataset

# (cd'ing into the existing clone for this example, but could also be a clone call)
cd packages/hello
# here is downloads a source package, but it could also gbp a source package from a subdataset
# i.e., a linked (upstream) repo
datalad run -m "Add version 2.10-2 source package" dget -d http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-2.dsc

Build a package (done once per source package update and CPU architecture)

datalad deb-build-package hello_2.10-2.dsc

Update a package within a distribution (done as necessary)

Again using the existing dataset hierarchy of this example, but could be done in a lean clone, just pull an update for a package dataset and saving it

cd ../..
datalad save -m "Package update" packages

[christian-monch]

datalad containers-add -d builder -i "envs/singularity_$(dpkg-architecture -q DEB_BUILD_ARCH).sif" --call-fmt 'singularity run --bind builder/cache/var/lib/apt:/var/lib/apt,builder/cache/var/cache/apt:/var/cache/apt,.:/pkg --pwd /pkg --fakeroot --cleanenv --containall --writable --no-home {img} {cmd}' "singularity-$(dpkg-architecture -q DEB_BUILD_ARCH)"

small error in the cmdline: "_" instead of "-" in "envs/singularity_$(dpkg-architecture ...).sif" It should be:

datalad containers-add -d builder -i "envs/singularity-$(dpkg-architecture -q DEB_BUILD_ARCH).sif" --call-fmt 'singularity run --bind builder/cache/var/lib/apt:/var/lib/apt,builder/cache/var/cache/apt:/var/cache/apt,.:/pkg --pwd /pkg --fakeroot --cleanenv --containall --writable --no-home {img} {cmd}' "singularity-$(dpkg-architecture -q DEB_BUILD_ARCH)"

[loj]

I was able to successfully complete all steps outlined in #12 (comment)

[mslw]

I was also able to complete these steps.

[jsheunis]

Should I be able to complete all the steps outlined in this comment on a mac? These steps work fine:

datalad deb-new-distribution mystuff
cd mystuff
# give the builder dataset a "globally" accessible URL for this demo
datalad subdatasets --set-property url $(pwd)/builder builder

# configure and create a builder
datalad deb-configure-builder -d builder dockerbase=debian:bullseye

but getting an error when running this:

>> datalad deb-bootstrap-builder -d builder 
[ERROR  ] [Errno 2] No such file or directory: 'dpkg-architecture' (FileNotFoundError)

>> datalad --dbg deb-bootstrap-builder -d builder 
Traceback (most recent call last):
  File "/Users/jsheunis/opt/miniconda3/envs/debian/bin/datalad", line 8, in <module>
    sys.exit(main())
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/cli/main.py", line 140, in main
    _run(cmdlineargs)
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/cli/main.py", line 164, in _run
    ret = _run_with_debugger(namespace) \
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/cli/main.py", line 177, in _run_with_debugger
    return cmdlineargs.func(cmdlineargs)
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/cli/exec.py", line 107, in call_from_parser
    ret = list(ret)
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/interface/utils.py", line 357, in generator_func
    for r in _process_results(
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/interface/utils.py", line 544, in _process_results
    for res in results:
  File "/Users/jsheunis/Documents/psyinf/datalad-debian/datalad_debian/bootstrap_builder.py", line 81, in __call__
    binarch = Runner().run(
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/runner/runner.py", line 184, in run
    results_or_iterator = self.threaded_runner.run()
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/runner/nonasyncrunner.py", line 307, in run
    return self._locked_run()
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/site-packages/datalad/runner/nonasyncrunner.py", line 355, in _locked_run
    self.process = subprocess.Popen(self.cmd, **kwargs)         # nosec
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'dpkg-architecture'

> /Users/jsheunis/opt/miniconda3/envs/debian/lib/python3.9/subprocess.py(1821)_execute_child()
-> raise child_exception_type(errno_num, err_msg, err_filename)

[adswa]

@jsheunis does Mac have a dpkg-architecture command, or is it available for Mac? I think it fails to find this command, which on Linux lists the CPU architecture of your system. If it isn't available on mac, do you know if there is an alternative command for this?

The output on my system looks like this:

 dpkg-architecture -q DEB_BUILD_ARCH
amd64

[jsheunis]

There is a brew formula for dpkg: https://formulae.brew.sh/formula/dpkg. After installation, this runs as follows:

>> dpkg-architecture -q DEB_BUILD_ARCH
darwin-amd64

But it looks like the building process won't work in any case as @mih pointed out in this comment: #103 (comment)