Skip to content
This repository has been archived by the owner. It is now read-only.

pmbootstrap build --src: override source for any package #1210

Merged
merged 5 commits into from Feb 19, 2018

Conversation

@ollieparanoid
Copy link
Member

ollieparanoid commented Feb 6, 2018

This is outdated, updated version below.

Closes #1041. CC: @ata2001, @bhush9, @drebrez, @filippz, @flacks, @fourkbomb, @opendata26, @pavelmachek, @z3ntu

Changes

  • pmbootstrap init allows to specify a local linux kernel repository
  • New --local-src parameter for pmbootstrap build:
    pmbootstrap build linux-postmarketos-mainline --arch=armhf --local-src
    • Mounts the kernel source to /mnt/pmbootstrap-local-src inside the
      chroot
    • exports PMB_LOCAL_SRC=/mnt/pmbootstrap-local-src to abuild
  • pmb/helpers/frontend.py: Verifies, that the packages passed to
    pmbootstrap build use PMB_LOCAL_SRC when --local-src is
    specificed and raise an exception if they don't
  • linux-postmarketos-mainline APKBUILD adjustments:
    • update linux-v4.14-v4.15-rc6.patch checksum to upstream (they
      probably updated their git/patch tool, which caused a different
      checksum)
    • fetch(): don't download anything when using --local-src
    • unpack(): don't extract anything, but create a link to
      $PMB_LOCAL_SRC if it is specified
    • prepare(): don't apply any patches to the kernel source folder
      when using --local-src
    • package(): change the pkgdesc when using --local-src

Workflow

Initial setup:

$ pmbootstrap init # select any device, configure kernel repo path
$ pmbootstrap install

Regular testing:

$ pmbootstrap build --force --local-src --arch=armhf linux-postmarketos-mainline
$ pmbootstrap chroot -r -- apk del linux-postmarketos-mainline
$ pmbootstrap chroot -r -- apk add linux-postmarketos-mainline # rebuilds initfs, boot.img, ...
$ pmbootstrap flasher --flavor=postmarketos-mainline boot

The del, add could be made smoother in a future PR (unless someone has an idea how to do it in this one easily), but with this PR we have a first version where it's possible to build kernel trees from a local folder.

@z3ntu: regarding clean up or not: the way the APKBUILD worked already, a "build" folder gets created, in which the actual compilation takes place. we don't keep that during builds (could be implemented if necessary), but we have ccache which already makes it a lot faster. We need such a folder anyway, because the pmos user we use to build does not have permissions to write into the (outside-of-pmbootstrap) user's folder.

@drebrez

This comment has been minimized.

Copy link
Member

drebrez commented Feb 6, 2018

@ollieparanoid Nice job, I'm going to test it probably next week, I'll be away the next days.
I usually do pmbootstrap chroot -r -- apk add -u /mnt/pmbootstrap-packages/armhf/linux-postmarketos-mainline-xxx-yyy.apk, the exact path (with the actual version) is printed just few lines above during the build.

@bhush9

This comment has been minimized.

Copy link
Collaborator

bhush9 commented Feb 7, 2018

@ollieparanoid Now that I take a look at this from different perspective.. how hard it would be to allow to build any package from different git repository or svn checkout or whatever? Let's say sometime someone wants to build kwin from the unreleased git repository .. I wonder if you can make this solution generic enough to be used by all other packages?

@ollieparanoid

This comment has been minimized.

Copy link
Member Author

ollieparanoid commented Feb 7, 2018

@bhush9: I've thought about that and it would not be too hard actually. Just let pmbootstrap accept a string as --local-src, and default to the kernel in case it is not defined.
With that being said, the APKBUILDs would need to be adjusted to use PMB_SRC_DIR, and the build would need to be outside of the source tree (that's totally possible with cmake/kde stuff, right?)

So if you still think it makes sense, I'd extend this PR to allow using --local-src for all packages.

@bhush9

This comment has been minimized.

Copy link
Collaborator

bhush9 commented Feb 8, 2018

(that's totally possible with cmake/kde stuff, right?)

Yeah right.. it is possible with cmake packages and also it is recommended way of doing things tbh in case of cmake.. I remember lots of KDE packages doesn't support in-source-tree builds and needs seperate build dir for ease of doing cmake

@pavelmachek

This comment has been minimized.

Copy link
Member

pavelmachek commented Feb 8, 2018

@ollieparanoid

This comment has been minimized.

Copy link
Member Author

ollieparanoid commented Feb 9, 2018

Thanks a lot for the feedback, I'll update this to support all packages.

Please wait with testing/reviewing until the new version is up.

@ollieparanoid ollieparanoid force-pushed the feature/1041-kernel-local-git-repo branch from fa3968d to e5a4573 Feb 9, 2018

@ollieparanoid

This comment has been minimized.

Copy link
Member Author

ollieparanoid commented Feb 9, 2018

Overview

I have rewritten it with a focus on supporting all packages, not just the Linux kernel. Now it can be used like that:

% pmbootstrap build --src=/tmp/kwindowsystem kwindowsystem
[23:23:11] (native) build x86_64/kwindowsystem-5.42.0_p20180209232311-r0.apk (source: /tmp/kwindowsystem)

Changes

  • --local-src was shortened to --src.
  • The pkgver gets changed to include the date and time of the current build, with a _p suffix.
  • It works with all APKBUILDs now without modification, as long as they expect the source to be in builddir (which is the modern way of doing it in APKBUILDs).
  • Building outside of the source folder is not a requirement anymore, since the we're using a (rw) copy of the source folder during the build process.
  • When compiling the same package multiple times in a row with the same local source folder, the copying part is almost instant, because rsync is used to only copy the files that have been changed.
  • Adjusted linux-postmarketos-mainline to use builddir

I've played with the thought of defaulting to the kernel path, and letting the user specify that in pmbootstrap init (like it was possible in the first commit). However, that would add additional complexitiy with very little benefit in my opinion. So as it's implemented now, you'd need to specify the source folder of the Linux kernel as well, when building the Linux kernel:

$ pmbootstrap build --src=~/code/linux linux-postmarketos-mainline

Avoiding PMB_LOCAL_SRC by appending to the APKBUILD

APKBUILD files are simple shell scripts, and when a variable or function gets defined multiple times in it, the last one overrides the previous ones. So after copying the APKBUILD into the chroot, we generate and append the following block to make it use the local source:

# ** Overrides below appended by pmbootstrap for --src **

pkgver="5.42.0_p20180209232311"
pkgdesc="[/tmp/kwindowsystem] $pkgdesc"
_pmb_src_copy="/tmp/pmbootstrap-local-source-copy"

# Empty $source avoids patching in prepare()
_pmb_source_original="$source"
source=""
sha512sums=""

fetch() {
    # Update source copy
    msg "Copying source from host system: /tmp/kwindowsystem"
    rsync -a --exclude=".git/" --delete \
        "/mnt/pmbootstrap-source-override/" "$_pmb_src_copy"

    # Link local source files (e.g. kernel config)
    mkdir "$srcdir"
    local s
    for s in $_pmb_source_original; do
        is_remote "$s" || ln -sf "$startdir/$s" "$srcdir/"
    done

}

unpack() {
    ln -sv "$_pmb_src_copy" "$builddir"
}

@ollieparanoid ollieparanoid changed the title Support compiling mainline kernel from local git repository pmbootstrap build --src: override source for any package Feb 9, 2018

@z3ntu

This comment has been minimized.

Copy link
Member

z3ntu commented Feb 10, 2018

@ollieparanoid REALLY great work!!

I tried it with a package I have locally in my tree

(029401) [10:07:35] (native) % cd /home/pmos/build && busybox su pmos -c 'CARCH=x86_64 SUDO_APK='"'"'abuild-apk --no-progress'"'"' abuild -d -f'
>>> razergenie: abuild 3.1.0-r4
>>> razergenie: Checking sanity of /home/pmos/build/APKBUILD...
>>> WARNING: razergenie: No maintainer
>>> razergenie: Cleaning temporary build dirs...
>>> razergenie: Copying source from host system: /home/luca/Dev/RazerGenie
rsync: opendir "/mnt/pmbootstrap-source-override/cache" failed: Permission denied (13)
rsync: opendir "/mnt/pmbootstrap-source-override/tmp" failed: Permission denied (13)
IO error encountered -- skipping file deletion
rsync: send_files failed to open "/mnt/pmbootstrap-source-override/.kdev4/RazerGenie.kdev4": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1189) [sender=3.1.3]
>>> ERROR: razergenie: all failed
(029401) [10:07:35] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(029401) [10:07:35] NOTE: The failed command's output is above the ^^^ line in the logfile: /home/luca/.local/var/pmbootstrap/log.txt

zap didn't help

EDIT: I don't know how the cache and tmp folder got into existence, I deleted them.
Now it's failing with the file /mnt/pmbootstrap-source-override/.kdev4/RazerGenie.kdev4 which has -rw------- permissions (i.e. is not readable by the chroot user). Can this be handled in some way properly?
But after chmod'ing the build process worked :)

@ollieparanoid

This comment has been minimized.

Copy link
Member Author

ollieparanoid commented Feb 10, 2018

@z3ntu: thanks for testing! I've added flags to rsync now to ignore unreadable files (and added || true to ignore if its exit code is not 0), and added an unreadable file to the test case to make sure it gets ignored properly.

While doing that I've also noticed that abuild complains if the pkgdesc is too long, so it gets cut off now. And relative paths passed to --src were not converted to absolute paths yet, that is fixed as well.

@ollieparanoid ollieparanoid force-pushed the feature/1041-kernel-local-git-repo branch from b7c6fc9 to bc5054a Feb 10, 2018

@steamp0rt

This comment has been minimized.

Copy link
Contributor

steamp0rt commented Feb 18, 2018

please add support for this in menuconfig, too

@ollieparanoid

This comment has been minimized.

Copy link
Member Author

ollieparanoid commented Feb 18, 2018

I'd prefer if we get this PR merged first (review/testing would be appreciated!) since it should be rock stable right now. Enabling the feature for menuconfig could be done in a follow up PR.

@ollieparanoid ollieparanoid force-pushed the feature/1041-kernel-local-git-repo branch from bc5054a to 8968540 Feb 18, 2018

@z3ntu
Copy link
Member

z3ntu left a comment

didn't see anything bad otherwise :)

:param original_source: the original source is used instead of
overriding it with --src.
"""

This comment has been minimized.

Copy link
@z3ntu

z3ntu Feb 18, 2018

Member

the parameters original_pkgver and now are not in the docstring

return original_pkgver

# Append current date
no_suffix = original_pkgver.split("_", 1)[0]

This comment has been minimized.

Copy link
@z3ntu

z3ntu Feb 18, 2018

Member

So this takes everything before the first underscore but what does that achieve?

This comment has been minimized.

Copy link
@ollieparanoid

ollieparanoid Feb 19, 2018

Author Member

When the original pkgver already has a suffix, e.g. 1.0.0_git20180101, then the old suffix gets replaced. A pkgver with 2 suffixes is invalid, so this needs to be done. I'll update the docstring accordingly, thanks for reviewing! \o/

ollieparanoid added some commits Feb 6, 2018

Support compiling mainline kernel from local git repository
* `pmbootstrap init` allows to specify a local linux kernel repository
* New `--local-src` parameter for `pmbootstrap build`:
  `pmbootstrap build linux-postmarketos-mainline --arch=armhf --local-src`
  - Mounts the kernel source to /mnt/pmbootstrap-local-src inside the
    chroot
  - exports `PMB_LOCAL_SRC=/mnt/pmbootstrap-local-src` to `abuild`
* `pmb/helpers/frontend.py`: Verifies, that the packages passed to
  `pmbootstrap build` use `PMB_LOCAL_SRC` when `--local-src` is
  specificed and raise an exception if they don't
* linux-postmarketos-mainline APKBUILD adjustments:
  - update linux-v4.14-v4.15-rc6.patch checksum to upstream (they
    probably updated their git/patch tool, which caused a different
    checksum)
  - fetch(): don't download anything when using `--local-src`
  - unpack(): don't extract anything, but create a link to
    `$PMB_LOCAL_SRC` if it is specified
  - prepare(): don't apply any patches to the kernel source folder
    when using `--local-src`
  - package(): change the `pkgdesc` when using `--local-src`
Allow building any package with local source
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum

ollieparanoid added some commits Feb 10, 2018

Various fixes
* rsync: ignore unreadable files (also adjusted build --src testcase to
  have an unreadable file)
* pkgdesc: cut off before the maximum character length gets reached and
  abuild aborts the build because of that
* use os.path.realpath, so when passing a relative path to --src it
  gets replaced with the absolute path

@ollieparanoid ollieparanoid force-pushed the feature/1041-kernel-local-git-repo branch from 8968540 to 80223ed Feb 19, 2018

@ollieparanoid ollieparanoid merged commit 0f371e4 into master Feb 19, 2018

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.05%) to 75.151%
Details

@ollieparanoid ollieparanoid deleted the feature/1041-kernel-local-git-repo branch Feb 19, 2018

PureTryOut added a commit that referenced this pull request Feb 20, 2018

pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum

PureTryOut added a commit that referenced this pull request Feb 21, 2018

pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
@scintill

This comment has been minimized.

Copy link
Contributor

scintill commented Mar 2, 2018

Thanks for this feature! I think I found a bug on architecture-specific packages. The following won't build with --src. (I used an empty folder for testing, but I originally found the bug with non-empty folder.)

pkgname=test-build-src
pkgver=0.0
pkgrel=0
pkgdesc="demonstrate bug in pmbootstrap build --src"
url="https://github.com/"
arch="armhf"
license="unknown"

package() {
	mkdir -p "$pkgdir"
}

It builds without --src, or, if arch="all" then it will build succesfully with or without --src. The error message is something like the following:

(012655) [22:41:20] (native) % rm /tmp/APKBUILD.append
rm: can't remove '/tmp/APKBUILD.append': No such file or directory

I think the issue has something to do with it trying rm in the native chroot when it should be using the buildroot.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.