This repository has been archived by the owner. It is now read-only.

Refactor pmb/build/package.py (make depends work like in abuild) #935

Merged
merged 3 commits into from Nov 26, 2017

Conversation

@ollieparanoid
Member

ollieparanoid commented Nov 25, 2017

This PR will fix #824 (wrong depends handling), which is the last blocker before merging the Plasma Mobile PR. And it will allow extending package code again, because the code is split up in small functions and these have test cases, with new features like native cross compilation for autoconf packages (see @zhuowei's proof of concept in #665). Needless to say, testing and reviewing is greatly appreciated.

EDIT: Test cases ran through locally, but fail with Travis. Fixed! 馃憤

Change of behavior

pmbootstrap build device-samsung-i9100 explains it all:

[01:18:04] Dependency handling in 'pmbootstrap build' has been changed.
[01:18:04] Previously we only built and installed the 'makedepends' from the APKBUILDs, now we use the 'depends', too.
[01:18:04]
[01:18:04] Your options:
[01:18:04] * Ignore depends (fast, old behavior, may cause problems with some packages):
[01:18:04]   pmbootstrap build device-samsung-i9100 -i
[01:18:04] * Build with depends (kernel!) and specify the right architecture:
[01:18:04]   pmbootstrap build device-samsung-i9100 --arch=armhf
[01:18:04]
[01:18:04] This change was necessary to be more compatible with Alpine's abuild.
[01:18:04] The default architecture is the native one (x86_64 in your case), so you need to overwrite
[01:18:04] it now to get the kernel dependency of your device package for the right architecture.
[01:18:04] Sorry for the inconvenience.
[01:18:04]
[01:18:04] ERROR: Missing -i or --arch parameter
[01:18:04] Run 'pmbootstrap log' for details.
[01:18:04] See also: <https://postmarketos.org/troubleshooting>

How to test (easy!)

  • Build packages like you would normally do
  • Do full installations

Detailed changes

  • Rename pmb/build/package.py to pmb/build/_package.py, so we can
    access the functions it contains in testcases, and still use
    pmb.build.package()
  • Refactor the entire file. Instead of one big function that does
    too many things, we have many small ones now, that are tested
    in the testsuite and easier to modify
  • Whenever building a package, pmbootstrap does not only build and
    install the "makedepends" (like we did before), now it does the
    same for the "depends". That's required to be compatible with
    abuild. The old behavior can still be used with 'pmbootstrap
    build --ignore-depends'.
  • Because of that change, noarch packages can no longer be built in
    the native chroot if we need them for a foreign chroot. A device-
    package depending on a kernel would pull in the same kernel for
    the native architecture otherwise.
  • Running 'pmbootstrap build device-...' without '--ignore-depends'
    and without a matching '--arch' displays a note that explains
    this change to the user and tells how to use it instead.
  • Noarch packages no longer get symlinked. That was only
    implemented for packages built in the native chroot, and now that
    is not always the case anymore. Symlinking these packages creates
    packages with broken dependencies anyway (e.g.
    device-samsung-i9100 can't be installed in x86_64, because
    linux-samsung-i9100 is armhf only).
  • Rename "carch" to "arch" wherever used. Naming it "carch"
    sometimes is confusing with no benefit.
  • Add a testcase for the aarch64 qemu workaround (because it failed
    first and I needed to know for sure if it is working again).
  • Improved some verbose logging, which helped with development of
    this feature.
  • Removed the old "build" test case (which was disabled in
    testcases_fast.sh) as the new "build_package" test case covers its
    functionallity.
Refactor pmb/build/package.py (make depends work like in abuild)
* Rename pmb/build/package.py to pmb/build/_package.py, so we can
  access the functions it contains in testcases, and still use
  pmb.build.package()
* Refactor the entire file. Instead of one big function that does
  too many things, we have many small ones now, that are tested
  in the testsuite and easier to modify
* Whenever building a package, pmbootstrap does not only build and
  install the "makedepends" (like we did before), now it does the
  same for the "depends". That's required to be compatible with
  abuild. The old behavior can still be used with 'pmbootstrap
  build --ignore-depends'.
* Because of that change, noarch packages can no longer be built in
  the native chroot if we need them for a foreign chroot. A device-
  package depending on a kernel would pull in the same kernel for
  the native architecture otherwise.
* Running 'pmbootstrap build device-...' without '--ignore-depends'
  and without a matching '--arch' displays a note that explains
  this change to the user and tells how to use it instead.
* Noarch packages no longer get symlinked. That was only
  implemented for packages built in the native chroot, and now that
  is not always the case anymore. Symlinking these packages creates
  packages with broken dependencies anyway (e.g.
  device-samsung-i9100 can't be installed in x86_64, because
  linux-samsung-i9100 is armhf only).
* Rename "carch" to "arch" wherever used. Naming it "carch"
  sometimes is confusing with no benefit.
* Add a testcase for the aarch64 qemu workaround (because it failed
  first and I needed to know for sure if it is working again).
* Improved some verbose logging, which helped with development of
  this feature.
* Removed the old "build" test case (which was disabled in
  testcases_fast.sh) as the new "build_package" test case covers its
  functionallity.
@zhuowei

This comment has been minimized.

Show comment
Hide comment
@zhuowei

zhuowei Nov 25, 2017

Collaborator

I ran into a dependency error when compiling the luneos branch, as luna-service2 was in luna-sysmgr's depends but not makedepends. I'm currently testing if this PR fixes that issue.

Collaborator

zhuowei commented Nov 25, 2017

I ran into a dependency error when compiling the luneos branch, as luna-service2 was in luna-sysmgr's depends but not makedepends. I'm currently testing if this PR fixes that issue.

ollieparanoid added some commits Nov 25, 2017

Use postmarketOS package mirror in testcases again
Otherwise Travis starts building gcc-armhf etc, which is not what
we want. I've monkeypatched pmb.build.is_necessary() to always
return True in the testcases for packages, which we do *not* wish
to be downloaded from the binary repo.
@PureTryOut

PureTryOut approved these changes Nov 26, 2017 edited

Rebuilt the entirety of Plasma Mobile using this PR (./pmbootstrap.py build --strict postmarketos-ui-plasma-mobile) and it all ran through succesfully!

@ollieparanoid ollieparanoid merged commit d3c77c3 into master Nov 26, 2017

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+2.5%) to 63.794%
Details

@ollieparanoid ollieparanoid deleted the refactor/pmb-build-package branch Nov 26, 2017

ollieparanoid added a commit that referenced this pull request Nov 27, 2017

Fix #941: Use the right arch for foreign-arch-only packages
This is a follow-up to #935.

* fix regression #941: pmbootstrap doesn't automatically pick the
  right architecture for building when none is specified
* remove obsolete --noarch-arch parameter

ollieparanoid added a commit that referenced this pull request Nov 28, 2017

Fix #941: Use the right arch for foreign-arch-only packages (#943)
This is a follow-up to #935.

* fix regression #941: pmbootstrap doesn't automatically pick the
  right architecture for building when none is specified
* remove obsolete --noarch-arch parameter

PureTryOut added a commit that referenced this pull request Feb 21, 2018

Fix #824: Refactor pmb/build/package.py (make depends work like in ab鈥
鈥ild) (#935)

* Rename pmb/build/package.py to pmb/build/_package.py, so we can
  access the functions it contains in testcases, and still use
  pmb.build.package()
* Refactor the entire file. Instead of one big function that does
  too many things, we have many small ones now, that are tested
  in the testsuite and easier to modify
* Whenever building a package, pmbootstrap does not only build and
  install the "makedepends" (like we did before), now it does the
  same for the "depends". That's required to be compatible with
  abuild. The old behavior can still be used with 'pmbootstrap
  build --ignore-depends'.
* Because of that change, noarch packages can no longer be built in
  the native chroot if we need them for a foreign chroot. A device-
  package depending on a kernel would pull in the same kernel for
  the native architecture otherwise.
* Running 'pmbootstrap build device-...' without '--ignore-depends'
  and without a matching '--arch' displays a note that explains
  this change to the user and tells how to use it instead.
* Noarch packages no longer get symlinked. That was only
  implemented for packages built in the native chroot, and now that
  is not always the case anymore. Symlinking these packages creates
  packages with broken dependencies anyway (e.g.
  device-samsung-i9100 can't be installed in x86_64, because
  linux-samsung-i9100 is armhf only).
* Rename "carch" to "arch" wherever used. Naming it "carch"
  sometimes is confusing with no benefit.
* Add a testcase for the aarch64 qemu workaround (because it failed
  first and I needed to know for sure if it is working again).
* Improved some verbose logging, which helped with development of
  this feature.
* Removed the old "build" test case (which was disabled in
  testcases_fast.sh) as the new "build_package" test case covers its
  functionallity.
* Only build indexes if the packages folder exists for that arch (Travis
  couldn't run a test case otherwise)

PureTryOut added a commit that referenced this pull request Feb 21, 2018

Fix #941: Use the right arch for foreign-arch-only packages (#943)
This is a follow-up to #935.

* fix regression #941: pmbootstrap doesn't automatically pick the
  right architecture for building when none is specified
* remove obsolete --noarch-arch parameter
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.