Skip to content
This repository has been archived by the owner. It is now read-only.

Automatically compute the minimum size for the partition and resize it during the boot process #127

merged 3 commits into from Aug 3, 2017


None yet
3 participants
Copy link

drebrez commented Jul 2, 2017

changes in the pmbootstrap install command:

  • before creating the partition, it computes the size of the files that will be copied in.

Initramfs init script new flow:

  • loading splashscreen is now displayed as soon as possible (right after the hooks)
  • no changes in mounting the boot partition and extracting the initramfs-extra
  • wait_root_partition: waits until either pmOS_root or LUKS partition is found
  • resize_root_partition: if the partition is inside the device-mapper it means that is stored as a subpartitions, in this case, using parted it grows the partition to use all the available space (also remount the subpartition otherwise devmapper doesn't recognize the new size)
  • unlock_root_partition: if the partition is encrypted it starts the usb unlock procedure and waits until the cryptsetup status returns "active". Once unlocked it shows again the loading splashscreen.
  • start_usb_unlock: small change to kill only the telnetd used for unblocking the partition, leaving any other telnetd alive (usb-shell hook), very useful for debugging
  • resize_root_filesystem: performs a filesystem check first (with e2fsck) and use resize2fs to grow the filesystem to use the whole partition size
    (e2fsck and resize2fs requires the /etc/mtab file, see
  • mount_root_partition: self explanatory, it calls find_root_partition again to get the correct one, and if the partition was encrypted, now it correctly finds the decrypted pmOS_root partition inside the devmapper. Added a new splashscreen with "unable to mount root partition" in case of problems mounting it.

Added and files to the static code analysis checks.


This comment has been minimized.

Copy link

ollieparanoid commented Jul 3, 2017

With this commit, we could get much smaller system.img files - and bigger files, if we need them 👍
Two things missing now:

  • A guide on how to resize the system partition via telnet/ssh once it has been flashed to the device
  • Travis CI checks passing (I recommend running test/ locally before making PRs)

Thank you!


This comment has been minimized.

Copy link

ollieparanoid commented Jul 17, 2017

Update from @drebrez in the channel:

for the partition resize, I've managed to do it easily using parted but since we don't have it in the initramfs I was looking to do it with fdisk
and we have to do it in the initramfs because you can not resize it if it's mounted


This comment has been minimized.

Copy link
Member Author

drebrez commented Jul 17, 2017

using parted:

partition with postmarketOS: /dev/block/mmcblk0p3
steps to resize:
=> parted -s /dev/block/mmcblk0p3 resizepart 2 100%
=> kpartx -afs /dev/block/mmcblk0p3
=> cryptsetup luksOpen /dev/mapper/mmcblk0p3p2 root
=> e2fsck -f /dev/mapper/root
=> resize2fs -p /dev/mapper/root
=> mount -w -t ext4 /dev/mapper/root /sysroot

This comment has been minimized.

Copy link
Member Author

drebrez commented Jul 19, 2017

Yesterday I managed to resize the partition programmatically inside the initramfs using fdisk, the next step is to resize the filesystem and to do it inside the initramfs it means to add additional packages because I didn't find any applet in busybox allowing it.
I'm not sure but maybe it's possible to enlarge a live/mounted partition later (in pmos)


This comment has been minimized.

Copy link
Member Author

drebrez commented Jul 20, 2017

Another good solution would be to save all the additional tools (cryptsetup, parted, resize2fs, ...) that we need, inside the boot partition (or a dedicated one) of the pmos system image.
In this way we will reduce the initramfs size and it might help solving the #126.

@ollieparanoid @PabloCastellano what do you think?

Here some sizes based on what we include in

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx"

3'075 KiB

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx /sbin/e2fsck /usr/sbin/resize2fs"

3'771 KiB

BINARIES="/bin/busybox /bin/busybox-extras /usr/sbin/telnetd /sbin/kpartx"

1'567 KiB

@ollieparanoid ollieparanoid referenced this pull request Jul 20, 2017


[titan] Cannot boot kernel compiled with pmOS #126

3 of 9 tasks complete

This comment has been minimized.

Copy link

ollieparanoid commented Jul 20, 2017

I have quoted you and answered in #126 about the initramfs size. Suggestion: do the initramfs split #211 first in a second PR if everyone agrees, and then this PR.


This comment has been minimized.

Copy link

ollieparanoid commented Jul 28, 2017

This does not touch the aports folder yet. Great! So we can merge the aports subfolder PR #227 first, without getting a merge conflict here.

@drebrez drebrez force-pushed the drebrez:minimal_partition_size branch 3 times, most recently from 1eeab56 to dfb22f1 Jul 29, 2017


This comment has been minimized.

Copy link

PabloCastellano commented Aug 1, 2017

Please take into account how #299 (comment) would affect and let's fix it together


This comment has been minimized.

Copy link
Member Author

drebrez commented Aug 1, 2017

@PabloCastellano probably it shouldn't affect this PR, since from what I understand, the generation of the sparse image is done when the whole system.img is already created, right?


This comment has been minimized.

Copy link

ollieparanoid commented Aug 1, 2017

Here is how the two PRs relate:

  1. This PR builds the system image, but with a minimal size.
  2. The img2simg code converts the image (which according to @PabloCastellano's research would also shrink the image to the minimum size, but that will only work for the phones, which are compatible to the simg format at all)
  3. What remains is the question if your system partition has at least 850 MB, which should be removed altogether in my opinion (with this PR, the size is usually much smaller - and all the flashers seem to check the image size before flashing to a partition). I recommend, that we remove it in this PR, because it reduces the system image size for all devices.

@drebrez drebrez force-pushed the drebrez:minimal_partition_size branch from dfb22f1 to 58d170a Aug 2, 2017

@drebrez drebrez changed the title Automatically compute the minimum size for the partitions Automatically compute the minimum size for the partitions and resize it during the boot process Aug 2, 2017

@drebrez drebrez changed the title Automatically compute the minimum size for the partitions and resize it during the boot process Automatically compute the minimum size for the partition and resize it during the boot process Aug 2, 2017

@drebrez drebrez removed the dont_merge_pr label Aug 2, 2017


This comment has been minimized.

Copy link

ollieparanoid commented Aug 2, 2017

Excellent work again, @drebrez! I'll make time to test this PR later today or tomorrow.

Travis isn't quite happy yet - the check for /aports/main/postmarketos-mkinitfs/ is failing. Could you install shellcheck in your local Linux installation and fix up the testing script? (Ideally would be to specify as additional input file for shellcheck with the -x parameter).


This comment has been minimized.

Copy link

PabloCastellano commented Aug 2, 2017

@drebrez, @ollieparanoid: Thanks for clarifying. I was misunderstanding this PR 😃

Copy link

PabloCastellano left a comment

I've done some testing in my titan. Some comments:

  • Tested that both FDE / no FDE work (I can boot, enter the password and ssh into my phone)
  • I would add some more echoes to make debugging easier while looking at the pmOS_init.log
  • Do we really want to resize the partition every time on boot?

My pmOS_init shows some error presumably coming from parted. Does it happen to your as well?

### postmarketOS initramfs ###
mount subpartitions of /dev/mmcblk0p36
WARNING: usb shell is active on
This is a security hole! Only use it for debugging, and
uninstall the usb-shell hook afterwards!
12259 blocks
Error: Partition(s) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64 on /dev/mmcblk0p36 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.

This comment has been minimized.

Copy link
Member Author

drebrez commented Aug 2, 2017

@ollieparanoid I also noticed that shellcheck fails on Travis, and looking here I'm not quite sure what I have to do, should I add the # shellcheck disable=SC1091 directive or add the -x option.
Btw why it fails in Travis but not on my pc? They use different configuration/version?

@PabloCastellano thanks for testing and I agree that we can add more echos to better understand what's happening looking the log file.
The parted error is more like a warning, I can try to remove the partition from the devicemapper first to see if the message goes away.
About the resize every boot, if we want to run it only once, the only solution I see is to write a "marker" file in the pmOS_boot partition and perform it only when the file doesn't exist, @ollieparanoid what you think about it?


This comment has been minimized.

Copy link

ollieparanoid commented Aug 2, 2017

@drebrez: shellcheck gives me the same output, as Travis' version. I guess you have another version installed, I have 0.4.6.
Anyway, I've adjusted the here to use -x properly. I like that you have extended it to check the and!


This comment has been minimized.

Copy link
Member Author

drebrez commented Aug 2, 2017

@ollieparanoid yep, I had an old shellcheck version, now that I upgraded it I have the same error as Travis. Adding the -x option seems the correctly solution.
I think that we should extend the check for all the shell scripts that we have around, but others have warning/suggestions to be fixed before including them in the script.
What's your suggestion about pablo's question to resize the partition every boot or not?


This comment has been minimized.

Copy link

ollieparanoid commented Aug 2, 2017

@drebrez: extending the shellchecking across more scripts is a good idea, but we should do that in a new PR.

Can we check whether we need to resize the partitions? If we could do that, I'd prefer if we only do it when necessary.


This comment has been minimized.

Copy link
Member Author

drebrez commented Aug 2, 2017

@PabloCastellano I tried to avoid the parted warning message about "not able to inform the kernel..." but no success, even if nothing is mounted that message appears...

Resize root partition only if there is unallocated space at the end o…
…f the device.

Added more echos to make debugging easier while looking at the pmOS_init.log.
Updated script to run shellcheck with `-x` option.

This comment has been minimized.

Copy link
Member Author

drebrez commented Aug 3, 2017

@PabloCastellano I updated the PR, added more echos like you suggested and added a check to resize the partition only if there is free space after the partition.


This comment has been minimized.

Copy link

PabloCastellano commented Aug 3, 2017

@drebrez Very good job. I haven't had the time to test it but I prefer it this way 😃

Another possibility that came to my mind yesterday would be using a marker so that we resize when the marker is present (the other way around). This way, pmbootstrap sets it, the partition gets resized the first time and then the marker is removed.

Copy link

ollieparanoid left a comment

I have tested this PR a lot, and it works fine for me!
(All tests have been done with the new initramfs code.)

  1. test: lg-mako, old system image
  • result: telnet does not work (as expected)
  1. test: lg-mako, new system partition with fde
  • resizing works as expected:
localhost:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/root        810.9M    279.7M    490.4M  36% /
  1. test: lg-mako, new system partition without fde:
  • resize works as expected:
localhost:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
                        812.9M    280.2M    491.8M  36% /
  1. test: lg-mako, reboot after previous test
  • booting through works again
  1. test: samsung-i9100 with fde and new system partition
  • booting through is still working as expected

@ollieparanoid ollieparanoid merged commit 147863d into postmarketOS:master Aug 3, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed

@drebrez drebrez deleted the drebrez:minimal_partition_size branch Aug 3, 2017

ollieparanoid added a commit that referenced this pull request Aug 9, 2017

Initial support for the Mozilla Flame device (#335)
* Challenge/buildinfo: slightly improve output

Use "arch" instead of "carch", print "OK!" when a file has been verified
(which is much easier to recognize than "File has the same content").

* [weston] Added compositor-fbdev-Added-parameter-pixman-type patch

It fixes the red screen bug by running weston --pixman-type=2


* pmbootstrap challenge: Put buildinfo.json in changed files list.

...and some smaller fixes:
* make the diff output easier to read
* verify, that only .apk, .buildinfo.json and the APKINDEX have
  changed inside the local repository folder. Because the file
  names of these changes will be used to release files from
  staging to release.

* Move challenge code to own folder (#64)

...and add a stub for 'pmbootstrap challenge APKINDEX.tar.gz'.

* Rewrite pmb.parse.apkindex(): Can get the full index as dict now

Previously, it was only possible to get information about one
package inside the APKINDEX at a time.
This is needed for #64 to verify the APKINDEX.

Please note, that this implementation is actually slower, than
the previous one. But the code is more readable and it makes
caching possible (which will speed up the APKINDEX massively,
especially for the buildinfo.json file generation!)

* pmb.parse.apkindex.parse(): Use caching

Big speed improvement for buildinfo.json files, down from 30-60s to 1-5s.

* Implement challenge for APKINDEX.tar.gz (part of #64)

* gcc-aarch64: remove dependency on bash

This was added by mistake while testing the gcc-aarch64 aport.

* Enabled module installing for the n900 kernel

* Fix #96: Check if build is necessary is broken.

When an APKINDEX contained a package with multiple versions,
pmbootstrap did not use the last version to determine if the
package is out of date (regression from af8c9fc).

* Fixed module loading for n900

* Add testcases for pmbootstrap challenge.

* pmb/challenge/ had to be renamed to pmb/challenge/,
  so the "internal" functions of that file could be accessed, while
  still providing the short notation pmb.challenge.apk().
* zap asks for each buildroot_* chroot, if you want to remove it, not
  only for the one with the device arch
* add new pmb.chroot.tempfolder() function, that creates a temporary
  folder, that belongs to "user" and deletes it, if it already exists.
  this function gets used in a few challenge testcases.

* Close #94: Always rebuild apks when aports change

...even if the pkgver and pkgrel have *not* changed. This should
make development much more intuitive. The detection works by looking
at the last modified timestamps, just like `make` does it.

* Fix race condition in testcase test_challenge_build

* Fix commandline help wording

* Add support for the binary repository, inactive by default (#64)

* New commandline parameter --mirror-pmOS, where the binary repository
  URL for postmarketOS can be specified (empty by default as of now,
  this will be filled with the real URL once the repo works)
* Do not build packages, when they are in the binary repository and
  the version of the package in the binary repository is up-to-date.
* Add a testcase for

* Slightly adjust
* Change "GNU/Linux" into "Linux distribution", because the Alpine Linux developers don't see their distro as GNU/Linux
* Use "hello-world" as example package, as it does not make much sense right now to compile heimdall for armhf

* Update musl-* to 1.1.16-r13

* Added --no-fde parameter

* Use variable in and add labels to partitions

* Find root partition using labels

For some reason /dev/disk/by-label/pmOS_root symlink is not present and
blkid from busybox doesn't support the -L parameter. We could install
blkid from util-linux but I prefered to use shell commands instead

* Mount unencrypted root partition (close #86)

- Do not decrypt if not necessary
- Do not launch telnetd if partition is unencrypted
- Basic initramfs logging to /tmp/boot.log

* Update min apk version/add more apk version checks

* Minimum version: 2.7.2 (which fixes two CVEs)
* Check the minimum apk version before doing something with apk and
  before entering the chroot manually (previously, it has just checked
  the apk-tools-static version, which gets used to set up the chroot)
* Reword the message for an outdated APK version. Most likely it is
  just the outdated http cache, instead of a man-in-the-middle attack.

See also:

* Close #103: Document how to use chroot commands with parameters

* Update musl-* to 1.1.16-r14

* Support relative path for initializing config

Previously, if you passed something like ~/build to the init function
for work directory, it would create a directory, .\~/ in the current
working directory instead of resolving ~/ to the user's home directory.
This allow allows using ../ to specify a path.

* [postmarketos-base][hammerhead] Use weston --pixman-type

Add a new optional variable weston_pixman_type in deviceinfo
This is used to fix the red screen issue (close #54)

* aportgen: Added end of file newline

* Added extract-dtb tool

* gitignore: Added vim temporary files

* Slightly improve error handling

* Fix: Do not swallow traces when crashing before log init (e.g.
  during argument parsing)
* Show a link to the troubleshooting page, when an error happens
* (Formatting done by autopep8 in pmb/config/

* Prepare for aarch64 host support, fix typo in arch mapping

Relates to #106.

* pmb.chroot.apk.installed(): use apk's internal database

...instead of running apk every time to get the list of installed
packages and their versions. The internal package database from
apk has the same format, as the extracted APKINDEX file (except
that it has more key-value pairs, which we ignore/do not need
right now). So the APKINDEX code has been extended to parse both
tar-packed APKINDEX files and regular text files in the APKINDEX

This is required for #108, for a better detection of outdated
packages (because the internal package database  saves the
package's timestamp, too). A nice benefit is, that this is faster
than calling apk every time and it doesn't fill up the log as much.

I've also used this improved function for determining the apk
version (for the outdated version check), and I've deleted
pmb.parse.other.package_split(), as it is not needed anymore.

* Add new device lg-d855

* Rename args.no_fde to args.full_disk_encryption

* postmarketos-mkinitfs: Major refactoring

* rewrote find_root_partition to make it compatible with the i9100
  again (and added a big description comment about how it works)
* usb network and dhcp server gets started independent of hooks
  now. this really makes sense for development now, we can disable
  this later (and set up networking via OpenRC, when the usb-shell
  hook is not installed)
* telnet splash + unlock telnetd code only runs, when the rootfs
  is encrypted
* usb_setup_android wrapped with a generic usb_setup function, so
  we can support networking for more device types in a clean way
* the IP of the device is now in, and
  gets included in all hooks.
* rewrote logging function. it logs all stdout and stderr to
  /pmOS_init.log, except when PMOS_NO_OUTPUT_REDIRECT is specified
  as kernel command line. This way we can see error output from
  all commands.
* used shfmt to format everything, so it looks consistent again
* added automatic shellchecking for and
* clear naming scheme for initfs functions, they all start with a
  verb now (e.g. setup_usb_network, start_udhcpd, ...)

* Fix /proc/cmdline grepping: /proc must be mounted first

* Adjust defconfig path

* Porting to Moto G 2014 "titan"

- Use samsung-i9100 as template

* APKBUILD + deviceinfo (work in progress)



$ unpackbootimg -i boot.img
Android magic found at: 0
BOARD_KERNEL_CMDLINE androidboot.bootdevice=msm_sdcc.1 androidboot.hardware=qcom vmalloc=400M utags.blkdev=/dev/block/platform/msm_sdcc.1/by-name/utags buildvariant=userdebug

* Use LineageOS kernel

Copied defconfig from

* Kernel compiles!

- Removed -Werror
- Use linux-lg-mako as base
- Split big .diff in .patch files so that they are automatically applied

- Applied patches to kernel
  Previously it was stuck showing this error:


  Other interesting changes to merge:

* After menuconfig and accept all default values

* Flash time!

* Removed some unnecessary changes from gcc5 big patch


* Use dtb image. It finally boots!


* udev rule to use touchscreen + calibration

* deviceinfo_generate_bootimg

* Added and initfs-hook hacks

Weston works! (and has red screen bug)

Note: I have to manually ssh and
sh /etc/postmarketos-mkinitfs/hooks/

* Added deviceinfo_weston_pixman_type variable to fix red screen bug

* Changes done after code review

* Changed broadcom config for hammerhead

* Added bcmdhd calibration from google git

* Reverted accidental menuconfig change

* Added dependency on linux-firmware

* Bumped version on hammerhead packages

* Change initialization order, execute hooks before the usb network setup

* Generate boot.img files in mkinitfs

* Only, when deviceinfo_generate_bootimg="true" in the deviceinfo
* It respects a new deviceinfo_kernel_cmdline parameter from the deviceinfo

* Activate boot.img generation in all fastboot-compatible devices

* Device samsung i9070 (#123)

* Add Samsung Galaxy S Advance (i9070) device-specific package

* Add Samsung Galaxy S Advance (i9070) kernel package

* Add initfs-hook script to set framebuffer settings

* Enable adb root access in recovery mode (TWRP)

* Add usb configuration in initfs-hook script
Add weston calibration values
Update description of kernel changes

* Remove TMRP recovery and change init script to boot directly postmarketOS

* Device sony amami (#100)

* Add device: sony-amami
* Add kernel package for sony-amami

* Show path to boot.img and description of commands after pmbootstrap install

This change got lost in this PR merge:

* [titan] Removed duplicate deviceinfo_generate_bootimg in deviceinfo

* i9100: remove recovery/download mode instructions txt file

This info belongs in the wiki instead.

* pmbootstrap flasher: use boot.img file with fastboot

* Close #77: Implement 'pmbootstrap flasher export'

* Fix #128: Remove obsolete 'deviceinfo_flash_offset_base'

* Fixed typo in new boot.img flasher

* Added flavor to the flash_system command for consistency, This fixes issue #133

* Changed flavor to an optional argument

* Add 'pmbootstrap aportgen busybox-static-armhf' (and other archs)

* Generate aports for busybox-static-armhf and -aarch64

...and add generating both aports to the testsuite.

* Get rid of the _pmb_build_in_native_chroot hack

* Display device count in 'pmbootstrap init'

* Close #92: pmOS-specific /etc/motd, /etc/issue

* deviceinfo: Add link to reference and reword comments

* Display short troubleshooting URL on exception

* Fix #107: Meaningful error for grsecurity/chroot_deny_chmod

* [i9070] Use kernel modules / Wifi support (!)

* Add kernel module compilation
* Add wifi firmware from vendor files (to `/lib/firmware/postmarketos`) and fix modules install path
* Add patch to change hardcoded dhd firmware/nvram paths
* Add wifi module (dhd) to the list of kernel modules that should be loaded at boot time


Thanks you very much, drebrez \o/

* Fix #139: Update musl-armhf to 1.1.16-r15

* Update busybox-static-* to 1.27.0-r1

* Close #137: Remove "deviceinfo_modules" / minor deviceinfo changes

Minor changes:
* device-lg-hammerhead: remove left-over comment
* device-samsung-i9100: remove fastboot section

* This fixes the following error:
mkbootimg: error: argument --base: invalid parse_int value: ''

So now postmarketos-mkinitfs can really generate boot.img files.

* [mako] Use red screen workaround

* Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129)

TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution.
pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has.

I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository.

To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on.

In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option.

In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds".

This commit also fixes the working_dir argument in, which was simply ignored before.

Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases).

The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.

* deviceinfo: add heimdall partitions / make flasher variables optional

* Fix race condition in test_challenge_build (APKINDEX timestamp)

* Challenge APKINDEX.tar.gz: add logging

* Challenge APKINDEX.tar.gz: allow files in the repository

* Fix #107: Grsec check: read file in /proc as root

* Properly update /etc/apk/repositories when the mirrors change

This is important for the binary repository scripts, so it's feasible
to test the binary package build and challenge process locally without
setting up a new chroot whenever changing the repo URLs.

Also it behaves a bit more intuitively, because it really uses the
repo URL specified on the commandline, even when the chroot is already
set up.

* Depends parsing: Properly handle empty depends / buildinfo: strict depends

* Properly support specifying a local folder as --mirror-pmOS

This is required for developing and testing the binary repository
scripts (see #64). Changes:
* When specified, the local folder gets mounted inside the chroots
  as /mnt/postmarketos-mirror
* The apkindex_files() function outputs the correct path to the local
  repository (it does *not* hash the URL in that case, which would
  be wrong)
* /etc/apk/repositories: when the pmOS mirror is a local folder,
  the path "/mnt/postmarketos-mirror" gets added to that file instead
  of the outside path (so apk finds it properly inside the chroot)

* Add new dependency test package: hello-world-wrapper

* Add ConfigFS based usb network configuration

This extends the initramfs init script to setup USB networking on kernels

* AutoPEP8 1.3.2 slightly changed automatic formatting

* Fix #194: ismount() fails on symlinks

* Update gcc-* to 6.4.0

* postmarketos-mkinitfs: Bump pkgver to 0.1.5

* Challenge APK files: less output by default

* Update heimdall script to flash both (kernel+initramfs) at once

* RX51 - Add touchscreen calibration udev rule

* [RX51] Add linux-firmware package

This removes the call to `firmware_install` in the APKGBUILD for
linux-nokia-rx51, and adds the `linux-firmware` package as a dependency
for device-nokia-rx51.

There are two immediate benefits for this:
1) wifi now works

2) the FM radio device was causing the boot to take ~2 minutes longer
than it should, now that the firmware is present there's no delay in
initializing this device

* Account for uInitrd filename change in u-boot script (rx-51)

The uInitrd filename is appended with 'flavor' so, for example, with the
rx-51 the generated file is named 'uInitrd-nokia-rx51'. This corrects
the rx51's u-boot script to account for this change in filename.

* [RX51] Update to mainline kernel

A couple of issues are resolved by moving to the mainline kernel (from Pali's

1) 'reboot' command now works as intended
2) boot is *much* faster
3) charging LED seems to be working now

* Reduce complexity in (#160)

Thanks, @cclauss!

* Dictionary lookup of event loop actions

This was kind of an experiment that adds lots of blank lines to this script -- feel free to reject it.

The observation that triggered this was that in main() all actions can execute using just one parameter, args.  This means that it is possible to use args.action to do a dict lookup to get the corresponding function.  This approach adds tons of blank lines to the script in exchange for a rapid seek to the action implementation and a substantial reduction of the McCabe complexity of main().

$ __flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics__
./ C901 'main' is too complex (23)
./pmb/aportgen/ C901 'rewrite' is too complex (17)
[ ... ]

* Remove action_dict and use locals() instead

* Fix PR #160, Fix #172 (#173)

Thanks craftyguy!
Probably meant to call 'globals()' and not 'locals()'. As @MartijnBraam
pointed out in IRC, probably best to have these functions in a separate
module so they can be parsed with getattr

* [RX51] Export u-boot kernel image with flasher

This properly exports the uImage kernel, which is used by u-boot, when
running 'flasher export'. Note the change to the uImage name to follow
the pattern "uImage-$flavor", which in the case of the RX 51 is

* Add acpid package (#170)

Thanks craftyguy!

This commit adds the acpid daemon. This requires a kernel with NETLINK
support compiled in before daemon will run. acpid will be extremely
helpful for devices that have external buttons which need event
handling. The nokia-rx51 is full of such devices/switched!

This was heavily leveraged from the Arch Linux PKGBUILD, thank you Arch

* [weston] Added more patches

- Add support for ABGR (Asus grouper / Nexus 7 2012 tablet)
- Print guessed pixman type

* Close #174: Put frontend functions in extra file (#185)

* [RX51] Enable DRM in kernel

This enables DRM (Direct Rendering Manager) support in the kernel for
the RX51. This opens up the possibility of using the Weston DRM backend,
which in turn allows for DPMS (I hope) and other cool tricks.

n900:~# glxinfo|head
name of display: :0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4

n900:~# glxgears
94 frames in 5.0 seconds = 18.772 FPS

* Two trivial changes (typo + info message) (#179)

Thanks, Pablo Castellano!

* Fixed typo

* Guide user about what are the next steps after flashing kernel and password

* Increased postmarketos-mkinitfs' pkgver and rewording

Changes requested:

* Fix #200: Parse depends_dev in APKBUILDs

* Fix #145: add deviceinfo_flash_offset_base again (#175)

* Fix #145: add deviceinfo_flash_offset_base again

* Add default OFFSET_BASE value in flasher if deviceinfo_flash_offset_base variable is not set
...and set correct default base value in create_bootimg function

* Adjust gcc-armhf/gcc-aarch64 to upstream

* [RX51] Add support for physical events w/ acpid

This adds a way to handle physical events like keypad slide switch,
screen lock, camera lid, and others. This uses acpid from busybox (I was
wrong about the real acpid package), and anl file to map events
to 'scripts'. The scripts are symlinked to /etc/acpi/,
automatically by the new post-install script, where
they are handled based on which script was called. This allows for
easily adding whatever is necessary for #178.

If this PR is merged, I will create a new wiki page detailing how others
can use this framework on other devices.

There's an issue with Busybox's acpid, where it is unable to detect new
devices added after the daemon loads. This is the case on the N900,
where openrc starts acpid well before the kernel is done modprobing
drivers (e.g. gpio_keys), so it's necessary to restart the acpid daemon
after some time to 'load' the new devices.

* Add asus-grouper (#180)

* Add device HTC Desire (Bravo) support (#202)

* Add HTC Desire (Bravo) device-specific and kernel package

* Enable kernel virtual terminal support

* Add framebuffer and usb configuration in initfs-hook script

* Add udev required kernel options and set correct framebuffer resolution

* Add touchscreen device rules and enabled kernel logs (dmesg)

* Add deviceinfo_flash_offset_base in deviceinfo and update other offsets

* Fix #209: Cross-compiling fails with C++ code

* Various distccd related improvements, mostly respect --verbose (#216)

I've done some refactoring while debugging #209.
* Unused file `pmb/build/` removed (that was a
  left over from `_pmb_build_in_native_chroot` hack
* Do verbose logging in distccd, when `pmbootstrap --verbose` is
  being invoked
* Restart distccd, when the commandline has changed (e.g. when the
  currently running version was not verbose, and the new one is
  verbose.) Prior to this change, it only got restarted, when the
  architecture changed (so it did not allow changing the job count
  on the fly for example).
* Insert missing whitespace in arguments help.

* Fix #162: flash_system: meaningful error for missing image

* Update busybox-static-* to upstream

* Close #215: Write explicity what passwords are for in install

* Handle generating weston.ini in post-install script (#201)

This PR automatically creates the weston.ini in the
postmarketos-base post-install script based on options set in the
deviceinfo file for a particular device. This replaces #191.
If weston.ini exists, then no modifications are made.

(Optional) options supported in this PR for deviceinfo are:
deviceinfo_weston_core_modules - defaults to NONE if none specified
deviceinfo_weston_core_backend - defaults to '' if none specified
deviceinfo_weston_keymap_rules - keymap rule, found under /usr/share/X11/xkb/rules/
deviceinfo_weston_keymap_model - keymap model

* Re-enable XWayland for devices that had it enabled (#221)

This re-enables the XWayland module for devices which did not use the
'dirty hack' that was removed in #201.

* [RX51] Set keyboard layout for weston

This configures the keymap in weston so that the rx51 keyboard behaves
as expected (e.g. shift key, numbers, etc)

* pmbootstrap shutdown: Make cryptsetup closing slightly more robust

I've had a strange case where `cryptsetup status` did not work
inside the chroot anymore, and only zapping it would resolve it.
But I couldn't zap, because the status check was preventing shutdown
(on which zap depends) from working. This commit works around that.

* [RX51] Update to Linux-4.12.3

* Fix #219: pmbootstrap could not find g++-armhf/g++-aarch64

* Fix PEP8 syntax

Spaces around a plus sign.

* Close #195: Ignore parameters in dependencies (#225)

* Ignore `>`, `<`, `=` and `!` operators, when they are specified in
  the dependencies. This was the desired behavior before, but it was
  not implemented correctly (so it wouldn't ignore them everywhere).
  Of course the real fix would be to honor these operators like apk
  does. But this isn't feasible right now, and it should work for
  most, if not all, our use-cases. I have documented this in the wiki
  under build internals and if we happen to need correct operator
  handling, we should do it then.

Minor other changes:
* `pmbootstrap parse_apkindex`: support optional package parameter to
  only show the parsed content for one package.
* Support building most python APKBUILDs by replacing ${pkgname#py-}

* Fix #219 for real!

100% tested. Assuming that you don't need to test an obvious change,
because it only changes one line is dangerous. I will learn from this,
sorry for the inconvenience.

`gcc-armhf` compiles fine again with that change, and it is able to
cross-compile packages as it used to.

* Improve the FDE hint, you need to type in the password twice.

* Close #198: Add possibility to override the kernel commandline during boot (#217)

Usage example:
pmbootstrap flasher boot --cmdline "PMOS_NO_OUTPUT_REDIRECT"

Thanks, drebrez!

* Fix #230: pmbootstrap flasher flash_system is broken

Regression from the cmdline argument feature.

* Created splash screen generator (#206)

Nice configurable splash screen generator, thanks Martijn Braam!

* Ignore directories while searching for loop devices (#235)

Thanks, cmdr2! Here's the text from his PR:

Fixes an issue on alpine-vanilla, where `/dev/loop` is a directory (containing `/dev/loop/0`, `/dev/loop/1` ..etc), and causes `./pmbootstrap install` to fail.

Ignoring the `/dev/loop` directory works because `/dev/` also contains files named `/dev/loop0`, `/dev/loop1` ..etc, which the new code will still pick up.

The `install` command passes after this fix on my alpine-vanilla.

* Update runtime error message thrown for grsec/hardened kernels (#236)

* Update runtime error message thrown for grsec/hardened kernels, saying that they are not supported right now, #107

* Remove unnecessary check for grsec enable/disable status - the build will not work in either case

* New splash images in mkinitfs (#240)

Thanks, Martijn Braam!

* Made postmarketos-mkinitfs use postmarketos-splash to generate splash images
* Added screen sizes to all deviceinfo files
* Bumped pkgrel for all device packages

* Do not store aports directory in config file (#247)

This addresses #246 by not storing the aports directory in the config
file. The default location is still available (from config/,
and can be overridden.

* De-couple weston from postmarketos-base (#233)

Thanks, craftyguy!

- UI is selectable with `pmbootstrap init`. Currently only 'weston' and 'none' are options. It'll automatically pick up any new `postmarketos-ui-*` packages added at later dates.
- splits off weston packages install from postmarketos-base and puts
them into postmarketos-ui-weston. Also note that NO weston packages are
installed by "pmbootstrap install" by default unless the user selects a ui in the `init`
- configuration of weston is now in postmarketos-ui-weston.
- the demos have been spun off to `postmarketos-demos`, and `postmarketos-ui-weston` lists this package as a dependency.

* Close #212: Split heimdall in heimdall-isorec and heimdall-bootimg (#254)

* Unified linux-postmarketos kernel for qemu and n900 (#228)

The creates the linux-postmarketos package which for now supports the n900 (and probably the other maemo devices) and qemu with the vexpress-a9 machine simulation.

I've put the generated dtbs in /usr/share/dtb for now and set the dtb field in the deviceinfo for the n900 and my new qemu device.

* Introduced linux-postmarketos-omap
* Unified kernel progress
* Created kernel bootable with qemu
* Updated n900 deviceinfo for generic kernel
* Changed qemu device to vexpress
* Updated APK comments and added linux-postmarketos-dev package
* Append dtb in mkinitfs
* Fixed bootscript on n900 for the generic kernel
* Don't detect double flavors with -dtb appending
* Added graphics drivers for vexpress (qemu)
* Added more drivers for qemu

This adds virtio network support in qemu for the vexpress-a9 machine.
The keyboard and mouse don't work yet.
You can boot into weston if --no-fde is specified

* Enable syslog on boot

This enables the busybox syslog implementation, which logs to
/var/log/messages, on boot. This is very helpful for debugging
applications which write to syslog (e.g. NetworkManager)

syslog is being set up to log to shared memory (not to disk), so log is
lost 1) on reboot and 2) if the service is restarted. It can be read
with `logread`

* Fix #242: Validate input in pmb.helpers.cli

* Validate all inputs from `pmbootstrap init`
* Add a new `confirm()` function, that validates input of yes/no
  questions properly
* All questions loop until they have a valid answer now

* Added more drivers to the x86_64 kernel (#272)

This adds support for ide, sata, mice, touchscreens,
displays and serial ports in qemu.

* Low hanging fruit basket (Close #220, Close #239) (#253)

* postmarketos-splash: change arch from "all" to "noarch"
* lg-mako: use .tar.gz instead of .zip (because that's the reference
* #220: Allow specifying multiple packages for checksum, build,
* #239: Add chroot shortcuts (--rootfs and --buildroot)
* Show chroot and command before entering chroot

* Fix #260, fix #87: Don't cross-compile when not necessary (#265)

* Add dtbtool package

* Actually use the PROGNAME variable and change filename to camelCase as in upstream

* Minor spelling fixes

* Various improvements for 'pmbootstrap export' (Fix #232) (#250)

* Various improvements for 'pmbootstrap export' (Fix #232)
* Do not try to generate the system image, but print a notice that
  it was not generated yet
* Display export path
* New default path: /tmp/postmarketOS-export (instead of current
  working dir)
* Create the export folder, if it does not exist yet
* Prettier output in the export message
* Adjust export message in install for consistency

* pmbootstrap install: Add link to usb-hook (#268)

* Fix #166: pmbootstrap shutdown: umount deep folder levels first (#274)

* Refactored `umount_all` to get the list of folders to be umounted from
  `umount_all_list`, so we can test that function in a test case.
* Adjusted `umount_all_list` to return the deep folder levels first
* Added a testcase for that
* Remove redundant calls to `umount_all()` (which were from a time before
  `pmbootstrap` was released, in which failing commands did not cause
  `pmbootstrap` to raise an exception)

* [RX51] Enable swap device and mount /boot by default

This enables the existing swap device (set up by Maemo) and sets up
/boot to mount on startup

* Fix #226: Building busybox recurses forever. (#252)

If you really want to build busybox, I recommend turning the
"timestamp based rebuilds" feature off, otherwise it will build
for all architectures all the time whenever you change something,
even if you do not increase the version number (that's the idea
of that feature). This is, because busybox is a dependency for
basiscally everything, so it must get updated whenever you install
something, in case it was out of date.

It is easier to simply rename the package.

* Fix #269: Always use truncate to create empty image file (#276)

I've realized that truncate is provided by busybox anyway, so no
need to install coreutils for this.

This should fix creating system image files on ZFS, where fallocate
is not implemented.

* Split up initramfs to make it smaller (Close #211) (#257)

Huge thank you to @drebrez for his amazing work on this PR!

* Add generation of initramfs-extra with additional binaries
Extract both initramfs with `pmbootstrap initfs extract`
Add new splashscreens for missing partitions/files errors
Changes in init script:
 - use busybox findfs applet to find boot partition
 - mount boot partion
 - extract initramfs-extra
 - show error splashscreens accordingly and loop forever
 - start usb unlock directly from unlock_root_partition (hook removed)

* Print out a text message for serial debugging in case of errors
Add initramfs-extra files to `pmbootstrap initfs ls` output

* Fix trailing whitespace in comment...

* ls: Indicate which initramfs we're looking at / add wiki link

I've rewritten the initramfs-development article to reflect the
changes made in this PR. It will be a good read for someone who
extracted the initramfs and wants to know why we have two files.

* Add device htc-ace to master (#281)

@VictorNine did the port, all credit goes to him (and @drebrez and everyone else who helped him)!

* Added device HTC Ace (#181)

* Add HTC Desire HD (Ace) device-specific and kernel package

* Enable kernel virtual terminal support

* Added touchscreen device rules

* Added linux-firmware pkg. WIFI is now working

* device-htc-ace: Add deviceinfo_flash_offset_base

* Fix #282: '/bin/ash': Permission denied

This was a regression from the improved input validation PR, which
created the chroot_native folder with the wrong permissions (to test
if we have write access to the work folder).

sjamaan found out, that this was the cause of the issue, and also
explained how to fix it, thanks a lot!

* Close #194: Aports subfolders! See migration guide in the wiki (#227)

Migration guide:

* Bump pmbootstrap version to 0.2.0

* Move linux-postmarketos to from device to main

* Move two cross-packages to the cross folder

* Usability improvements, fix #278 (#279)

* Quote architecture in logging message for easier reading
* Added shortcut arguments for --rootfs, --buildroot and --suffix
* Simply remove beforehand link to nowhere if exists (fix #278)
Fixed crash when symlink already existed but pointed to a non-existing location

* Update to Linux Kernel 4.12.4 (#285)

This updates the linux-postmarketos kernel to 4.12.4, and fixes an issue
where the APKBUILD was pulling down the 4.12 source but not the 4.12.x
patches. I think that pulling down the major.minor source and a sep.
package for patches to update to the major.minor.minor(er?) release adds
unnecessary complexity since already hosts the sources with
these patches applied (e.g. in this case the 4.12.4 source)

* pmb.helpers.ui scan for UIs in new aport structure (#290)

This fixes #289.

* Move wpa_supplicant to default runlevel (#286)

Having it in boot doesn't actually work, the service is being started
too early and fails. This change moves it to the default runlevel. This
change also removes "wifi-handler" since that service doesn't actually

* Fix #288: Checksum *-install scripts in APKBUILDs (#294)

Per the APKBUILD reference, , package *-install scripts should be checksummed to guard against incomplete/corrupt downloads. The postmarketos-base package is one example where this is not being implemented currently.

* Fix checksum of postmarketos-base

* Allow 'pmbootstrap menuconfig' without specifying "linux-" (#295)

We can allow both ways:
* pmbootstrap menuconfig linux-motorola-titan
* pmbootstrap menuconfig motorola-titan

The former will show a tip about the second
This also prevents users from running menuconfig on aports like
'device-motorola-titan' or 'mkbootimg'

* Implemented checksum test in the travis script (#298)

* Implemented checksum test in the travis script
* Added workaround for squashed forced pushes
* Made test possible to run locally
* Match multiple git versions for the working directory message

* Add possibility to export Odin flashable tar (#297)

Add possibility to export Odin flashable tar for devices where the
flasher method is heimdall-isorec or heimdall-bootimg

* Fix test script not crashing hard enough in travis (#301)

Made exit 1 on errors

* Fix #300: Properly unset environment in chroot (#302)

* Print "git diff" output when the checksums tests has failed (#306)

In some cases this output is useful to identify the root of the problem.
See #303 (comment)

* Fixed dtb installation with mainline kernel (#307)

* Fixed dtb installation with mainline kernel

* Travis checksum pr fix (#308)

* Check the correct branch in PR runs

* Add device Samsung Galaxy Note 2 (N7100) (#293)

* [RX51] workaround lines issue (#309)

This works around the lines issue using a patch from Ti:

* init: Allow to specify extra packages that will be always installed to rootfs (#304)

* Fix #223: Use `.tar.gz` instead of `.zip` for kernel sources (#315)

* Update musl-* to 1.1.16-r17 (#320)

* Implement sparse system images (fix #299) (#303)

* Packaged libsparse

libsparse from the Android project provides multiple tools like img2simg
and simg2img.
These are used to split a large image for the system partition into
separate smaller chunks with sparse headers

This is required for several devices (at least bullhead, fp2 and titan)
because it fixes the "Invalid sparse file format at header magi" error


* Added new variable deviceinfo_flash_sparse (fixes #299)

Right after the system image is generated, pmbootstrap checks this
variable. In case it is true, run img2simg on it

* motorola-titan: enable deviceinfo_flash_sparse

* libsparse: use source from github: anestisb/android-simg2img

It is not that easy to use the upstream archive because everytime
you download it, the files have the current date as creation date
and that makes the file have a different checksum every download
#303 (comment)

* [linux-postmarketos] enable maint. patching of kernel source (close #311) (#313)

This enables applying maint. release patchsets to the kernel source
tarball, in line with Alpine Linux's linux-vanilla APKBUILD.
This addresses #311.

* Close #314: Add --details-to-stdout parameter (#317)

When this parameter is set, `args.logfd` (the log file descriptor)
gets set to `sys.stdout`, so all logging gets directly sent to the
active pmbootstrap session, and not to a log file.

This will be used for Travis jobs, because they get killed after
10 minutes without any output (and builds may take longer).

* Update busybox-static-* to upstream

* Automatically compute the minimum size for the partition and resize it during the boot process (#127)

* Automatically compute the minimum size for the partitions
* Automatically resize the pmOS_root partition during the boot process
* Resize root partition only if there is unallocated space at the end of the device.
* Added more echos to make debugging easier while looking at the pmOS_init.log.
* Updated script to run shellcheck with `-x` option.

* Fix #319: pmbootstrap menuconfig didn't work for mainline kernel (#328)

Also did some minor refactoring.

* Package Plasma Mobile (All credit to PureTryOut!) (#284)

Plasma mobile compiles, but we didn't get it to run yet.
Thanks to PureTryOut for all this amazing porting work! \o/

* Update busybox-static-* to upstream

* Fixed checksum on travis for removed packages (#332)

* Add check for removed packages

* Change for testing travis

* Fixed package depth

* Removed consolekit2 (upstreamed) / other plasma fixes (#331)

* Removed consolekit2 from the repo as it's now upstreamed

* Add missing makedepend to kwin

* Fix: gcc-armhf not reproducible (#64) (#333)

libstdc++.a from gcc-armhf was not reproducible on Travis (it was, when built locally!). These .a files are just archives of object files .o, and in this case it was caused by a random order of the .o files in the archive.

This PR patches the package() function of the APKBUILD when running pmbootstrap aportgen gcc-armhf (same for aarch64 of course), to extract all .a files, and repack them to be reproducible (by sorting the files before packing them).

As usually, we can still inherit everything from the upstream gcc aport from Alpine, and apply our changes on top of that.

Travis without the patch:

> CHALLENGE FAILED for usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a:File 'usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a' is different!

Travis with the patch (I've instructed Travis to run off this branch to test it):

> Done. Your build exited with 0.

* Minor fixes (#316)

* Fixed typo and specify list order

* Simplify code for jobs default value

* Fix 'install: can't stat' message, when no hook is installed (#310)

* Add `-c, --clear` argument for `log` and `log_distccd` actions to clear log files (#344)

* Fix weston on sony-amami (#340)

* Close #334: Check if pkgname is the aport folder name (#347)

...and remove an redundant sentence with broken grammar in another
error message.

* Fix #349: Verify pkgname against folder name after replacing vars

This fixes a regression introduced in 7fe2bc1.
Sorry for the inconvenience!

* Fix refresh rate on sony-amami (#348)

Patch added to hardcode pixclock in framebuffer driver

* Initial Mozilla Flame packages

* Update dtbtool to use LineageOS upstream

* Use mkbootimg --dt flag only when deviceinfo_bootimg_qcdt is defined

* Update mkbootimg to use LineageOS upstream

* Add sparse image flashing on device-mozilla-flame

* Only support SD card install on Mozilla Flame

* Remove useless on mozilla flame

* Package DTBs in /usr/lib/linux-mozilla-flame/

* Build and install modules in linux-mozilla-flame

craftyguy added a commit that referenced this pull request Aug 13, 2017

Automatically compute the minimum size for the partition and resize i…
…t during the boot process (#127)

* Automatically compute the minimum size for the partitions
* Automatically resize the pmOS_root partition during the boot process
* Resize root partition only if there is unallocated space at the end of the device.
* Added more echos to make debugging easier while looking at the pmOS_init.log.
* Updated script to run shellcheck with `-x` option.

PureTryOut added a commit that referenced this pull request Feb 21, 2018

Automatically compute the minimum size for the partition and resize i…
…t during the boot process (#127)

* Automatically compute the minimum size for the partitions
* Automatically resize the pmOS_root partition during the boot process
* Resize root partition only if there is unallocated space at the end of the device.
* Added more echos to make debugging easier while looking at the pmOS_init.log.
* Updated script to run shellcheck with `-x` option.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.