This repository has been archived by the owner. It is now read-only.

Unified linux-postmarketos kernel for qemu and n900 #228

Merged
merged 19 commits into from Jul 25, 2017

Conversation

Projects
None yet
3 participants
@MartijnBraam
Member

MartijnBraam commented Jul 22, 2017

The creates the linux-postmarketos package which for now supports the n900 (and probably the other maemo devices) and qemu with the vexpress-a9 machine simulation.

I've put the generated dtbs in /usr/share/dtb for now and set the dtb field in the deviceinfo for the n900 and my new qemu device.

ToDo

  • Append the dtb to the kernel somewhere in the buildsystem (instead of in the kernel package)
  • Integrate launching qemu somehow
  • qemu graphics support
  • qemu storage support
  • qemu keyboard support
  • qemu networking support
  • qemu mouse support
  • Rebase this stuff for the newer device packages in the master branch
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 22, 2017

Member

I am glad, that you are working on this!

However, I think we need to discuss the handling of the board architectures a bit more. The two of us agreed on having multiple builds for different boards is a good idea (dear reader, if you think otherwise, please let us know).

So with the current PR you're just ignoring all other boards, but that won't scale. I have two approaches in mind:

  1. Have a single package, and generate the other kernel binaries as subpackages (so as output we'd have linux-postmarketos-omap3, linux-postmarketos-otherboard, ...).
  • Advantages:
    • Just one package to maintain, that builds all versions
  • Disadvantages:
    • Building that package will take forever once we have a reasonable amount of boards. (This will probably become problematic for #64 , which will utilize Travis to rebuild and verify these packages - and Travis has a maximum runtime of 50 minutes.!)
    • When compiling/packaging one board fails, you must rebuild and repackage all of them, which will take lots of time!
  1. Have one package per board.
  • Advantages:
    • shorter compile/package time (see above)
  • Disadvantages:
    • a lot more maintenance effort to keep in sync

Another idea I have is writing some Python code, so we can have pmbootstrap aportgen linux-postmarketos-omap3. Then we could do the second approach easily, and it would scale - we would only need to maintain the unified aportgen code for all boards and generate the rest from there.

So my proposal would be 2. + aportgen, what do you think, everyone?

Member

ollieparanoid commented Jul 22, 2017

I am glad, that you are working on this!

However, I think we need to discuss the handling of the board architectures a bit more. The two of us agreed on having multiple builds for different boards is a good idea (dear reader, if you think otherwise, please let us know).

So with the current PR you're just ignoring all other boards, but that won't scale. I have two approaches in mind:

  1. Have a single package, and generate the other kernel binaries as subpackages (so as output we'd have linux-postmarketos-omap3, linux-postmarketos-otherboard, ...).
  • Advantages:
    • Just one package to maintain, that builds all versions
  • Disadvantages:
    • Building that package will take forever once we have a reasonable amount of boards. (This will probably become problematic for #64 , which will utilize Travis to rebuild and verify these packages - and Travis has a maximum runtime of 50 minutes.!)
    • When compiling/packaging one board fails, you must rebuild and repackage all of them, which will take lots of time!
  1. Have one package per board.
  • Advantages:
    • shorter compile/package time (see above)
  • Disadvantages:
    • a lot more maintenance effort to keep in sync

Another idea I have is writing some Python code, so we can have pmbootstrap aportgen linux-postmarketos-omap3. Then we could do the second approach easily, and it would scale - we would only need to maintain the unified aportgen code for all boards and generate the rest from there.

So my proposal would be 2. + aportgen, what do you think, everyone?

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 22, 2017

Member

My current plan was ignoring the size of the kernel until it becomes a real problem since not a lot of devices have mainline kernel support yet. The devices that have very tiny boot partitions aren't devices that have mainline support at all. Splitting this up in seperate packages for different board architectures is relatively easy (just duplicating the package and changing the config) or can be done with the subpackage support in alpine.

I'm currently modifying the APKBUILD to be closer to the linux-vanilla one again since we have module loading working for this so it also generates a linux-postmarketos-dev package.

Enableing more architectures in the kernel doesn't seem to increase the build time a lot, it just makes the resulting kernel image bigger.

Member

MartijnBraam commented Jul 22, 2017

My current plan was ignoring the size of the kernel until it becomes a real problem since not a lot of devices have mainline kernel support yet. The devices that have very tiny boot partitions aren't devices that have mainline support at all. Splitting this up in seperate packages for different board architectures is relatively easy (just duplicating the package and changing the config) or can be done with the subpackage support in alpine.

I'm currently modifying the APKBUILD to be closer to the linux-vanilla one again since we have module loading working for this so it also generates a linux-postmarketos-dev package.

Enableing more architectures in the kernel doesn't seem to increase the build time a lot, it just makes the resulting kernel image bigger.

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 22, 2017

Member

That sounds like a good plan to me!

(In my opinion, maintaining different packages becomes a problem when shipping a security patch for example (or if we decided to also build -hardened versions, with Alpine's grsec fork).)

Member

ollieparanoid commented Jul 22, 2017

That sounds like a good plan to me!

(In my opinion, maintaining different packages becomes a problem when shipping a security patch for example (or if we decided to also build -hardened versions, with Alpine's grsec fork).)

@craftyguy

This comment has been minimized.

Show comment
Hide comment
@craftyguy

craftyguy Jul 23, 2017

Member

Just to confirm, the direction is to ultimately create one unified kernel for all devices (based on mainline) rather than having multiple mainline kernels for multiple devices?

Member

craftyguy commented Jul 23, 2017

Just to confirm, the direction is to ultimately create one unified kernel for all devices (based on mainline) rather than having multiple mainline kernels for multiple devices?

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 23, 2017

Member

Yeah the grand plan is have one kernel for everything (like on desktop linux). We might have to settle for a few seperate linux kernels (build from the same tree so no extra maintenance time) so the kernel binary doesn't grow too big.

Member

MartijnBraam commented Jul 23, 2017

Yeah the grand plan is have one kernel for everything (like on desktop linux). We might have to settle for a few seperate linux kernels (build from the same tree so no extra maintenance time) so the kernel binary doesn't grow too big.

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 23, 2017

Member

@ollieparanoid do you know where and how in the build process the dtbs should be appended now?

Member

MartijnBraam commented Jul 23, 2017

@ollieparanoid do you know where and how in the build process the dtbs should be appended now?

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 23, 2017

Member

In mkinitfs: before creating the uinitrd and bootimg I suggest a new function for appending the dtb file (so here).

We could do one of the following:

  1. Create a vmlinuz+dtb file in mkinitfs and adjust all code paths to use vmlinuz+dtb instead of vmlinuz when it is present.
  2. In the mainline kernel APKBUILD, do not install the kernel as vmlinuz, but as vmlinuz-without-dtb or something, then create a vmlinuz file in mkinitfs, when the vmlinuz-without-dtb file is present.

I think the second solution is more elegant, what do you think?

Member

ollieparanoid commented Jul 23, 2017

In mkinitfs: before creating the uinitrd and bootimg I suggest a new function for appending the dtb file (so here).

We could do one of the following:

  1. Create a vmlinuz+dtb file in mkinitfs and adjust all code paths to use vmlinuz+dtb instead of vmlinuz when it is present.
  2. In the mainline kernel APKBUILD, do not install the kernel as vmlinuz, but as vmlinuz-without-dtb or something, then create a vmlinuz file in mkinitfs, when the vmlinuz-without-dtb file is present.

I think the second solution is more elegant, what do you think?

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 23, 2017

Member

I realize that the mkinitfs does a lot more now than making the initramfs. Maybe we should rename that program. But this is where I think it belongs.

Member

ollieparanoid commented Jul 23, 2017

I realize that the mkinitfs does a lot more now than making the initramfs. Maybe we should rename that program. But this is where I think it belongs.

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 23, 2017

Member

And about how to append: cat vmlinuz dtbfile > vmlinuz+dtb should do it.

Member

ollieparanoid commented Jul 23, 2017

And about how to append: cat vmlinuz dtbfile > vmlinuz+dtb should do it.

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 23, 2017

Member

I think it's better to use vmlinuz-dtb since that's used by more linux mainline stuff and the current stuff we use. Also all current images are suffixed with the flavor so I think you would get:

For devices with dtb:

vmlinuz-postmarketos-dtb
vmlinuz-postmarketos

and for devices without dtb (I don't think any mainline devices would boot without dtb) you just don't have the -dtb file.

Then create_bootimg() needs to use the -dtb if the deviceinfo_dtb variable is set and otherwise use the plain kernel.

The non-mainline devices currently don't have the deviceinfo_dtb set and write the image+dtb to the plain vmlinuz-flavor file so they keep working without a change.

Member

MartijnBraam commented Jul 23, 2017

I think it's better to use vmlinuz-dtb since that's used by more linux mainline stuff and the current stuff we use. Also all current images are suffixed with the flavor so I think you would get:

For devices with dtb:

vmlinuz-postmarketos-dtb
vmlinuz-postmarketos

and for devices without dtb (I don't think any mainline devices would boot without dtb) you just don't have the -dtb file.

Then create_bootimg() needs to use the -dtb if the deviceinfo_dtb variable is set and otherwise use the plain kernel.

The non-mainline devices currently don't have the deviceinfo_dtb set and write the image+dtb to the plain vmlinuz-flavor file so they keep working without a change.

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 23, 2017

Member

In d307488 I've written some (untested) code that implements my idea above.

Member

MartijnBraam commented Jul 23, 2017

In d307488 I've written some (untested) code that implements my idea above.

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 23, 2017

Member

I'm hitting an issue now with mkinitfs. It's called twice directly in succession with different parameters.

Once with mkinitfs postmarketos and then mkinitfs postmarketos-dtb
log:

(171/171) Installing xorg-server-xwayland (1.19.3-r2)
Executing busybox-1.27.0-r2.trigger
Executing dbus-1.10.18-r0.trigger
Executing kmod-24-r0.trigger
Executing fontconfig-2.12.4-r0.trigger
Executing mkfontscale-1.1.2-r0.trigger
Executing mkfontdir-1.0.7-r0.trigger
Executing postmarketos-mkinitfs-0.1.8-r0.trigger
==> initramfs: creating /boot/initramfs-postmarketos
install: can't stat 'drm': No such file or directory
install: can't stat 'drm_kms_helper': No such file or directory
install: can't stat 'ext4': No such file or directory
Creating 800x480 splashscreen
Creating 800x480 splashscreen
==> kernel: appending device-tree omap3-n900
==> initramfs: creating uInitrd
Image Name:   uInitrd
Created:      Sun Jul 23 14:57:48 2017
Image Type:   ARM Linux RAMDisk Image (uncompressed)
Data Size:    7063001 Bytes = 6897.46 kB = 6.74 MB
Load Address: 00000000
Entry Point:  00000000
==> kernel: creating uImage
Image Name:   postmarketos
Created:      Sun Jul 23 14:57:48 2017
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3826793 Bytes = 3737.10 kB = 3.65 MB
Load Address: 80008000
Entry Point:  80008000
Executing glib-2.52.1-r0.trigger
Executing shared-mime-info-1.8-r0.trigger
Executing gdk-pixbuf-2.36.6-r1.trigger
Executing gtk-update-icon-cache-2.24.31-r0.trigger
OK: 627 MiB in 187 packages
(032481) [16:57:52] Calculate depends of packages ['postmarketos-mkinitfs'], arch: armhf
(032481) [16:57:52] Calculate depends of packages [], arch: armhf
(032481) [16:57:52] (rootfs_nokia-rx51) mkinitfs postmarketos
(032481) [16:57:52] (rootfs_nokia-rx51) % mkinitfs -o /boot/initramfs-postmarketos 4.12.0
==> initramfs: creating /boot/initramfs-postmarketos
install: can't stat 'drm': No such file or directory
install: can't stat 'drm_kms_helper': No such file or directory
install: can't stat 'ext4': No such file or directory
Creating 800x480 splashscreen
Creating 800x480 splashscreen
==> kernel: appending device-tree omap3-n900
==> initramfs: creating uInitrd
Image Name:   uInitrd
Created:      Sun Jul 23 14:57:59 2017
Image Type:   ARM Linux RAMDisk Image (uncompressed)
Data Size:    7062918 Bytes = 6897.38 kB = 6.74 MB
Load Address: 00000000
Entry Point:  00000000
==> kernel: creating uImage
Image Name:   postmarketos
Created:      Sun Jul 23 14:57:59 2017
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3826793 Bytes = 3737.10 kB = 3.65 MB
Load Address: 80008000
Entry Point:  80008000
(032481) [16:57:59] Calculate depends of packages ['postmarketos-mkinitfs'], arch: armhf
(032481) [16:57:59] Calculate depends of packages [], arch: armhf
(032481) [16:57:59] (rootfs_nokia-rx51) mkinitfs postmarketos-dtb
(032481) [16:57:59] ERROR: [Errno 2] No such file or directory: '/home/martijn/.local/var/pmbootstrap/chroot_rootfs_nokia-rx51/usr/share/kernel/postmarketos-dtb/kernel.release'
(032481) [16:57:59] Run 'pmbootstrap log' for details.
Member

MartijnBraam commented Jul 23, 2017

I'm hitting an issue now with mkinitfs. It's called twice directly in succession with different parameters.

Once with mkinitfs postmarketos and then mkinitfs postmarketos-dtb
log:

(171/171) Installing xorg-server-xwayland (1.19.3-r2)
Executing busybox-1.27.0-r2.trigger
Executing dbus-1.10.18-r0.trigger
Executing kmod-24-r0.trigger
Executing fontconfig-2.12.4-r0.trigger
Executing mkfontscale-1.1.2-r0.trigger
Executing mkfontdir-1.0.7-r0.trigger
Executing postmarketos-mkinitfs-0.1.8-r0.trigger
==> initramfs: creating /boot/initramfs-postmarketos
install: can't stat 'drm': No such file or directory
install: can't stat 'drm_kms_helper': No such file or directory
install: can't stat 'ext4': No such file or directory
Creating 800x480 splashscreen
Creating 800x480 splashscreen
==> kernel: appending device-tree omap3-n900
==> initramfs: creating uInitrd
Image Name:   uInitrd
Created:      Sun Jul 23 14:57:48 2017
Image Type:   ARM Linux RAMDisk Image (uncompressed)
Data Size:    7063001 Bytes = 6897.46 kB = 6.74 MB
Load Address: 00000000
Entry Point:  00000000
==> kernel: creating uImage
Image Name:   postmarketos
Created:      Sun Jul 23 14:57:48 2017
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3826793 Bytes = 3737.10 kB = 3.65 MB
Load Address: 80008000
Entry Point:  80008000
Executing glib-2.52.1-r0.trigger
Executing shared-mime-info-1.8-r0.trigger
Executing gdk-pixbuf-2.36.6-r1.trigger
Executing gtk-update-icon-cache-2.24.31-r0.trigger
OK: 627 MiB in 187 packages
(032481) [16:57:52] Calculate depends of packages ['postmarketos-mkinitfs'], arch: armhf
(032481) [16:57:52] Calculate depends of packages [], arch: armhf
(032481) [16:57:52] (rootfs_nokia-rx51) mkinitfs postmarketos
(032481) [16:57:52] (rootfs_nokia-rx51) % mkinitfs -o /boot/initramfs-postmarketos 4.12.0
==> initramfs: creating /boot/initramfs-postmarketos
install: can't stat 'drm': No such file or directory
install: can't stat 'drm_kms_helper': No such file or directory
install: can't stat 'ext4': No such file or directory
Creating 800x480 splashscreen
Creating 800x480 splashscreen
==> kernel: appending device-tree omap3-n900
==> initramfs: creating uInitrd
Image Name:   uInitrd
Created:      Sun Jul 23 14:57:59 2017
Image Type:   ARM Linux RAMDisk Image (uncompressed)
Data Size:    7062918 Bytes = 6897.38 kB = 6.74 MB
Load Address: 00000000
Entry Point:  00000000
==> kernel: creating uImage
Image Name:   postmarketos
Created:      Sun Jul 23 14:57:59 2017
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3826793 Bytes = 3737.10 kB = 3.65 MB
Load Address: 80008000
Entry Point:  80008000
(032481) [16:57:59] Calculate depends of packages ['postmarketos-mkinitfs'], arch: armhf
(032481) [16:57:59] Calculate depends of packages [], arch: armhf
(032481) [16:57:59] (rootfs_nokia-rx51) mkinitfs postmarketos-dtb
(032481) [16:57:59] ERROR: [Errno 2] No such file or directory: '/home/martijn/.local/var/pmbootstrap/chroot_rootfs_nokia-rx51/usr/share/kernel/postmarketos-dtb/kernel.release'
(032481) [16:57:59] Run 'pmbootstrap log' for details.
@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 23, 2017

Member

The appending works now, now I only need @ollieparanoid's opinion to check another box in the start post.

Member

MartijnBraam commented Jul 23, 2017

The appending works now, now I only need @ollieparanoid's opinion to check another box in the start post.

MartijnBraam added some commits Jul 23, 2017

Added more drivers for qemu
This adds virtio network support in qemu for the vexpress-a9 machine.
The keyboard and mouse don't work yet.
You can boot into weston if --no-fde is specified

@ollieparanoid ollieparanoid referenced this pull request Jul 24, 2017

Closed

Device qemu #56

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 25, 2017

Member

This kernel seems to work way better on the n900 than the one currently in the master branch (because the display boot bug is dependend on some race condition and this magically makes it work better)

The x86 qemu build can't run yet because of #260 and the qemu armfh build doesn't have input working but still boots fine.

@ollieparanoid how about merging the progress so far to improve the development for the n900 (and let more people mess with qemu)

Member

MartijnBraam commented Jul 25, 2017

This kernel seems to work way better on the n900 than the one currently in the master branch (because the display boot bug is dependend on some race condition and this magically makes it work better)

The x86 qemu build can't run yet because of #260 and the qemu armfh build doesn't have input working but still boots fine.

@ollieparanoid how about merging the progress so far to improve the development for the n900 (and let more people mess with qemu)

@ollieparanoid

Sorry, that it too me so long to answer. I'll be happy to merge this after the typo has been corrected.

@@ -0,0 +1,23 @@
pkgname=device-qemu-amd64

This comment has been minimized.

@ollieparanoid

ollieparanoid Jul 25, 2017

Member

Clever how you've used qemu as vendor and amd64 as device name!

@ollieparanoid

ollieparanoid Jul 25, 2017

Member

Clever how you've used qemu as vendor and amd64 as device name!

Show outdated Hide outdated aports/linux-postmarketos/APKBUILD
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 25, 2017

Member

*took (why can't I edit the review, GitHub?)

...and Travis CI is failing. Please fix that, too.

Member

ollieparanoid commented Jul 25, 2017

*took (why can't I edit the review, GitHub?)

...and Travis CI is failing. Please fix that, too.

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 25, 2017

Member

You can change the review comment (at least I can change yours). you get a edit icon beside the view changes on hover

Member

MartijnBraam commented Jul 25, 2017

You can change the review comment (at least I can change yours). you get a edit icon beside the view changes on hover

MartijnBraam and others added some commits Jul 25, 2017

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 25, 2017

Member

BTW: I didn't think of putting "-dtb" in the flavor, great solution!

Member

ollieparanoid commented Jul 25, 2017

BTW: I didn't think of putting "-dtb" in the flavor, great solution!

@ollieparanoid ollieparanoid merged commit 21228a1 into master Jul 25, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ollieparanoid ollieparanoid deleted the feature/qemu-omap branch Jul 25, 2017

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 25, 2017

Member

(Deleted the branch, I suggest making a new branch for the next PR - please tell me if you think that this does not make sense.)

Member

ollieparanoid commented Jul 25, 2017

(Deleted the branch, I suggest making a new branch for the next PR - please tell me if you think that this does not make sense.)

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Jul 25, 2017

Member

yeah the branch should be deleted I think with github flow

Member

MartijnBraam commented Jul 25, 2017

yeah the branch should be deleted I think with github flow

ollieparanoid added a commit that referenced this pull request Aug 9, 2017

Initial support for the Mozilla Flame device (#335)
* Challenge/buildinfo: slightly improve output

Use "arch" instead of "carch", print "OK!" when a file has been verified
(which is much easier to recognize than "File has the same content").

* [weston] Added compositor-fbdev-Added-parameter-pixman-type patch

It fixes the red screen bug by running weston --pixman-type=2

#54

* pmbootstrap challenge: Put buildinfo.json in changed files list.

...and some smaller fixes:
* make the diff output easier to read
* verify, that only .apk, .buildinfo.json and the APKINDEX have
  changed inside the local repository folder. Because the file
  names of these changes will be used to release files from
  staging to release.

* Move challenge code to own folder (#64)

...and add a stub for 'pmbootstrap challenge APKINDEX.tar.gz'.

* Rewrite pmb.parse.apkindex(): Can get the full index as dict now

Previously, it was only possible to get information about one
package inside the APKINDEX at a time.
This is needed for #64 to verify the APKINDEX.

Please note, that this implementation is actually slower, than
the previous one. But the code is more readable and it makes
caching possible (which will speed up the APKINDEX massively,
especially for the buildinfo.json file generation!)

* pmb.parse.apkindex.parse(): Use caching

Big speed improvement for buildinfo.json files, down from 30-60s to 1-5s.

* Implement challenge for APKINDEX.tar.gz (part of #64)

* gcc-aarch64: remove dependency on bash

This was added by mistake while testing the gcc-aarch64 aport.

* Enabled module installing for the n900 kernel

* Fix #96: Check if build is necessary is broken.

When an APKINDEX contained a package with multiple versions,
pmbootstrap did not use the last version to determine if the
package is out of date (regression from af8c9fc).

* Fixed module loading for n900

* Add testcases for pmbootstrap challenge.

* pmb/challenge/apk.py had to be renamed to pmb/challenge/apk_file.py,
  so the "internal" functions of that file could be accessed, while
  still providing the short notation pmb.challenge.apk().
* zap asks for each buildroot_* chroot, if you want to remove it, not
  only for the one with the device arch
* add new pmb.chroot.tempfolder() function, that creates a temporary
  folder, that belongs to "user" and deletes it, if it already exists.
  this function gets used in a few challenge testcases.

* Close #94: Always rebuild apks when aports change

...even if the pkgver and pkgrel have *not* changed. This should
make development much more intuitive. The detection works by looking
at the last modified timestamps, just like `make` does it.

* Fix race condition in testcase test_challenge_build

* Fix commandline help wording

* Add support for the binary repository, inactive by default (#64)

* New commandline parameter --mirror-pmOS, where the binary repository
  URL for postmarketOS can be specified (empty by default as of now,
  this will be filled with the real URL once the repo works)
* Do not build packages, when they are in the binary repository and
  the version of the package in the binary repository is up-to-date.
* Add a testcase for pmb.build.is_necessary().

* Slightly adjust README.md
* Change "GNU/Linux" into "Linux distribution", because the Alpine Linux developers don't see their distro as GNU/Linux
* Use "hello-world" as example package, as it does not make much sense right now to compile heimdall for armhf

* Update musl-* to 1.1.16-r13

* Added --no-fde parameter

* Use variable in format.py and add labels to partitions

* Find root partition using labels

For some reason /dev/disk/by-label/pmOS_root symlink is not present and
blkid from busybox doesn't support the -L parameter. We could install
blkid from util-linux but I prefered to use shell commands instead

* Mount unencrypted root partition (close #86)

- Do not decrypt if not necessary
- Do not launch telnetd if partition is unencrypted
- Basic initramfs logging to /tmp/boot.log

* Update min apk version/add more apk version checks

* Minimum version: 2.7.2 (which fixes two CVEs)
* Check the minimum apk version before doing something with apk and
  before entering the chroot manually (previously, it has just checked
  the apk-tools-static version, which gets used to set up the chroot)
* Reword the message for an outdated APK version. Most likely it is
  just the outdated http cache, instead of a man-in-the-middle attack.

See also:
alpinelinux/aports@b849b48

* Close #103: Document how to use chroot commands with parameters

* Update musl-* to 1.1.16-r14

* Support relative path for initializing config

Previously, if you passed something like ~/build to the init function
for work directory, it would create a directory, .\~/ in the current
working directory instead of resolving ~/ to the user's home directory.
This allow allows using ../ to specify a path.

* [postmarketos-base][hammerhead] Use weston --pixman-type

Add a new optional variable weston_pixman_type in deviceinfo
This is used to fix the red screen issue (close #54)

* aportgen: Added end of file newline

* Added extract-dtb tool

* gitignore: Added vim temporary files

* Slightly improve error handling

* Fix: Do not swallow traces when crashing before log init (e.g.
  during argument parsing)
* Show a link to the troubleshooting page, when an error happens
* (Formatting done by autopep8 in pmb/config/init.py)

* Prepare for aarch64 host support, fix typo in arch mapping

Relates to #106.

* pmb.chroot.apk.installed(): use apk's internal database

...instead of running apk every time to get the list of installed
packages and their versions. The internal package database from
apk has the same format, as the extracted APKINDEX file (except
that it has more key-value pairs, which we ignore/do not need
right now). So the APKINDEX code has been extended to parse both
tar-packed APKINDEX files and regular text files in the APKINDEX
format.

This is required for #108, for a better detection of outdated
packages (because the internal package database  saves the
package's timestamp, too). A nice benefit is, that this is faster
than calling apk every time and it doesn't fill up the log as much.

I've also used this improved function for determining the apk
version (for the outdated version check), and I've deleted
pmb.parse.other.package_split(), as it is not needed anymore.

* Add new device lg-d855

* Rename args.no_fde to args.full_disk_encryption

* postmarketos-mkinitfs: Major refactoring

* rewrote find_root_partition to make it compatible with the i9100
  again (and added a big description comment about how it works)
* usb network and dhcp server gets started independent of hooks
  now. this really makes sense for development now, we can disable
  this later (and set up networking via OpenRC, when the usb-shell
  hook is not installed)
* telnet splash + unlock telnetd code only runs, when the rootfs
  is encrypted
* usb_setup_android wrapped with a generic usb_setup function, so
  we can support networking for more device types in a clean way
* the IP of the device is now in init_functions.sh, and init_functions.sh
  gets included in all hooks.
* rewrote logging function. it logs all stdout and stderr to
  /pmOS_init.log, except when PMOS_NO_OUTPUT_REDIRECT is specified
  as kernel command line. This way we can see error output from
  all commands.
* used shfmt to format everything, so it looks consistent again
* added automatic shellchecking for init.sh.in and
  init_functions.sh
* clear naming scheme for initfs functions, they all start with a
  verb now (e.g. setup_usb_network, start_udhcpd, ...)

* Fix /proc/cmdline grepping: /proc must be mounted first

* Adjust defconfig path

* Porting to Moto G 2014 "titan"

- Use samsung-i9100 as template

* APKBUILD + deviceinfo (work in progress)

- Removed ACTIVATE_DOWNLOAD_MODE.txt and ACTIVATE_RECOVERY.txt

From lineage-14.1-20170602-nightly-titan-signed.zip

$ unpackbootimg -i boot.img
Android magic found at: 0
BOARD_KERNEL_CMDLINE androidboot.bootdevice=msm_sdcc.1 androidboot.hardware=qcom vmalloc=400M utags.blkdev=/dev/block/platform/msm_sdcc.1/by-name/utags buildvariant=userdebug
BOARD_KERNEL_BASE 00008000
BOARD_RAMDISK_OFFSET 01000000
BOARD_SECOND_OFFSET 00f00000
BOARD_TAGS_OFFSET 00000100
BOARD_PAGE_SIZE 2048
BOARD_SECOND_SIZE 0
BOARD_DT_SIZE 1763328

* Use LineageOS kernel

Copied defconfig from
https://github.com/LineageOS/android_kernel_motorola_msm8226/blob/cm-14.1/arch/arm/configs/titan_defconfig

* Kernel compiles!

- Removed -Werror
- Use linux-lg-mako as base
- Split big .diff in .patch files so that they are automatically applied

- Applied patches to kernel
  Previously it was stuck showing this error:
  https://pastebin.com/L7Q01Q7s

  See:
  https://forum.armbian.com/index.php?/topic/545-building-kernel-with-gcc5/
  xapp-le/owl#4

  Other interesting changes to merge:
  zanezam/boeffla-kernel-cm-bacon@5236526

* After menuconfig and accept all default values

* Flash time!

* Removed some unnecessary changes from gcc5 big patch

* Enable CONFIG_DEVTMPFS=y

* Use dtb image. It finally boots!

* Disable CONFIG_KERNEL_XZ

* udev rule to use touchscreen + calibration

* deviceinfo_generate_bootimg

* Added profile.sh and initfs-hook hacks

Weston works! (and has red screen bug)

Note: I have to manually ssh and
sh /etc/postmarketos-mkinitfs/hooks/00-device-motorola-titan.sh

* Added deviceinfo_weston_pixman_type variable to fix red screen bug

* Changes done after code review

* Changed broadcom config for hammerhead

* Added bcmdhd calibration from google git

* Reverted accidental menuconfig change

* Added dependency on linux-firmware

* Bumped version on hammerhead packages

* Change initialization order, execute hooks before the usb network setup

* Generate boot.img files in mkinitfs

* Only, when deviceinfo_generate_bootimg="true" in the deviceinfo
* It respects a new deviceinfo_kernel_cmdline parameter from the deviceinfo

* Activate boot.img generation in all fastboot-compatible devices

* Device samsung i9070 (#123)

* Add Samsung Galaxy S Advance (i9070) device-specific package

* Add Samsung Galaxy S Advance (i9070) kernel package

* Add initfs-hook script to set framebuffer settings

* Enable adb root access in recovery mode (TWRP)

* Add usb configuration in initfs-hook script
Add weston calibration values
Update description of kernel changes

* Remove TMRP recovery and change init script to boot directly postmarketOS

* Device sony amami (#100)

* Add device: sony-amami
* Add kernel package for sony-amami

* Show path to boot.img and description of commands after pmbootstrap install

This change got lost in this PR merge:
#125

* [titan] Removed duplicate deviceinfo_generate_bootimg in deviceinfo

* i9100: remove recovery/download mode instructions txt file

This info belongs in the wiki instead.

* pmbootstrap flasher: use boot.img file with fastboot

* Close #77: Implement 'pmbootstrap flasher export'

* Fix #128: Remove obsolete 'deviceinfo_flash_offset_base'

* Fixed typo in new boot.img flasher

* Added flavor to the flash_system command for consistency, This fixes issue #133

* Changed flasher.run()s flavor to an optional argument

* Add 'pmbootstrap aportgen busybox-static-armhf' (and other archs)

* Generate aports for busybox-static-armhf and -aarch64

...and add generating both aports to the testsuite.

* Get rid of the _pmb_build_in_native_chroot hack

* Display device count in 'pmbootstrap init'

* Close #92: pmOS-specific /etc/motd, /etc/issue

* deviceinfo: Add link to reference and reword comments

* Display short troubleshooting URL on exception

* Fix #107: Meaningful error for grsecurity/chroot_deny_chmod

* [i9070] Use kernel modules / Wifi support (!)

* Add kernel module compilation
* Add wifi firmware from vendor files (to `/lib/firmware/postmarketos`) and fix modules install path
* Add patch to change hardcoded dhd firmware/nvram paths
* Add wifi module (dhd) to the list of kernel modules that should be loaded at boot time

Details:
#138

Thanks you very much, drebrez \o/

* Fix #139: Update musl-armhf to 1.1.16-r15

* Update busybox-static-* to 1.27.0-r1

* Close #137: Remove "deviceinfo_modules" / minor deviceinfo changes

Minor changes:
* device-lg-hammerhead: remove left-over comment
* device-samsung-i9100: remove fastboot section

* This fixes the following error:
mkbootimg: error: argument --base: invalid parse_int value: ''

So now postmarketos-mkinitfs can really generate boot.img files.

* [mako] Use red screen workaround

* Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129)

TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution.
---
pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has.

I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository.

To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on.

In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option.

In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds".

This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before.

Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases).

The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.

* deviceinfo: add heimdall partitions / make flasher variables optional

* Fix race condition in test_challenge_build (APKINDEX timestamp)

* Challenge APKINDEX.tar.gz: add logging

* Challenge APKINDEX.tar.gz: allow .diff.md files in the repository

* Fix #107: Grsec check: read file in /proc as root

* Properly update /etc/apk/repositories when the mirrors change

This is important for the binary repository scripts, so it's feasible
to test the binary package build and challenge process locally without
setting up a new chroot whenever changing the repo URLs.

Also it behaves a bit more intuitively, because it really uses the
repo URL specified on the commandline, even when the chroot is already
set up.

* Depends parsing: Properly handle empty depends / buildinfo: strict depends

* Properly support specifying a local folder as --mirror-pmOS

This is required for developing and testing the binary repository
scripts (see #64). Changes:
* When specified, the local folder gets mounted inside the chroots
  as /mnt/postmarketos-mirror
* The apkindex_files() function outputs the correct path to the local
  repository (it does *not* hash the URL in that case, which would
  be wrong)
* /etc/apk/repositories: when the pmOS mirror is a local folder,
  the path "/mnt/postmarketos-mirror" gets added to that file instead
  of the outside path (so apk finds it properly inside the chroot)

* Add new dependency test package: hello-world-wrapper

* Add ConfigFS based usb network configuration

This extends the initramfs init script to setup USB networking on kernels
using USB_CONFIGFS.

* AutoPEP8 1.3.2 slightly changed automatic formatting

* Fix #194: ismount() fails on symlinks

* Update gcc-* to 6.4.0

* postmarketos-mkinitfs: Bump pkgver to 0.1.5

* Challenge APK files: less output by default

* Update heimdall script to flash both (kernel+initramfs) at once

* RX51 - Add touchscreen calibration udev rule

* [RX51] Add linux-firmware package

This removes the call to `firmware_install` in the APKGBUILD for
linux-nokia-rx51, and adds the `linux-firmware` package as a dependency
for device-nokia-rx51.

There are two immediate benefits for this:
1) wifi now works

2) the FM radio device was causing the boot to take ~2 minutes longer
than it should, now that the firmware is present there's no delay in
initializing this device

* Account for uInitrd filename change in u-boot script (rx-51)

The uInitrd filename is appended with 'flavor' so, for example, with the
rx-51 the generated file is named 'uInitrd-nokia-rx51'. This corrects
the rx51's u-boot script to account for this change in filename.

* [RX51] Update to mainline kernel

A couple of issues are resolved by moving to the mainline kernel (from Pali's
kernel):

1) 'reboot' command now works as intended
2) boot is *much* faster
3) charging LED seems to be working now

* Reduce complexity in pmbootstrap.py (#160)

Thanks, @cclauss!

* Dictionary lookup of event loop actions

This was kind of an experiment that adds lots of blank lines to this script -- feel free to reject it.

The observation that triggered this was that in main() all actions can execute using just one parameter, args.  This means that it is possible to use args.action to do a dict lookup to get the corresponding function.  This approach adds tons of blank lines to the script in exchange for a rapid seek to the action implementation and a substantial reduction of the McCabe complexity of main().

$ __flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics__
```
./pmbootstrap.py:43:1: C901 'main' is too complex (23)
./pmb/aportgen/core.py:38:1: C901 'rewrite' is too complex (17)
[ ... ]
```

* Remove action_dict and use locals() instead

* Fix PR #160, Fix #172 (#173)

Thanks craftyguy!
Probably meant to call 'globals()' and not 'locals()'. As @MartijnBraam
pointed out in IRC, probably best to have these functions in a separate
module so they can be parsed with getattr

* [RX51] Export u-boot kernel image with flasher

This properly exports the uImage kernel, which is used by u-boot, when
running 'flasher export'. Note the change to the uImage name to follow
the pattern "uImage-$flavor", which in the case of the RX 51 is
"uImage-nokia-rx51".

* Add acpid package (#170)

Thanks craftyguy!

This commit adds the acpid daemon. This requires a kernel with NETLINK
support compiled in before daemon will run. acpid will be extremely
helpful for devices that have external buttons which need event
handling. The nokia-rx51 is full of such devices/switched!

This was heavily leveraged from the Arch Linux PKGBUILD, thank you Arch
maintainers/contributors!

* [weston] Added more patches

- Add support for ABGR (Asus grouper / Nexus 7 2012 tablet)
- Print guessed pixman type

* Close #174: Put frontend functions in extra file (#185)

* [RX51] Enable DRM in kernel

This enables DRM (Direct Rendering Manager) support in the kernel for
the RX51. This opens up the possibility of using the Weston DRM backend,
which in turn allows for DPMS (I hope) and other cool tricks.

n900:~# glxinfo|head
name of display: :0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4

n900:~# glxgears
94 frames in 5.0 seconds = 18.772 FPS

* Two trivial changes (typo + info message) (#179)

Thanks, Pablo Castellano!

* Fixed typo

* Guide user about what are the next steps after flashing kernel and password

* Increased postmarketos-mkinitfs' pkgver and rewording

Changes requested:
#179

* Fix #200: Parse depends_dev in APKBUILDs

* Fix #145: add deviceinfo_flash_offset_base again (#175)

* Fix #145: add deviceinfo_flash_offset_base again

* Add default OFFSET_BASE value in flasher if deviceinfo_flash_offset_base variable is not set
...and set correct default base value in create_bootimg function

* Adjust gcc-armhf/gcc-aarch64 to upstream

* [RX51] Add support for physical events w/ acpid

This adds a way to handle physical events like keypad slide switch,
screen lock, camera lid, and others. This uses acpid from busybox (I was
wrong about the real acpid package), and anl acpi.map file to map events
to 'scripts'. The scripts are symlinked to /etc/acpi/handler.sh,
automatically by the new post-install script, where
they are handled based on which script was called. This allows for
easily adding whatever is necessary for #178.

If this PR is merged, I will create a new wiki page detailing how others
can use this framework on other devices.

There's an issue with Busybox's acpid, where it is unable to detect new
devices added after the daemon loads. This is the case on the N900,
where openrc starts acpid well before the kernel is done modprobing
drivers (e.g. gpio_keys), so it's necessary to restart the acpid daemon
after some time to 'load' the new devices.

* Add asus-grouper (#180)

* Add device HTC Desire (Bravo) support (#202)

* Add HTC Desire (Bravo) device-specific and kernel package

* Enable kernel virtual terminal support

* Add framebuffer and usb configuration in initfs-hook script

* Add udev required kernel options and set correct framebuffer resolution

* Add touchscreen device rules and enabled kernel logs (dmesg)

* Add deviceinfo_flash_offset_base in deviceinfo and update other offsets

* Fix #209: Cross-compiling fails with C++ code

* Various distccd related improvements, mostly respect --verbose (#216)

I've done some refactoring while debugging #209.
* Unused file `pmb/build/crosscompiler.py` removed (that was a
  left over from `_pmb_build_in_native_chroot` hack
* Do verbose logging in distccd, when `pmbootstrap --verbose` is
  being invoked
* Restart distccd, when the commandline has changed (e.g. when the
  currently running version was not verbose, and the new one is
  verbose.) Prior to this change, it only got restarted, when the
  architecture changed (so it did not allow changing the job count
  on the fly for example).
* Insert missing whitespace in arguments help.

* Fix #162: flash_system: meaningful error for missing image

* Update busybox-static-* to upstream

* Close #215: Write explicity what passwords are for in install

* Handle generating weston.ini in post-install script (#201)

This PR automatically creates the weston.ini in the
postmarketos-base post-install script based on options set in the
deviceinfo file for a particular device. This replaces #191.
If weston.ini exists, then no modifications are made.

(Optional) options supported in this PR for deviceinfo are:
deviceinfo_weston_core_modules - defaults to NONE if none specified
deviceinfo_weston_core_backend - defaults to 'fbdev-backend.so' if none specified
deviceinfo_weston_keymap_rules - keymap rule, found under /usr/share/X11/xkb/rules/
deviceinfo_weston_keymap_model - keymap model

* Re-enable XWayland for devices that had it enabled (#221)

This re-enables the XWayland module for devices which did not use the
'dirty hack' that was removed in #201.

* [RX51] Set keyboard layout for weston

This configures the keymap in weston so that the rx51 keyboard behaves
as expected (e.g. shift key, numbers, etc)

* pmbootstrap shutdown: Make cryptsetup closing slightly more robust

I've had a strange case where `cryptsetup status` did not work
inside the chroot anymore, and only zapping it would resolve it.
But I couldn't zap, because the status check was preventing shutdown
(on which zap depends) from working. This commit works around that.

* [RX51] Update to Linux-4.12.3

* Fix #219: pmbootstrap could not find g++-armhf/g++-aarch64

* Fix PEP8 syntax

Spaces around a plus sign.

* Close #195: Ignore parameters in dependencies (#225)

* Ignore `>`, `<`, `=` and `!` operators, when they are specified in
  the dependencies. This was the desired behavior before, but it was
  not implemented correctly (so it wouldn't ignore them everywhere).
  Of course the real Fix would be to honor these operators like apk
  does. But this isn't feasible right now, and it should work for
  most, if not all, our use-cases. I have documented this in the wiki
  under build internals and if we happen to need correct operator
  handling, we should do it then.

Minor other changes:
* `pmbootstrap parse_apkindex`: support optional package parameter to
  only show the parsed content for one package.
* Support building most python APKBUILDs by replacing ${pkgname#py-}
  properly

* Fix #219 for real!

100% tested. Assuming that you don't need to test an obvious change,
because it only changes one line is dangerous. I will learn from this,
sorry for the inconvenience.

`gcc-armhf` compiles fine again with that change, and it is able to
cross-compile packages as it used to.

* Improve the FDE hint, you need to type in the password twice.

* Close #198: Add possibility to override the kernel commandline during boot (#217)

Usage example:
pmbootstrap flasher boot --cmdline "PMOS_NO_OUTPUT_REDIRECT"

Thanks, drebrez!

* Fix #230: pmbootstrap flasher flash_system is broken

Regression from the cmdline argument feature.

* Created splash screen generator (#206)

Nice configurable splash screen generator, thanks Martijn Braam!

* Ignore directories while searching for loop devices (#235)

Thanks, cmdr2! Here's the text from his PR:

Fixes an issue on alpine-vanilla, where `/dev/loop` is a directory (containing `/dev/loop/0`, `/dev/loop/1` ..etc), and causes `./pmbootstrap install` to fail.

Ignoring the `/dev/loop` directory works because `/dev/` also contains files named `/dev/loop0`, `/dev/loop1` ..etc, which the new code will still pick up.

The `install` command passes after this fix on my alpine-vanilla.

* Update runtime error message thrown for grsec/hardened kernels (#236)

* Update runtime error message thrown for grsec/hardened kernels, saying that they are not supported right now, #107

* Remove unnecessary check for grsec enable/disable status - the build will not work in either case

* New splash images in mkinitfs (#240)

Thanks, Martijn Braam!

* Made postmarketos-mkinitfs use postmarketos-splash to generate splash images
* Added screen sizes to all deviceinfo files
* Bumped pkgrel for all device packages

* Do not store aports directory in config file (#247)

This addresses #246 by not storing the aports directory in the config
file. The default location is still available (from config/__init__.py),
and can be overridden.

* De-couple weston from postmarketos-base (#233)

Thanks, craftyguy!

- UI is selectable with `pmbootstrap init`. Currently only 'weston' and 'none' are options. It'll automatically pick up any new `postmarketos-ui-*` packages added at later dates.
- splits off weston packages install from postmarketos-base and puts
them into postmarketos-ui-weston. Also note that NO weston packages are
installed by "pmbootstrap install" by default unless the user selects a ui in the `init`
- configuration of weston is now in postmarketos-ui-weston.
- the demos have been spun off to `postmarketos-demos`, and `postmarketos-ui-weston` lists this package as a dependency.

* Close #212: Split heimdall in heimdall-isorec and heimdall-bootimg (#254)

* Unified linux-postmarketos kernel for qemu and n900 (#228)

The creates the linux-postmarketos package which for now supports the n900 (and probably the other maemo devices) and qemu with the vexpress-a9 machine simulation.

I've put the generated dtbs in /usr/share/dtb for now and set the dtb field in the deviceinfo for the n900 and my new qemu device.

* Introduced linux-postmarketos-omap
* Unified kernel progress
* Created kernel bootable with qemu
* Updated n900 deviceinfo for generic kernel
* Changed qemu device to vexpress
* Updated APK comments and added linux-postmarketos-dev package
* Append dtb in mkinitfs
* Fixed bootscript on n900 for the generic kernel
* Don't detect double flavors with -dtb appending
* Added graphics drivers for vexpress (qemu)
* Added more drivers for qemu

This adds virtio network support in qemu for the vexpress-a9 machine.
The keyboard and mouse don't work yet.
You can boot into weston if --no-fde is specified

* Enable syslog on boot

This enables the busybox syslog implementation, which logs to
/var/log/messages, on boot. This is very helpful for debugging
applications which write to syslog (e.g. NetworkManager)

syslog is being set up to log to shared memory (not to disk), so log is
lost 1) on reboot and 2) if the service is restarted. It can be read
with `logread`

* Fix #242: Validate input in pmb.helpers.cli

* Validate all inputs from `pmbootstrap init`
* Add a new `confirm()` function, that validates input of yes/no
  questions properly
* All questions loop until they have a valid answer now

* Added more drivers to the x86_64 kernel (#272)

This adds support for ide, sata, mice, touchscreens,
displays and serial ports in qemu.

* Low hanging fruit basket (Close #220, Close #239) (#253)

* postmarketos-splash: change arch from "all" to "noarch"
* lg-mako: use .tar.gz instead of .zip (because that's the reference
  APKBUILD)
* #220: Allow specifying multiple packages for checksum, build,
  aportgen
* #239: Add chroot shortcuts (--rootfs and --buildroot)
* Show chroot and command before entering chroot

* Fix #260, fix #87: Don't cross-compile when not necessary (#265)

* Add dtbtool package

* Actually use the PROGNAME variable and change filename to camelCase as in upstream

* Minor spelling fixes

* Various improvements for 'pmbootstrap export' (Fix #232) (#250)

* Various improvements for 'pmbootstrap export' (Fix #232)
* Do not try to generate the system image, but print a notice that
  it was not generated yet
* Display export path
* New default path: /tmp/postmarketOS-export (instead of current
  working dir)
* Create the export folder, if it does not exist yet
* Prettier output in the export message
* Adjust export message in install for consistency

* pmbootstrap install: Add link to usb-hook (#268)

* Fix #166: pmbootstrap shutdown: umount deep folder levels first (#274)

* Refactored `umount_all` to get the list of folders to be umounted from
  `umount_all_list`, so we can test that function in a test case.
* Adjusted `umount_all_list` to return the deep folder levels first
* Added a testcase for that
* Remove redundant calls to `umount_all()` (which were from a time before
  `pmbootstrap` was released, in which failing commands did not cause
  `pmbootstrap` to raise an exception)

* [RX51] Enable swap device and mount /boot by default

This enables the existing swap device (set up by Maemo) and sets up
/boot to mount on startup

* Fix #226: Building busybox recurses forever. (#252)

If you really want to build busybox, I recommend turning the
"timestamp based rebuilds" feature off, otherwise it will build
for all architectures all the time whenever you change something,
even if you do not increase the version number (that's the idea
of that feature). This is, because busybox is a dependency for
basiscally everything, so it must get updated whenever you install
something, in case it was out of date.

It is easier to simply rename the package.

* Fix #269: Always use truncate to create empty image file (#276)

I've realized that truncate is provided by busybox anyway, so no
need to install coreutils for this.

This should fix creating system image files on ZFS, where fallocate
is not implemented.

* Split up initramfs to make it smaller (Close #211) (#257)

Huge thank you to @drebrez for his amazing work on this PR!

* Add generation of initramfs-extra with additional binaries
Extract both initramfs with `pmbootstrap initfs extract`
Add new splashscreens for missing partitions/files errors
Changes in init script:
 - use busybox findfs applet to find boot partition
 - mount boot partion
 - extract initramfs-extra
 - show error splashscreens accordingly and loop forever
 - start usb unlock directly from unlock_root_partition (hook removed)

* Print out a text message for serial debugging in case of errors
Add initramfs-extra files to `pmbootstrap initfs ls` output

* Fix trailing whitespace in comment...

* ls: Indicate which initramfs we're looking at / add wiki link

I've rewritten the initramfs-development article to reflect the
changes made in this PR. It will be a good read for someone who
extracted the initramfs and wants to know why we have two files.

* Add device htc-ace to master (#281)

@VictorNine did the port, all credit goes to him (and @drebrez and everyone else who helped him)!

* Added device HTC Ace (#181)

* Add HTC Desire HD (Ace) device-specific and kernel package

* Enable kernel virtual terminal support

* Added touchscreen device rules

* Added linux-firmware pkg. WIFI is now working

* device-htc-ace: Add deviceinfo_flash_offset_base

* Fix #282: '/bin/ash': Permission denied

This was a regression from the improved input validation PR, which
created the chroot_native folder with the wrong permissions (to test
if we have write access to the work folder).

sjamaan found out, that this was the cause of the issue, and also
explained how to fix it, thanks a lot!

* Close #194: Aports subfolders! See migration guide in the wiki (#227)

Migration guide:
https://github.com/postmarketOS/pmbootstrap/wiki/Migration-to-aports-subfolders

* Bump pmbootstrap version to 0.2.0

* Move linux-postmarketos to from device to main

* Move two cross-packages to the cross folder

* Usability improvements, fix #278 (#279)

* Quote architecture in logging message for easier reading
* Added shortcut arguments for --rootfs, --buildroot and --suffix
* Simply remove beforehand link to nowhere if exists (fix #278)
Fixed crash when symlink already existed but pointed to a non-existing location

* Update to Linux Kernel 4.12.4 (#285)

This updates the linux-postmarketos kernel to 4.12.4, and fixes an issue
where the APKBUILD was pulling down the 4.12 source but not the 4.12.x
patches. I think that pulling down the major.minor source and a sep.
package for patches to update to the major.minor.minor(er?) release adds
unnecessary complexity since kernel.org already hosts the sources with
these patches applied (e.g. in this case the 4.12.4 source)

* pmb.helpers.ui scan for UIs in new aport structure (#290)

This fixes #289.

* Move wpa_supplicant to default runlevel (#286)

Having it in boot doesn't actually work, the service is being started
too early and fails. This change moves it to the default runlevel. This
change also removes "wifi-handler" since that service doesn't actually
exist.

* Fix #288: Checksum *-install scripts in APKBUILDs (#294)

Per the APKBUILD reference, , package *-install scripts should be checksummed to guard against incomplete/corrupt downloads. The postmarketos-base package is one example where this is not being implemented currently.

* Fix checksum of postmarketos-base

* Allow 'pmbootstrap menuconfig' without specifying "linux-" (#295)

We can allow both ways:
* pmbootstrap menuconfig linux-motorola-titan
* pmbootstrap menuconfig motorola-titan

The former will show a tip about the second
This also prevents users from running menuconfig on aports like
'device-motorola-titan' or 'mkbootimg'

* Implemented checksum test in the travis script (#298)

* Implemented checksum test in the travis script
* Added workaround for squashed forced pushes
* Made test possible to run locally
* Match multiple git versions for the working directory message

* Add possibility to export Odin flashable tar (#297)

Add possibility to export Odin flashable tar for devices where the
flasher method is heimdall-isorec or heimdall-bootimg

* Fix test script not crashing hard enough in travis (#301)

Made check_checksums.py exit 1 on errors

* Fix #300: Properly unset environment in chroot (#302)

* Print "git diff" output when the checksums tests has failed (#306)

In some cases this output is useful to identify the root of the problem.
See #303 (comment)

* Fixed dtb installation with mainline kernel (#307)

* Fixed dtb installation with mainline kernel

* Travis checksum pr fix (#308)

* Check the correct branch in PR runs

* Add device Samsung Galaxy Note 2 (N7100) (#293)

* [RX51] workaround lines issue (#309)

This works around the lines issue using a patch from Ti: https://lkml.org/lkml/2017/6/30/163

* init: Allow to specify extra packages that will be always installed to rootfs (#304)

* Fix #223: Use `.tar.gz` instead of `.zip` for kernel sources (#315)

* Update musl-* to 1.1.16-r17 (#320)

* Implement sparse system images (fix #299) (#303)

* Packaged libsparse

libsparse from the Android project provides multiple tools like img2simg
and simg2img.
These are used to split a large image for the system partition into
separate smaller chunks with sparse headers

This is required for several devices (at least bullhead, fp2 and titan)
because it fixes the "Invalid sparse file format at header magi" error

#299

* Added new variable deviceinfo_flash_sparse (fixes #299)

Right after the system image is generated, pmbootstrap checks this
variable. In case it is true, run img2simg on it

* motorola-titan: enable deviceinfo_flash_sparse

* libsparse: use source from github: anestisb/android-simg2img

It is not that easy to use the upstream archive because everytime
you download it, the files have the current date as creation date
and that makes the file have a different checksum every download
#303 (comment)

* [linux-postmarketos] enable maint. patching of kernel source (close #311) (#313)

This enables applying maint. release patchsets to the kernel source
tarball, in line with Alpine Linux's linux-vanilla APKBUILD.
This addresses #311.

* Close #314: Add --details-to-stdout parameter (#317)

When this parameter is set, `args.logfd` (the log file descriptor)
gets set to `sys.stdout`, so all logging gets directly sent to the
active pmbootstrap session, and not to a log file.

This will be used for Travis jobs, because they get killed after
10 minutes without any output (and builds may take longer).

* Update busybox-static-* to upstream

* Automatically compute the minimum size for the partition and resize it during the boot process (#127)

* Automatically compute the minimum size for the partitions
* Automatically resize the pmOS_root partition during the boot process
* Resize root partition only if there is unallocated space at the end of the device.
* Added more echos to make debugging easier while looking at the pmOS_init.log.
* Updated static_code_analysis.sh script to run shellcheck with `-x` option.

* Fix #319: pmbootstrap menuconfig didn't work for mainline kernel (#328)

Also did some minor refactoring.

* Package Plasma Mobile (All credit to PureTryOut!) (#284)

Plasma mobile compiles, but we didn't get it to run yet.
Thanks to PureTryOut for all this amazing porting work! \o/

* Update busybox-static-* to upstream

* Fixed checksum on travis for removed packages (#332)

* Add check for removed packages

* Change for testing travis

* Fixed package depth

* Removed consolekit2 (upstreamed) / other plasma fixes (#331)

* Removed consolekit2 from the repo as it's now upstreamed

* Add missing makedepend to kwin

* Fix: gcc-armhf not reproducible (#64) (#333)

libstdc++.a from gcc-armhf was not reproducible on Travis (it was, when built locally!). These .a files are just archives of object files .o, and in this case it was caused by a random order of the .o files in the archive.

This PR patches the package() function of the APKBUILD when running pmbootstrap aportgen gcc-armhf (same for aarch64 of course), to extract all .a files, and repack them to be reproducible (by sorting the files before packing them).

As usually, we can still inherit everything from the upstream gcc aport from Alpine, and apply our changes on top of that.

Travis without the patch:
https://api.travis-ci.org/jobs/260402679/log.txt?deansi=true

> CHALLENGE FAILED for usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a:File 'usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a' is different!

Travis with the patch (I've instructed Travis to run off this branch to test it):
https://api.travis-ci.org/jobs/260806203/log.txt?deansi=true

> Done. Your build exited with 0.

* Minor fixes (#316)

* Fixed typo and specify list order

* Simplify code for jobs default value

* Fix 'install: can't stat' message, when no hook is installed (#310)

* Add `-c, --clear` argument for `log` and `log_distccd` actions to clear log files (#344)

* Fix weston on sony-amami (#340)

* Close #334: Check if pkgname is the aport folder name (#347)

...and remove an redundant sentence with broken grammar in another
error message.

* Fix #349: Verify pkgname against folder name after replacing vars

This fixes a regression introduced in 7fe2bc1.
Sorry for the inconvenience!

* Fix refresh rate on sony-amami (#348)

Patch added to hardcode pixclock in framebuffer driver

* Initial Mozilla Flame packages

* Update dtbtool to use LineageOS upstream

* Use mkbootimg --dt flag only when deviceinfo_bootimg_qcdt is defined

* Update mkbootimg to use LineageOS upstream

* Add sparse image flashing on device-mozilla-flame

* Only support SD card install on Mozilla Flame

* Remove useless profile.sh on mozilla flame

* Package DTBs in /usr/lib/linux-mozilla-flame/

* Build and install modules in linux-mozilla-flame

PureTryOut added a commit that referenced this pull request Feb 21, 2018

Unified linux-postmarketos kernel for qemu and n900 (#228)
The creates the linux-postmarketos package which for now supports the n900 (and probably the other maemo devices) and qemu with the vexpress-a9 machine simulation.

I've put the generated dtbs in /usr/share/dtb for now and set the dtb field in the deviceinfo for the n900 and my new qemu device.

* Introduced linux-postmarketos-omap
* Unified kernel progress
* Created kernel bootable with qemu
* Updated n900 deviceinfo for generic kernel
* Changed qemu device to vexpress
* Updated APK comments and added linux-postmarketos-dev package
* Append dtb in mkinitfs
* Fixed bootscript on n900 for the generic kernel
* Don't detect double flavors with -dtb appending
* Added graphics drivers for vexpress (qemu)
* Added more drivers for qemu

This adds virtio network support in qemu for the vexpress-a9 machine.
The keyboard and mouse don't work yet.
You can boot into weston if --no-fde is specified
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.