Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Completely bespoke stage-1 init, shell-free #66

Merged
merged 158 commits into from Feb 4, 2020

Conversation

@samueldr
Copy link
Member

@samueldr samueldr commented Dec 24, 2019

As much as possible shell-free.


This is a big chunk of code. Don't be alarmed.

The goal is to make an init that is entirely based on tasks and dependencies, rather than a chunk of shell script that is hopefully ordered right.

Another goal is to keep the cross-compilation requirement.

A final goal is to keep the closure size as small as possible. The budget is ~8MiB total for the compressed initrd.

In a future improvement, it should be possible to, at first, select a generation to boot into (though stage-2 only), and furthermore, kexec to a kernel+initrd loaded from the system, making this a stage-0, that loads the real stage-1. It may also be possible to go kexec-free, and simply switch_root to another stage-1, though this sounds unlikely to be useful in the Mobile space.

This is currently an incomplete implementation, here's non-exhaustive list of stuff to be done:

  • qualcomm display hacks
  • usb gadget
  • adb gadget
  • rndis gadget
  • dhcp server for networking
  • sshd for networking

Hopefully, those gadget and networking tasks will be easier to implement in a tasks and dependency-based system, rather than chunks of shell. Narrator: "They were"


A thousand-foot view

This adds an mruby build harness. What is that? It allows selecting gems to be compiled in the mruby build, and also allows building an mruby-based project into a binary. This is something that could graduate into Nixpkgs at some point.

This, then, adds the new init script. As of the creation of the PR, it does not clean up old initrd stuff.


Language choice

One of the goal was to get a real programming language. Shell doesn't work here, especially since it's not even full bash!

The current choice is not a final choice, but the current choice for getting things rolling. I initially explored Rust, but had issues getting Rust programs cross-compiling. I probably want to explore using Rust again. Hopefully the language choice will not seep through and re-writing the main ideas and concept will be possible without changing the nix-side implementation.

You don't like Ruby? Eh, everyone has their flaws.


Implementation quality

This is naïve! I don't order the tasks before running them, thus at the very worst I could loop as many time as there are tasks (if all dependencies are met in execution). A first optimization would be to order the tasks on dependencies before running them. Though, the loop executes tightly enough for our needs that it is not causing grief.

Other things may be messy, I am making it all up along the way.

@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch 5 times, most recently from 1a0e129 to 3db9e36 Dec 25, 2019
@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch from 3db9e36 to d76c680 Jan 2, 2020
@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch 4 times, most recently from 6dfabaf to 71f37b0 Jan 12, 2020
@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch 2 times, most recently from 3c6777a to 62dc253 Jan 13, 2020
@samueldr samueldr marked this pull request as ready for review Jan 14, 2020
@samueldr
Copy link
Member Author

@samueldr samueldr commented Jan 14, 2020

As of right now, the implementation is to parity and better than the previous init.

Enhancements:

  • gadgetfs usb gadget support (google-walleye, xiaomi-lavender)

I also believe it is faster to boot. At the very least, there are no artificial delays added. This does have the drawback of having the "empty, white coloured" logo transition happen quickly enough that on some devices it looks like it goes from empty to coloured. Oh well!

Implementation of generation selection is a task for a future PR. Right now I'll be inspecting this PR as best as an author can, and I'll merge it by the end of this week.

The only current issue is that I don't have a google-marlin or a xiaomi-tissot device to test with. I don't know whether they are android_usb or gadgetfs devices. If on the current implementation rndis and adb worked, it was android_usb. Otherwise it's likely it was gadgetfs. cc @danielfullmer @lheckemann

@samueldr samueldr changed the title [WIP] Completely bespoke init, shell-free Completely bespoke init, shell-free Jan 14, 2020
@samueldr samueldr changed the title Completely bespoke init, shell-free Completely bespoke stage-1 init, shell-free Jan 14, 2020
@lheckemann
Copy link
Member

@lheckemann lheckemann commented Jan 14, 2020

Exciting! I'll try this soon.

@danielfullmer
Copy link
Contributor

@danielfullmer danielfullmer commented Jan 15, 2020

I've tested the examples/demo boot and system images on marlin and they seem to be working fine.

marlin should be android_usb, see: https://android.googlesource.com/device/google/marlin/+/refs/tags/android-10.0.0_r25/init.common.usb.rc
However, this does not work currently when setting mobile.usb.mode = "android_usb". Here is the relevant excerpt from the UART. If the rest of the logs are needed just let me know.

....
I, [1970-05-12T03:24:54.717480 #1]  INFO -- : Running Tasks::Mount(adb, /dev/usb-ffs/adb, {:type=>"functionfs", :options=>["nosuid", "noexec", "nodev"]})...
mount: mounting adb on /dev/usb-ffs/adb failed: No such file or directory
F, [1970-05-12T03:24:54.730695 #1] FATAL -- : ********************
F, [1970-05-12T03:24:54.730801 #1] FATAL -- : ********************
F, [1970-05-12T03:24:54.730849 #1] FATAL -- : ********************
F, [1970-05-12T03:24:54.731080 #1] FATAL -- : Handling exception
F, [1970-05-12T03:24:54.731144 #1] FATAL -- : System::CommandError: Command failed... `mount -t functionfs -o nosuid,noexec,nodev adb /dev/usb-ffs/adb` (255)
F, [1970-05-12T03:24:54.731193 #1] FATAL -- : `init` will exit and the kernel will crash.
F, [1970-05-12T03:24:54.731238 #1] FATAL -- : ********************
[   16.649998] c0      1 Kernel panic - not syncing: Attempted to kill init! exitcode=0x00006300
....

Just guessing at this point since I don't know much about this, but notice this line in the init.common.usb.rc from the link above:
write /sys/class/android_usb/android0/f_ffs/aliases adb,mtp,ptp
which occurs immediately before the mount for /dev/usb-ffs/adb. Also note this commit LineageOS/android_device_google_marlin@e94a88f
which would seem to indicate that this should happen before mounting /dev/usb-ffs/adb. I've run out of time tonight to continue testing, however.

Oh, also, hard-reboot and hard-shutdown seem to be missing, so I had to comment them out of examples/demo/configuration.nix

Finally, I know there was some discussion about using systemd in stage-1 NixOS/nixpkgs#72401, and I would be interested in your comments about whether this is feasible / useful for mobile-nixos.

@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch 2 times, most recently from 28a25e3 to c16748d Jan 20, 2020
@samueldr
Copy link
Member Author

@samueldr samueldr commented Jan 20, 2020

Thanks @danielfullmer for noticing the stray hard-shutdown in the demo configuration. I haven't rebuilt the system.img since... a few months back.

@samueldr
Copy link
Member Author

@samueldr samueldr commented Jan 20, 2020

@danielfullmer: if you can try google-marlin with the latest changes, I believe it should be fixed.

@danielfullmer
Copy link
Contributor

@danielfullmer danielfullmer commented Jan 20, 2020

@samueldr Well there's some progress--but now it crashes in ffs_function_enable in the kernel:

273629 #1]  INFO -- : Running Tasks::MSMFBHandle...
I, [1970-05-18T00:59:34.275383 #1]  INFO -- : Running Tasks::Mount(/dev/dis[   14.811936] c1    478 devpts: called with bogus options
k/by-partlabel/vendor_a, /vendor, {:type=>"ext4", :options=>["ro", "nosuid", "noexec", "nodev"]})...
I, [1970-05-18T00:59:34.29[   14.823468] c1      1 enable_store: android_usb: already disabled
7561 #1]  INFO -- : Running Tasks::Mount(adb, /dev/usb-ffs/adb, {:type=>"functionfs", :options=>["nosuid", "noexec", "nodev"]})...
I, [1970-05-18T00:59:34.300622 #1]  INFO -- : Running Tasks::Mount(configfs, /sys/kernel/config, {:type=>"configfs", :options=>["nosuid", "noexec", "nodev"]})...
I, [1970-05-18T00:59:34.303433 #1]  INFO -- : Running Tasks::Mount(devpts, /dev/pts, {:type=>"devpts", :options=>["nosuid", "noexec", "mode=620", "ptmxmode=0666", "gid=3"]})...
I, [1970-05-18T00:59:34.313892 #1]  INFO -- : Running Tasks::Mount(ramfs, /run/keys, {:type=>"ramfs", :options=>["nosuid", "nodev", "mode=750", "gid=96"]})...
I, [1970-05-18T00:59:34.317933 #1]  INFO -- : Running Tasks::SetupGadgetMode...
I, [1970-05-18T00:59:34.318033 #1]  INFO -- : Configuring ANDROID_USB Gadget.
[   14.935451] c2      1 Internal error: Accessing user space memory outside uaccess.h routines: 96000005 [#1] PREEMPT SMP
[   14.945201] Modules linked in:
[   14.948238] c2      1 CPU: 2 PID: 1 Comm: init Tainted: G        W      3.18.140 #2
[   14.955875] c2      1 Hardware name: HTC Corporation. MSM8996pro v1.1 + PMI8996 Marlin A (DT)
[   14.964383] c2      1 task: 0000000000000000 task.stack: 0000000000000000
[   14.971158] c2      1 PC is at ffs_function_enable+0xc/0x38
[   14.976711] c2      1 LR is at enable_store+0x114/0x2d0
[   14.981918] c2      1 pc : [<ffffffc00074c3b4>] lr : [<ffffffc000768c44>] pstate: 20400145
[   14.990164] c2      1 sp : ffffffc0f98fbd50
@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch 2 times, most recently from 6035c5a to e48ad35 Jan 21, 2020
@danielfullmer
Copy link
Contributor

@danielfullmer danielfullmer commented Jan 21, 2020

I was able to get adb to work. Here are the changes I had made: (not intended to be applied as-is, but for discussion)

diff --git a/devices/google-marlin/default.nix b/devices/google-marlin/default.nix
index a3f7391..515a273 100644
--- a/devices/google-marlin/default.nix
+++ b/devices/google-marlin/default.nix
@@ -44,4 +44,10 @@
   };
 
   mobile.system.type = "android";
+
+  mobile.usb.mode = "android_usb";
+  # Google
+  mobile.usb.idVendor = "18D1";
+  # "Pixel" rndis+adb
+  mobile.usb.idProduct = "4EE4";
 }
diff --git a/modules/adb.nix b/modules/adb.nix
index 19a7429..9cf567d 100644
--- a/modules/adb.nix
+++ b/modules/adb.nix
@@ -24,7 +24,8 @@ with lib;
     ];
 
     mobile.boot.stage-1 = {
-      usb.features = [ "ffs" ];
+      #usb.features = [ "ffs" ];
+      usb.features = [ "adb" ];
 
       tasks = [
         (pkgs.writeText "adbd-task.rb" ''
diff --git a/modules/stage-1/tasks/usb-gadget-task.rb b/modules/stage-1/tasks/usb-gadget-task.rb
index a2ed09c..aff7eb2 100644
--- a/modules/stage-1/tasks/usb-gadget-task.rb
+++ b/modules/stage-1/tasks/usb-gadget-task.rb
@@ -144,7 +144,11 @@ class System::AndroidUSB
 
   def activate!()
     System.write(File.join(path_prefix, "enable"), "0")
+    System.write(File.join(path_prefix, "idVendor"), "18D1")
+    System.write(File.join(path_prefix, "idProduct"), "4DD4")
     System.write(File.join(path_prefix, "bDeviceClass"), "0")
+    System.write(File.join(path_prefix, "bDeviceSubClass"), "0")
+    System.write(File.join(path_prefix, "bDeviceProtocol"), "0")
     System.write(File.join(path_prefix, "functions"), @features.join(","))
     sleep(0.1)
     System.write(File.join(path_prefix, "enable"), "1")

I followed the AOSP usb init script for marlin as closely as I could:
https://android.googlesource.com/device/google/marlin/+/refs/tags/android-10.0.0_r25/init.common.usb.rc#69
and noticing that ${sys.usb.config} here would something like rndis,adb-- not ffs. Presumably this has to do with the f_ffs/aliases set earlier.
Setting the idVendor and idProduct between the disable / enable steps was also needed to get it to work--otherwise it didn't have the correct usb id in lsusb.

samueldr added 7 commits Jan 24, 2020
This is part of what may be required to fix ADB on google-marlin.
It looks like it didn't matter for some device, but may be an issue for
`google-marlin`.

I'm not sure where the `ffs` feature name came from.
boot/init/lib/mounting.rb Outdated Show resolved Hide resolved
@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch from 244b184 to b2fb30f Feb 3, 2020
@samueldr samueldr mentioned this pull request Feb 4, 2020
0 of 1 task complete
@samueldr samueldr force-pushed the samueldr-wip:feature/new-init branch from b2fb30f to 95016e1 Feb 4, 2020
@samueldr
Copy link
Member Author

@samueldr samueldr commented Feb 4, 2020

Merging with one known regression:

  • When a filesystem cannot be mounted because it cannot be found, stage-1 will loop indefinitely. See #78
@samueldr samueldr merged commit 8ee49d9 into NixOS:master Feb 4, 2020
@samueldr samueldr deleted the samueldr-wip:feature/new-init branch Feb 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.