[WIP] Improve setup

[Gerschtli]

Hey,

this is a WIP pull request to gather some initial feedback.

The idea was to install all basic packages on boot via a custom nix-channel.

Steps to complete this PR from my perspective:

• All files in /bin should be linked to ~/.nix-profile
  • login
  • login-inner
  • proot
  • sh
• All files in /etc should be linked to ~/.nix-profile
  • home.nix.default
  • nix/nix.conf
  • resolv.conf
  • group and passwd (optional, as I don't know how to get the current uid via nix)
• All files in /usr/bin should be linked to ~/.nix-profile
  • env

Furthermore, I need to refactor the default.nix a bit, it's just a plain set of derivations..

Maybe we should improve the log messages with colors or something else, but this wasn't my primary concern.

What do you say to the way I implemented this? Feedback welcome :)

[t184256]

I read 3, you push 8 more =)

[t184256]

I'm against using nix-env as opposed to depending on stuff from h-m config.

[t184256]

Wait a minute, I feel like I'm missing something.

You want to have an updateable basic-environment even in scenarios when the user doesn't opt in into h-m?

I feel like it brings more complexity and, at the same time, I don't see many use cases for that. Could you elaborate why you think this is necessary?

What I proposed was simpler. Provide initial links in the zipball, pull in a derivation that would replace them from h-m config. No intermediate stage with an opaque uncontrollable magic thingie stealthily added through nix-env, no worries about discoverability, conflicting with the user wishes, updates...

[Gerschtli]

Okay, my idea was as follows:

We build the initial zip ball, as plain and basic as possible. On first boot, we want to replace the static links and fixed paths in the login script to be maintainable after a nix-collect-garbage. With the requirement to not force the user to set up and use home-manager, I implemented the basic-environment env. This derivation ensures that all necessary scripts and packages are installed. Of course there could be more parameterized and the final home-manager module will allow much more fine grained control over the dependencies and scripts. But for now (for the PR) I thought it would be enough to just install the basic-environment.

I needed to come up with a solution to provide a reasonable working environment for both use cases: with and without home-manager.

[t184256]

The main idea itself is neat, but could you please comment on the renamings? I find the new naming confusing. If buildPkgs is not always pinned anymore, why start calling it pinned all of a sudden?

EDIT: This is about 'Use pinnedPkgs only if initialBuild is true '.

[t184256]

I implemented the basic-environment env. This derivation ensures that all necessary scripts and packages are installed. Of course there could be more parameterized and the final home-manager module will allow much more fine grained control over the dependencies and scripts. But for now (for the PR) I thought it would be enough to just install the basic-environment.

In general, I'm fine with merging any small improvements that work and redoing stuff later if I see fit.

But, since we've started disscussing it anyway, let's pause for a moment. I see some ways to parametrize basic-environment if it's pulled in via h-m (function parameters, h-m module). But how would one parametrize it if it's installed through nix-env? Is there some way that I don't know of?

[Gerschtli]

Of course! I could not find any better ones:

The packages pinnedPkgs, crossPkgs and crossStaticPkgs are always pinned.

On initialBuild (when building the zip ball) buildPkgs is equal to pinnedPkgs to ensure complete reproducabilty of the zip ball.

When using this repo as nix-channel, initialBuild is false. This means that buildPkgs is the current installed version of nixpkgs. This allows the user to update scripts to use the newest dependencies.

I'm open for better names of these attributes!

[Gerschtli]

But how would one parametrize it if it's installed through nix-env? Is there some way that I don't know of?

I think there was a way via configuring the parameters in ~/.config/nixpkgs/config.nix but I need to find the exact way to do it..

[t184256]

The packages pinnedPkgs, crossPkgs and crossStaticPkgs are always pinned.

Oh, my bad, I see now. Concerns waived.

[Gerschtli]

Given a nix expression:

with import <nixpkgs> { };

writeText "example" ''
  ${builtins.toJSON config.example}
''

And a ~/.config/nixpkgs/config.nix with:

{
  example = {
    value1 = 5;
    value2 = "test";
  };
}

The result file would contain:

{"value1":5,"value2":"test"}

Through this way, we could parameterize the derivation installed via nix-env.

[Gerschtli]

What I mean is that my phone is already compiling 10 minutes. I don't know how long it will take, but if a fresh user tries nix-on-droid, he will surely be disappointed to wait an unknown amount of time before being able to do anything useful with the app..

[t184256]

True.

But the prospect of managing everything with Nix sounds so good.

And I can see two ways of alleviating that:

1. I can periodically push to cachix.

2. (if we roll high enough on diplomacy) we can add proot-static to nixkpgs itself, so that the Hydra itself will rebuild it and push it to the official cache.

[Gerschtli]

How do you want to build proot via Hydra? You need an android device as underlying architecture if I'm not missing something.

We could use the pinned packages to build proot and update the pinned version regularly. In this way, there would not be any chance that the user would need to rebuild it locally.

[t184256]

You don't. You just need a real aarch64 device, and Hydra has them. Same logic as with all the other packages.

(provided that whatever is built now will actually work and is not some utter garbage)

We could use the pinned packages to build proot and update the pinned version regularly. In this way, there would not be any chance that the user would need to rebuild it locally.

Hey, yeah, that's also an option.

[Gerschtli]

Ah that sounds good.

I think, we should pin the nixpkgs and push the proot/talloc builds for these special versions to completely avoid rebuilds on the android phones.

Meanwhile there should me an aarch64 build device for pushing the proot/talloc builds to cachix. (I would appreciate it if you would push the builds to cachix so I can test this PR locally :D)

[Gerschtli]

BTW, I would love to refactor this mess we (I) made [1] :D I'm already confused with all the naming stuff and what parameter callPackage will inject in each file. I don't know how you feel, but this hurts :D It is definitely alright as a proof of concept, but it is getting more and more complex..

But to start refactor, we should finish the proot/talloc thing first. Otherwise this will result in big merge conflicts..

EDIT: [1]: Yes I know, I even wrote most of this stuff by myself but it was more of a hacking session as you can guess :D

[t184256]

I think, we should pin the nixpkgs and push the proot/talloc builds for these special versions to completely avoid rebuilds on the android phones.

Sounds sensible.

Meanwhile there should me an aarch64 build device for pushing the proot/talloc builds to cachix.

Will work on that after I finish fighting Android Studio and solve the broken symlinks problem.

I now have access to two devices, which will also enable me to test both wipe-reinstall and simply-rebuild scenarios =)

(I would appreciate it if you would push the builds to cachix so I can test this PR locally :D)

It should be just ~20 minutes of compiling.

But to start refactor, we should finish the proot/talloc thing first.

Not only. To land this, we need

1. the symlinks problem solved
2. proot/talloc pushed to cachix
3. proot/talloc pinned
4. some basic smoke testing to check that it's somewhat functional

and I'm not as fast as you are =) You can do 3 while I'm fighting the first two.

[t184256]

(provided that whatever is built now will actually work and is not some utter garbage)

uhh... the output is dynamically-linked. gotta fix that too

[t184256]

I'm giving up for now.

If I link proot with glibc, it fails with Bus error. Building a usable proot (i.e., linking it to bionic) on-device is too hard for me now. It is doable (termux can do that and that's how I've started in the first place), but it's complicated. I'll try to remove proot-managing commits from the branch.

[t184256]

OK, now it's your turn to review =) I've done some testing and fixed what I've found (mostly h-m installation being broken).

I've removed some of the commits regarding native proot building, so it'll be a force-push. The originals are present in a FAILED-on-device-proot-buildingbranch if you need them.

Testing requires a new version of the app, or the sessions after the first one would fail. Channel URL can be specified as a build parameter.

[Gerschtli]

Oh boy, this is really complicated.. Can you explain in words what exactly was the problem and we do you need so many cases?

AFAICT the symlinks of login (failing in the android app) and proot-static, sh and login-inner are the problem right? So the symlink blob in the ~/.nix-profile is the real source of problems, I see.. I need a moment to work this through :D

[t184256]

Ha-ha, it totally is.

Short explanation: the real files are hidden under layers and layers of symlinks. This would not be a problem at all if only those were working symlinks. They are not.

Complication 1: symlinks can point to, for example, /nix/store/123-whatever. This doesn't make sense for the outside Android system. You see such a symlink (not pointing to /data/data/com.termux.nix/files), you prepend /data/data/com.termux.nix/files/usr to it. Now it works.

Complication 2: symlinks can eventually point to /long/path/that/has/a/broken_symlink/and/something/after/it. When you have this thing, you have to iterate over it and see which one of the parents is a broken symlink, fix it, resolve it, add the remainder and continue resolving. It's bonkers.

Here login finds proot-static. There's also a Java-land implementation of this that finds login: nix-community/nix-on-droid-app@f4d6e55

You can marvel at my notes on how login is found:

    // /d/d/c.t.n/f/usr/bin/login -> /d/d/c.t.n/f/usr/nix/v/n/p/p/USERNAME/profile/login
    // /d/d/c.t.n/f/usr/nix/v/n/p/p/USERNAME/profile/login is broken
    // /d/d/c.t.n/f/usr/nix/v/n/p/p/USERNAME/profile -> profile-1-link
    // /d/d/c.t.n/f/usr/nix/v/n/p/p/USERNAME/profile-1-link -> /nix/store/...-user-environment
    // /nix/store/...-user-environment/bin
    // /d/d/c.t.n/f/usr/nix/store/...-user-environment/bin -> /nix/store/...-basic-environment/bin
    // /d/d/c.t.n/f/usr/nix/store/...-basic-environment/bin/login -> /nix/store/...-login/bin/login

or uncomment the debugging output in the shell script and see a whole screen of detective work to find proot-static.

Also, shipping login that is a broken symlink broke my tablet setup, actually, as I have an external process (Easer) that gains root, enters the Nix environment and executes an evdev/uinput keyboard remapper when I connect my Bluetooth keyboard. So, maybe one day I'll move this symlink resolver into a separate file to reuse it in other 'Android chroots into Nix' scenarios.

But hey, now we have an upgrade path that doesn't require wiping. Worth it.

[t184256]

LGTM. And this one is big enough already.

Will merge when you'll say it looks OK, but don't hesitate to voice more concerns (or reraise some that I forgot about, if there are any). I'm fine with minor commit rephrasing/reordering/fusing, otherwise I'll just merge them all as is, w/o squashing.

[Gerschtli]

LGTM too. I will be busy the next days let's merge it as it is right now and do the next things in new PRs.

Don't forget to update the default nix-channel url to master branch. And I think we should update the README according to the newest changes.

I installed it on my phone and the initial boot and hm-install work without problems.

[t184256]

Fine. Thanks a lot for your hard work.

I've updated the README. I've also removed the license header from the default config template, as I'm 99+% sure you didn't intend to enforce GPL on user config.

[Gerschtli]

Yeah, you are right, thank you!