Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macos big sur installer fixes #3996

Merged
merged 8 commits into from Oct 20, 2020
Merged

Conversation

@abathur
Copy link
Member

@abathur abathur commented Sep 8, 2020

Goal: make sure the installer scripts will run across macOS versions

Caveats, in case this ends up like the last one... 馃槵

  • I have tested installs with multiple macOS versions on a single system, but not on Linux.
  • This work is just focused on procedural differences necessary to install on Big Sur as seen on existing Intel hardware; it has nothing directly to do with the ARM/DTK effort. (I gather the install will fail on a DTK without a release tarball available for that architecture.)

This will also fix #3852 and fix #3957.

A few notes (repeated from a later comment):

  1. I had to fix a few macOS install breaks introduced by other commits already in master. If a new release needs to be cut from master before these changes land, those fixes will need to be cherry-picked/ported to keep macOS installs (for all versions) working correctly.
  2. After daemon installs, commands throw file 'nixpkgs' was not found in the Nix search path... errors on invocation. I don't think I caused this (maybe ec9dd9a did)? If that's correct, it'll need attention outside the scope of this PR.
@domenkozar
Copy link
Member

@domenkozar domenkozar commented Sep 8, 2020

cc @LnL7

scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
Copy link
Member

@LnL7 LnL7 left a comment

Looks pretty good to me.

scripts/install-nix-from-closure.sh Outdated Show resolved Hide resolved
scripts/install-nix-from-closure.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Show resolved Hide resolved
esac
i=$((i+1))
done
diskutil apfs list -plist "$1" | xmllint --xpath "/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict/key[text()='Name']/following-sibling::string[contains(translate(text(),'N','n'),'nix')]/text()" -
Copy link
Member

@LnL7 LnL7 Sep 8, 2020

Nice, did you check this with a few conditions?

Copy link
Member Author

@abathur abathur Sep 14, 2020

While thinking through this again I did notice and fix a pair of issues:

  • after 16d6181 it should break after the first match like the existing loop
  • after 4b42097 it should match at the beginning of the volume name only

Does that cover what you have in mind?

scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
@abathur abathur force-pushed the macos_big_sur_fixes branch 2 times, most recently from 7ccb5b0 to af6a895 Sep 13, 2020
@abathur abathur marked this pull request as ready for review Sep 14, 2020
@abathur
Copy link
Member Author

@abathur abathur commented Sep 17, 2020

@domenkozar I think this is roughly ready. Notes:

  • Despite adding support, 0baa6e7 also adds a temporary check to limit Big Sur installs in case Apple releases it before some issues related to supporting Big Sur in nixpkgs can be ironed out (requested by @LnL7).
  • Changes here probably merit a broad installer test before the next release
  • When I tested this on daemon installs, nix-commands throw file 'nixpkgs' was not found in the Nix search path... errors on invocation. I'm planning to open a separate issue for that unless you see something I've done that you think could cause it (I suspect ec9dd9a may be the source).

@Atemu
Copy link
Member

@Atemu Atemu commented Sep 28, 2020

When I tested this on daemon installs, nix-commands throw file 'nixpkgs' was not found in the Nix search path... errors on invocation.

This already happened on my recent Catalina installation. Doing step 3. from here fixes the issue.

See #2033 for more info.

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Sep 28, 2020

@abathur I'm a bit short on time to test this. However, if you generate the installer script, it should be trivial to test with github actions once 11.0 is available for testing.

@abathur
Copy link
Member Author

@abathur abathur commented Sep 28, 2020

@domenkozar I have them for the last two commits. I'll ping you on IRC.

@abathur
Copy link
Member Author

@abathur abathur commented Oct 8, 2020

If anyone wants to test this, I've hosted two darwin-x86_64 tarballs with matching stub install scripts (these scripts don't check the tarball hash).

  1. To test the actual Big Sur install, target https://files.t-ravis.com/install-allowed (which will download nix-3.0pre20200914_4b42097-x86_64-darwin.tar.xz, built from 4b42097).
  2. To test the install guards that exclude Big Sur systems until Nixpkgs support is ready, target https://files.t-ravis.com/install-blocked (which will download nix-3.0pre20200917_0baa6e7-x86_64-darwin.tar.xz, built from 0baa6e7).

For reference, I also re-ran some trivial CI jobs with these URLs.

Notes:

  • You shouldn't see any difference between the allowed/blocked installers here; the "blocked" installer should only affect Big Sur+, and travis-ci hasn't added Big Sur runners yet.
  • The macOS jobs "fail" after successfully installing because there's no nixpkgs pre-specified on the daemon installs (single-user should be OK). I'm not sure if that behavior is correct, but I'm pretty sure it's out-of-scope here.
  • All of the Linux jobs are canceled--there's no Linux installer.

@abathur
Copy link
Member Author

@abathur abathur commented Oct 15, 2020

@domenkozar GitHub Actions added Big Sur preview runners today, so I've done comparison runs with the installers above:

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Oct 15, 2020

Could we also run with the tbd branch?

@abathur abathur changed the title Macos big sur fixes Macos big sur installer fixes Oct 15, 2020
abathur added 8 commits Oct 19, 2020
Keeping this commit narrow for reviewability, but some of these
conditionals will change in subsequent commits in this PR.

Fixes NixOS#3852.
- xpath -> xmllint: xpath's cli interface changed in Big Sur
  rather than add conditional logic for picking the correct
  syntax for xpath, I'm changing to xmllint --xpath, which
  appears to be consistent across versions I've tested...

- /plist/dict/key[text()='Writable']/following-sibling::true[1]
  doesn't do quite what's expected. It was written to try to
  select a <true /> node paired with the Writable key, but it
  will also select the *next* <true /> node that appears even
  if it was paired with another key.

- I think there's also a logic bug in the conditionals here.
  I'm not sure anyone ever actuall saw it, thanks to the xpath
  bug, though. With the xpath fix, this conditional passes if /nix
  does not exist, / IS writable, and the version is Catalina+.

  I think it meant to test for /nix does not exist, / is NOT
  writable, and the version is Catalina+. I reworked this lightly
  to make it a little clearer at the code level.
As mentioned in previous commit, Big Sur changes the syntax for the
xpath command slightly.

In the process of testing out replacements for these, I noticed a few
small simplification wins.
Fixes NixOS#3957. Just runs both forms to minimize moving parts.
The move from release.nix to flake.nix appears to have lost some
changes from NixOS#3628 / 1c56f18, leaving
create-darwin-volume.sh out of the release tarball.

Under the assumption that this was just an accident/byproduct of when
flake.nix split off and not intentional, I am restoring those edits.
Some of the changes in NixOS#3788 to support non-systemd Nix installs
don't appear to be aware that the darwin installer exists, which
resulted in some skipped steps and inappropriate instructions.
Env vars for ZSH were moved from /etc/zshrc to /etc/zshenv in NixOS#3608
to address an issue with zshrc getting clobbered by OS updates, but
/etc/zshenv doesn't exist by default--so *nothing* would get set up
for zsh users unless they already happened to have /etc/zshenv.

Creating these files if they don't exist. Also cut separate creation
of profile.d/nix.sh, which isn't needed now.
@abathur abathur force-pushed the macos_big_sur_fixes branch from 0baa6e7 to f289bdb Oct 19, 2020
@abathur
Copy link
Member Author

@abathur abathur commented Oct 19, 2020

I've force-pushed a squashed history which collapses all of the incremental fix commits back down into:

  • 4 commits fixing issues directly necessary for Big Sur, which can be cherry-picked to 2.3-maintenance: 9c3dc9d 1f02b65 e736f8f fe80790
  • 3 commits fixing issues problems already present in master: 3a8699a b719f68 c40bad4
  • 1 commit which blocks install on Big Sur install, which we can eventually drop/revert depending on the status of the tbd/stdenv work: f289bdb

@domenkozar domenkozar merged commit e0ca98c into NixOS:master Oct 20, 2020
2 checks passed
@domenkozar
Copy link
Member

@domenkozar domenkozar commented Oct 20, 2020

Thanks!

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Oct 20, 2020

Cherry-picked 4 commits to 2.3-maintenance, we'll hopefully release 2.3.8 once nixpkgs is ready.

@abathur
Copy link
Member Author

@abathur abathur commented Oct 25, 2020

@domenkozar I didn't notice that 2.3.8 has already been cut or I would've done this sooner, but I ran an install test just now on my laptop with Big Sur:

 - system: `"x86_64-darwin"`
 - host os: `Darwin 20.1.0, macOS 10.16`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.8`
 - channels(abathur): `"nixpkgs-21.03pre246624.cfed29bfcb2"`
 - nixpkgs: `/Users/abathur/.nix-defexpr/channels/nixpkgs`

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Oct 25, 2020

Nice!!

@abathur abathur deleted the macos_big_sur_fixes branch Mar 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants