Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

treewide: autoPatchelfHook / fixDarwinDylibNames only on Linux / Darwin #163924

Merged
merged 4 commits into from
Mar 26, 2022

Conversation

OPNA2608
Copy link
Contributor

Description of changes

#149731 rewrote autoPatchelfHook in a way where it won't silently skip over non-ELF binaries anymore. I assume that wasn't an intended change, but it's prolly better to fix some packages so they don't even pull in this hook when it's not needed.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@OPNA2608
Copy link
Contributor Author

Note that dmd will still fail with this, but at a different point in the build. That failure wasn't introduced by the autoPatchelfHook rewrite and affects master too, it will be fixed by #163989.

@symphorien
Copy link
Member

maybe you can add meta.platforms = linux to autoPatchelfHook so that forgetting to make it conditional results in an explicit error?

@OPNA2608
Copy link
Contributor Author

None of the other setup hook derivations (i.e. fixDarwinDylibNames, but it won't fail on Linux anyway) do it and makeSetupHook doesn't seem to support meta afaict. What would be the best way of doing that?

  autoPatchelfHook = (makeSetupHook {
    name = "auto-patchelf-hook";
    deps = [ bintools ];
    substitutions = {
      pythonInterpreter = "${python3.withPackages (ps: [ ps.pyelftools ])}/bin/python";
      autoPatchelfScript = ../build-support/setup-hooks/auto-patchelf.py;
    };
  } ../build-support/setup-hooks/auto-patchelf.sh).overrideAttrs (oa: {
    meta.platforms = lib.platforms.linux;
  });

Works but maybe there's a better way?

@symphorien
Copy link
Member

None of the other setup hook derivations (i.e. fixDarwinDylibNames, but it won't fail on Linux anyway) do it

If you see a value in modifying derivations to not use autoPatchelfHook on darwin, then it must fail loudly when not fixed otherwise it will bitrot.

Works but maybe there's a better way?

The way you found look good enough to me, but modifying makeSetupHook to optionally take meta as argument would be better.

@OPNA2608
Copy link
Contributor Author

OPNA2608 commented Mar 13, 2022

If you see a value in modifying derivations to not use autoPatchelfHook on darwin, then it must fail loudly when not fixed

Well, it already fails quite loudly. That's what I'm trying to fix here:

Bildschirmfoto von 2022-03-13 23-01-59

I guess nobody bothered with testing that PR on Darwin, so it wasn't noticed.

@OPNA2608 OPNA2608 force-pushed the fix/autoPatchelfHook_isLinux branch from 482bfcd to 86a2688 Compare March 15, 2022 22:17
@OPNA2608 OPNA2608 changed the base branch from staging-next to master March 15, 2022 22:17
@symphorien
Copy link
Member

Ah ok, weird that nobody noticed it then.
Setting meta.platforms would still improve the error message, but I leave it up to you.

@siraben
Copy link
Member

siraben commented Mar 22, 2022

Result of nixpkgs-review pr 163924 run on aarch64-darwin 1

22 packages marked as broken and skipped:
  • bazel-watcher
  • bazel_0_29
  • bazel_1
  • libtensorflow
  • python39Packages.augmax
  • python39Packages.baselines
  • python39Packages.dm-haiku
  • python39Packages.edward
  • python39Packages.einops
  • python39Packages.elegy
  • python39Packages.flax
  • python39Packages.optax
  • python39Packages.optuna
  • python39Packages.rl-coach
  • python39Packages.scikit-tda
  • python39Packages.tensorflow
  • python39Packages.tensorflow-datasets
  • python39Packages.tensorflow-probability
  • python39Packages.tensorflowWithoutCuda
  • python39Packages.tflearn
  • python39Packages.treex
  • python39Packages.umap-learn
2 packages failed to build:
  • bazel
  • gtkd
10 packages built:
  • Literate
  • bazel_4
  • dtools
  • dub
  • elasticsearch
  • ldc
  • mspds-bin
  • powershell
  • rund
  • ssm-session-manager-plugin

@OPNA2608
Copy link
Contributor Author

I was hoping to keep the changes small to get them into staging-next before the merge, but I suppose we can aim abit bigger now that that's over.

@symphorien We can do that, I guess there usually isn't much use to trying to patch ELF files for a Darwin target. For consistency's sake I'll do the same to fixDarwinDylibNames as well then.

@OPNA2608 OPNA2608 marked this pull request as draft March 22, 2022 17:19
After being rewritten in NixOS#149731, this hook
can fail on Mach-O binaries. Since patching ELF files on Darwin doesn't make
much sense anyway, we'll mark this as Linux-exclusive.
Copy link
Member

@siraben siraben left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coq also builds. Not sure what the previous errors were caused by.

@siraben siraben merged commit ed86344 into NixOS:staging Mar 26, 2022
@kubukoz
Copy link
Member

kubukoz commented Mar 29, 2022

Just wondering, what's the next step to get this into master?

@OPNA2608
Copy link
Contributor Author

The next step is for this to get picked up by a staging-next iteration. It missed the current one, so it'll happen in the next one. After that it'll get merged into master, and from there into the various channels: https://nixpk.gs/pr-tracker.html?pr=163924

@kubukoz
Copy link
Member

kubukoz commented Mar 29, 2022

lovely, thanks. I'm following master in my flakes so that point will be good enough for me :)

@kubukoz kubukoz mentioned this pull request Apr 1, 2022
13 tasks
@kubukoz kubukoz mentioned this pull request Apr 7, 2022
13 tasks
jakubgs added a commit to status-im/nixpkgs that referenced this pull request May 13, 2022
This is supposed to fix an issue caused by this PR:
NixOS#163924

Which made `autoPatchelfHook` available only on Linux, resulting in
builds of Android packages failing with:
```
error: Package ‘auto-patchelf-hook’ in /nix/store/...-nixpkgs-source/pkgs/build-support/trivial-builders.nix:73
    is not supported on ‘x86_64-darwin’, refusing to evaluate.
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request May 26, 2022
A PR changing behavior of `auto-patchelf-hook` on Linux broke Android SDK builds:

* NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:

* NixOS/nixpkgs#173376

But I need to test it properly with out builds first.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request May 26, 2022
A PR changing behavior of `auto-patchelf-hook` on Linux broke Android SDK builds:

* NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:

* NixOS/nixpkgs#173376

But I need to test it properly with out builds first.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request May 26, 2022
A PR changing behavior of `auto-patchelf-hook` on Linux broke Android SDK builds:

* NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:

* NixOS/nixpkgs#173376

But I need to test it properly with out builds first.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request May 27, 2022
A PR changing behavior of `auto-patchelf-hook` on Linux broke Android SDK builds:

* NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:

* NixOS/nixpkgs#173376

But I need to test it properly with out builds first.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
AndersonTorres pushed a commit that referenced this pull request May 30, 2022
This is supposed to fix an issue caused by this PR:
#163924

Which made `autoPatchelfHook` available only on Linux, resulting in
builds of Android packages failing with:
```
error: Package ‘auto-patchelf-hook’ in /nix/store/...-nixpkgs-source/pkgs/build-support/trivial-builders.nix:73
    is not supported on ‘x86_64-darwin’, refusing to evaluate.
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request Jun 2, 2022
Notable version changes:

- Coreutils `9.0` to `9.1`
- NodeJS `16.14.2` to `16.15.1`
- Clojure `1.11.1.1107` to `1.11.1.1113`
- Git `2.35.1` to `2.36.1`
- Curl `7.82.0` to `7.83.1`
- Android SDK Platform Tools `31.0.3` to `33.0.1`

Most important is the Coreutils upgrade to 9.1 which includes a fix for
iOS builds on new M1 ARM64 processors:
#12799

Also fixes broken Android SDK builds on Linux due to `auto-patchelf-hook` change:
NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:
NixOS/nixpkgs#173376

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request Jun 2, 2022
Notable version changes:

- Coreutils `9.0` to `9.1`
- OpenSSL `1.1.1n` to `1.1.1o`
- NodeJS `16.14.2` to `16.15.1`
- Clojure `1.11.1.1107` to `1.11.1.1113`
- Ruby `2.7.5p203` to `2.7.6p219`
- Cocoapods `1.11.0` to `1.11.3`
- Git `2.35.1` to `2.36.1`
- Curl `7.82.0` to `7.83.1`
- Android SDK Platform Tools `31.0.3` to `33.0.1`

Most important is the Coreutils upgrade to 9.1 which includes a fix for
iOS builds on new M1 ARM64 processors:
#12799

Also fixes broken Android SDK builds on Linux due to `auto-patchelf-hook` change:
NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:
NixOS/nixpkgs#173376

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request Jun 3, 2022
Notable version changes:

- Coreutils `9.0` to `9.1`
- OpenSSL `1.1.1n` to `1.1.1o`
- NodeJS `16.14.2` to `16.15.1`
- Clojure `1.11.1.1107` to `1.11.1.1113`
- Ruby `2.7.5p203` to `2.7.6p219`
- Cocoapods `1.11.0` to `1.11.3`
- Git `2.35.1` to `2.36.1`
- Curl `7.82.0` to `7.83.1`
- Android SDK Platform Tools `31.0.3` to `33.0.1`

Most important is the Coreutils upgrade to 9.1 which includes a fix for
iOS builds on new M1 ARM64 processors:
#12799

Also fixes broken Android SDK builds on Linux due to `auto-patchelf-hook` change:
NixOS/nixpkgs#163924

I've fixed this in `nixpkgs` PR:
NixOS/nixpkgs#173376

Signed-off-by: Jakub Sokołowski <jakub@status.im>
@OPNA2608 OPNA2608 deleted the fix/autoPatchelfHook_isLinux branch September 27, 2022 18:03
@ncfavier ncfavier mentioned this pull request Jun 8, 2023
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants