Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linux: assorted build cleanups #187252

Merged
merged 3 commits into from Aug 20, 2022
Merged

linux: assorted build cleanups #187252

merged 3 commits into from Aug 20, 2022

Conversation

K900
Copy link
Contributor

@K900 K900 commented Aug 18, 2022

This whole thing is utterly broken, but let's start here.
Should fix intermittent build failures with patchShebangs.

Split out from #187138.

Description of changes

Stop doing horrible things to makefiles. Manually audited changes to make sure they don't actually matter.

See also: torvalds/linux@master...K900:linux:insane for a 5.19 tree with two commits: one that applies the "fixup" as-is, and one that reverts everything that produces results that are objectively wrong. The final diff is the changes that we might want to actually make, audited again to make sure that code should not be hit under normal circumstances.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

"INSTALL_PATH=$(out)"
] ++ (optional isModular "INSTALL_MOD_PATH=$(out)")
++ optional installsFirmware "INSTALL_FW_PATH=$(out)/lib/firmware"
++ optionals buildDTBs ["dtbs_install" "INSTALL_DTBS_PATH=$(out)/dtbs"];

preInstall = ''
installFlagsArray+=("-j$NIX_BUILD_CORES")

# the install scripts expect to find installkernel in ~/bin/installkernel
export HOME=${installkernel}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, this replaces INSTALLKERNEL=.... Is there a reason this is better? It feels like setting HOME because something else somewhere expects the path to be just right is very brittle. On the other hand INSTALLKERNEL=${installkernel} seems pretty explicit.

Copy link
Contributor Author

@K900 K900 Aug 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous solution only worked by accident. The upstream install scripts look for ~/bin/${INSTALLKERNEL} and /sbin/${INSTALLKERNEL}, which after the mass-replacement turned into ${HOME}${INSTALLKERNEL} and ${INSTALLKERNEL} respectively, and the latter happened to work with what we were passing in. We could patch the path directly into the install scripts, but it's declared in different locations in different kernel releases, so this is easier.

Copy link
Member

@samueldr samueldr Aug 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this a bit clearer by adding the reasoning in the comment, also linking to e.g. the current LTS's installkernel usage as an example of why this is needed? I'm betting in 1+ year the next person looking into this will be confused too :).

Note that because of legacy kernel requirements, I agree that this solution is not too bad, since it is all executing under the context of a single derivation build.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am very much hoping we will not have this in 1+ year, but sure, I'll add a comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. Better now than in 1+ year.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unremoved. The obvious thing doesn't work.

@@ -114,10 +114,6 @@ let
++ optional (lib.versionAtLeast version "5.2" && lib.versionOlder version "5.4") ./gen-kheaders-metadata.patch;

prePatch = ''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be done in postPatch because it can break upstream patches.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is intentional, but I don't understand it enough to touch it right now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure about the rest but this bit needs to be done pre

e9a1ba0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a patch changes the shebang in scripts/ld-version.sh or a line below it, it will fail to apply. A better approach would be to conditionally patch the shebang or allow it to fail.

Copy link
Member

@samueldr samueldr Aug 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not change the established semantics of what is basically out of scope of the changes. This implementation has a lot of baggage from being really ancient.

If you believe it should be done, you're invited to open a new PR aimed at cleaning up the nits from the derivation.

@K900
Copy link
Contributor Author

K900 commented Aug 19, 2022

Some more testing results: all the x86_64 kernels build, latest and LTS also build on aarch64 native and x86_64 to aarch64 cross.

This whole thing is utterly broken, but let's start here.
Should fix intermittent build failures with patchShebangs.
@K900
Copy link
Contributor Author

K900 commented Aug 19, 2022

The device tree changes might be a bit out of scope for this as well, but they cause merge conflicts otherwise. If anyone wants to specifically merge the scripts stuff and not the DT stuff, I can split it out.

@K900 K900 changed the title linux: don't try to mass clean paths linux: assorted build cleanups Aug 19, 2022
# by the makefile overlords for our own nefarious ends.
#
# Note that the makefiles specifically look in ~/bin/installkernel, and
# writeShellScriptBin writes the script to <store path>/bin/installkernel,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now this is a good comment! I appreciate you writing it.

@K900
Copy link
Contributor Author

K900 commented Aug 20, 2022

I want to merge this later tonight so I can catch any potential fallout tomorrow. Any objections?

@K900 K900 merged commit 175f140 into NixOS:master Aug 20, 2022
@vcunat
Copy link
Member

vcunat commented Aug 21, 2022

This broke build of some kernels, so I'd suggest prompt fix or reverting for now. Example: https://hydra.nixos.org/build/188038089

@K900
Copy link
Contributor Author

K900 commented Aug 21, 2022

Investigating.

@K900 K900 mentioned this pull request Aug 21, 2022
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants