Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert grub update, grub 2.04 -> 2.02 #85704

Closed
wants to merge 2 commits into from

Conversation

@Infinisil
Copy link
Member

Infinisil commented Apr 21, 2020

Motivation for this change

This was already done for 19.09 (see 862f05c and parent) since Grub 2.04 seemed to cause problems for some people: #61718. This revert apparently never went into master though, so both 20.03 and master currently still use Grub 2.04. @emptyflask reported that they got hit by this issue when updating to 20.03. So let's revert it until we know more about it before we break more boot loaders.

Needs to be backported.

Ping @gnidorah @worldofpeace @disassembler @lheckemann

lheckemann added 2 commits Oct 4, 2019
This reverts commit 8ba94a8.

See #61718 for rationale.

(cherry picked from commit 4eb9725)
This reverts commit df4d0fa.

See #61718 for rationale.

(cherry picked from commit 862f05c)
@Infinisil Infinisil changed the title Revert grub update Revert grub update from 2.04 to 2.02 Apr 21, 2020
@Infinisil
Copy link
Member Author

Infinisil commented Apr 21, 2020

@GrahamcOfBorg test installer.simple installer.simpleSpecialized installer.simpleUefiGrub installer.simpleUefiGrubSpecialisation installer.separateBoot installer.separateBootFat installer.zfsroot installer.lvm installer.luksroot installer.encryptedFSWithKeyfile installer.swraid installer.simpleLabels installer.simpleProvided installer.btrfsSimple

@Infinisil Infinisil changed the title Revert grub update from 2.04 to 2.02 Revert grub update, grub 2.04 -> 2.02 Apr 21, 2020
@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite. So for that reason I'm inclined that we close this PR and have an actual solution, because we can't continue to revert it on stable releases. That is actually why it only happened on the release branch last time.

@Infinisil
Copy link
Member Author

Infinisil commented Apr 21, 2020

If somebody has a fix for it, or knows how to get to one, that would be better of course. But this is a hardware problem that's hard to reproduce and causes really bad consequences for the ones who encounter them. If the boot loader fails to run, those people are essentially locked out of their system, no way to even roll back, since that depends on the boot loader being functional.

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

@Infinisil I meant the issue from last time is a revert was done, but during the period of the revert no proactive action was taken. Part of that is, as you said, it's hardware specific.

Someone GRUB knowledgeable needs to somehow forward a bug to them https://www.gnu.org/software/grub/manual/grub/html_node/Reporting-bugs.html https://savannah.gnu.org/bugs/?group=grub. Which is hard when it seems to be so specific.

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

Is there other alternatives than we can do aside from a full revert #61718 (comment)

Though I think stateVersion is less reliable because people mess with it.

@lheckemann
Copy link
Member

lheckemann commented Apr 21, 2020

@Infinisil I intentionally reverted it only on 19.09 so that the update would remain on master and could be fixed later on.

Idea: this could be a little sillier than we thought. I've often had issues with my ESP being corrupted simply by rebooting too quickly (or too brutally :/ ) after a nixos-rebuild thanks to FAT's lack of journalling. Maybe it's actually a case of that?

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

Idea: this could be a little sillier than we thought. I've often had issues with my ESP being corrupted simply by rebooting too quickly (or too brutally :/ ) after a nixos-rebuild thanks to FAT's lack of journalling. Maybe it's actually a case of that?

tbh, sometimes I have brutal issues that involve nixos-rebuild as well 😁

@asymmetric
Copy link
Contributor

asymmetric commented Apr 21, 2020

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite.

@worldofpeace why can't we stick to 2.02 indefinitely? (or until the bug is resolved upstream)

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite.

@worldofpeace why can't we stick to 2.02 indefinitely? (or until the bug is resolved upstream)

Because that's just a terrible practice to do in any infrastructure. Just not use something indefinitely because it has a bug. I hope that doesn't come off too negative 😹 but I see this to be true.

@kolbycrouch
Copy link

kolbycrouch commented Apr 21, 2020

as a side note: grub 2.02 doesn't support reading from f2fs. Anyone who installed nixos with f2fs as / would no longer be able to boot.

@asymmetric
Copy link
Contributor

asymmetric commented Apr 21, 2020

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite.

@worldofpeace why can't we stick to 2.02 indefinitely? (or until the bug is resolved upstream)

Because that's just a terrible practice to do in any infrastructure. Just not use something indefinitely because it has a bug. I hope that doesn't come off too negative 😹 but I see this to be true.

Well I would rephrase it as "not use something until it has a bug".

The issue seems to be with upstream, so as you correctly point out, it should be resolved there. But until that's done, should we use the buggy version?

All that aside, thanks so much for your work on 20.03! ❤️

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

as a side note: grub 2.02 doesn't support reading from f2fs. Anyone who installed nixos with f2fs as / would no longer be able to boot.

Yeah, that's problematic. Being that it's Grub, I don't see a downgrade being straightforward. We could patch it in.

@kolbycrouch
Copy link

kolbycrouch commented Apr 21, 2020

My last 2 cents: AFAIK grub 2.04 is also needed to boot from a btrfs / that uses zstd compression.

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

My last 2 cents: AFAIK grub 2.04 is also needed to boot from a btrfs / that uses zstd compression.

Reviewing the News and changes https://git.savannah.gnu.org/cgit/grub.git/tree/NEWS#n1, we can quickly see that backwards compatibility isn't going to be a thing here. Lost features.

@worldofpeace
Copy link
Member

worldofpeace commented Apr 21, 2020

So reverting will probably break someone's setup, and not reverting may have someone jump into #61718. I'm not sure there's a happy medium here.

@panchaea
Copy link

panchaea commented Apr 22, 2020

Chiming in to add that this issue is affecting one of my machines as well.

I understand why reverting the update on master is out of the question at this point, but would it be possible to do so just for 20.03, as was done for 19.09? I don't imagine it would break many people's setups then.

@asymmetric
Copy link
Contributor

asymmetric commented Apr 22, 2020

This comment links to a Debian bug report explaining the issue in detail.

It seems the underlying problem is not in upstream, but rather a misconfiguration on the user's machine.

Given that, I think the best thing to do is add an entry to the release notes pointing to instructions on how to fix this manually, rather than reverting.

@worldofpeace
Copy link
Member

worldofpeace commented Apr 22, 2020

I understand why reverting the update on master is out of the question at this point, but would it be possible to do so just for 20.03, as was done for 19.09? I don't imagine it would break many people's setups then.

As mentioned, 2.04 and 2.02 have different features, and people probably already depend on them since it's committed. So this would be a breaking change to commit to an already released version of NixOS.

This comment links to a Debian bug report explaining the issue in detail.

It seems the underlying problem is not in upstream, but rather a misconfiguration on the user's machine.

Given that, I think the best thing to do is add an entry to the release notes pointing to instructions on how to fix this manually, rather than reverting.

Woah yeah, that looks about right. In the go/no-go it was discussed to do some sort of errata, it's just the information wasn't in the issue to make it.

@misuzu
Copy link
Contributor

misuzu commented Apr 22, 2020

If this is a configuration issue maybe we can add some asserts to grub module to prevent it?

@asymmetric
Copy link
Contributor

asymmetric commented Apr 23, 2020

@misuzu what asserts would you add?

At least some of the misconfigurations involve settings in the firmware, which AFAIK the OS has no access to.

@misuzu
Copy link
Contributor

misuzu commented Apr 23, 2020

@misuzu what asserts would you add?

First step would be to gather info about systems which affected by this issue: boot configuration, partition layouts of all disks, mount points, etc. Maybe certain pattern is there and can be easily prevented. If not, at least there would be clear reproduction steps which can be documented.

@asymmetric
Copy link
Contributor

asymmetric commented Apr 24, 2020

It would definitely have been nice if the module had informed me that compress=zstd isn't supported on grub 2.02 - would have saved me some hours of stress!

@Infinisil Infinisil closed this May 14, 2020
@Infinisil Infinisil deleted the Infinisil:revert-grub-update branch May 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants
You can’t perform that action at this time.