New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Installation media) Bootloader artwork refresh #33686

Merged
merged 3 commits into from Aug 29, 2018

Conversation

@samueldr
Member

samueldr commented Jan 10, 2018

Instructions for testing this PR by making a disk image: #33686 (comment)

See PR #26 of nixos-artwork

Motivation for this change

The current artwork is old, wait that's not an issue, but it does use the older NixOS snowflake logo!

This PR is more of a technical workout of the details needed to review bootloader artwork. I am not an artist, this is why this is a sober refresh, using existing bits.

This PR is geared towards the installation media, I don't intend to change defaults or prevent choices for the end-user installation.

Advantages
  • The GRUB menu on HiDPI screens should be more readable.

Goals

  1. Get this in 18.03 ☺

Discussion points

1. ISOLINUX

ISOLINUX setup is a given, I don't think there's anything to review really. I have upgraded the default resolution to 800x600. ISOLINUX will fallback to text mode if it cannot change to 800x600.

On a technical note, I have chosen to keep to 4:3 aspect ratio since this will probably be much less frequently used on real hardware with 16:9 aspect ratio. The appearance targets mainly the VirtualBox use-case. Furthermore, seeing stretched 4:3 on 16:9 will be par-for-the-course for the users that will see it on real hardware.

  • ISOLINUX font can't be changed (afaik)
2. systemd-boot (or lack thereof)

systemd-boot Does not have any themeing capabilities. This is the most contentious bit of this PR: we need to move off of systemd-boot to prettify EFI boot. This saddens me because I have no ill intention against systemd-boot, it seems to do its job perfectly, it just can't do the pretty pictures.

This is why there are the next points

3. New UEFI bootloader for the installer image

This is something that will need to be evaluated with great care. Here are the solutions I know of.

3.1. rEFInd

rEFInd will not be proposed for this change, read on for the initial explanation, then why it won't be used.

rEFInd is not a newcomer, it's a bootloader that I would like to think is a relatively underappreciated bootloader in the linux world. It has some good (but not great) themeing capabilities, as demonstrated by the proof-of-concept integration in the current PR.

It was marked as broken, I have fixed the package. This is something that can be fast-tracked to master ASAP (I'll look into it). has been fixed in master.

One disadvantage with refind is that it cannot (yet) be used as your general purpose EFI bootloader for your day-to-day usage NixOS experience. This may (unlikely) cause some expectation that refind can be used after the installation process.

Advantages of refind are that it has an excellent capability to scan existing volumes to allow booting from them. This could allow a NixOSinstaller image to boot an EFI bootloader present on the volume in case something happened. (This is currently not configured to work.) I'm giving a strong vote towards using refind for this specific reason: it may save your bacon one day.

  • rEFInd font has to be monospaced, limitation from rEFInd.

Verdict

Since it cannot be relied on for serial access, rEFInd will not be used.

I will, though, still include it in the live image. rEFInd could be an additional option in the bootloader menu, unthemed, and used for its good capabilities in scanning volumes to find bootable EFI programs. It should thus be documented somewhere.

3.2. grub

Nothing yet has been done, or worked on, but that's my next step: get something great for grub2. This means two things:

  • Usable background for the "simple" grub2 setup just like right now
  • More complex setup with a full-blown grub2 theme

Both of those will have to be done even if refind is used. Why do both? The first is an easy target, it will probably be much easier to get into 18.03. Furthermore, it will be necessary to keep the current "simple" way grub themeing work with nixos. Then, I'll need to look into properly getting themes working.

This options is probably safer, which I mean that the NixOS team probably has a much better understanding of how grub works. Grub would be used in the installer image only for the EFI boot. I'm pretty sure changing from ISOLINUX to grub would be a risky move, as bios boot is a bit more fragile. I think (haven't verified) that the hybrid iso image is a feature of isolinux anyway.

Grub configuration has been done.

  • Background image support in grub2, with default textual menu
  • Full theme for grub2

What's it look like?

The important part!

ISOLINUX

20180109214412

rEFInd
Show unused rEFInd theme

Main screen (with timeout)

20180109214217

Submenu (F2 pressed)

20180109214228

GNU GRUB

grub2-theme

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
Things to be done
  • Squash WIP commits intelligently

Fixes #22889

@samueldr samueldr referenced this pull request Jan 10, 2018

Merged

Bootloader artwork refresh #26

@samueldr samueldr changed the title from WIP: Bootloader artwork refresh to WIP: (Installation media) Bootloader artwork refresh Jan 10, 2018

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Jan 10, 2018

Member

From #nixos:

[10:06:50] <andi-> Does refit work with serial consoles?

@andir has a good point, we can't move to a bootloader that can't support serial. It should support serial, but it seems the author isn't making it a priority to ensure it works.

I think this means that grub is a necessity for this purpose.

Member

samueldr commented Jan 10, 2018

From #nixos:

[10:06:50] <andi-> Does refit work with serial consoles?

@andir has a good point, we can't move to a bootloader that can't support serial. It should support serial, but it seems the author isn't making it a priority to ensure it works.

I think this means that grub is a necessity for this purpose.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Jan 11, 2018

Member

Update ♯1

  • systemd-boot has been (temporarily) put back in place.
  • rEFInd has been added to the image, bootable from systemd-boot's interface (explanations follow).

rEFInd from systemd-boot?

This is redundant for 99% of the installer image users, but this might help some users with buggy EFI implementations. I know I would have loved it in two separate occasions last year.

The specific scenario I know of is that some buggy EFI implementation can lose their configuration. I have two separate computers with distinct EFI implementation that have this flaw, (exhibited in different ways). In both cases, it is made worse in not being able to manually add a path to the EFI boot options from the firmware interface. With the default NixOS setup, it makes both those computers unbootable from the existing volumes. (Both implementation will boot the default ESP:/EFI/boot/boot{arch}.efi efi programs.) With rEFInd available on the installer image, this would have allowed me to plug in the drive, select the rEFInd option; rEFInd scans the volumes in the computer, finds the efi bootloader (in my case grubx64.efi) and allows me to select then boot it. This may also help a user boot NixOS if some other rude operating system decides to make itself the default option on EFI setup (and somehow break the NixOS entry).

Total cost of adding rEFInd:

  • A new entry in the boot menu (negligible)
  • 277K in the the installer image (small!)
  • Some maintenance of the rEFInd package in nixpkgs (keeping it buildable).
Member

samueldr commented Jan 11, 2018

Update ♯1

  • systemd-boot has been (temporarily) put back in place.
  • rEFInd has been added to the image, bootable from systemd-boot's interface (explanations follow).

rEFInd from systemd-boot?

This is redundant for 99% of the installer image users, but this might help some users with buggy EFI implementations. I know I would have loved it in two separate occasions last year.

The specific scenario I know of is that some buggy EFI implementation can lose their configuration. I have two separate computers with distinct EFI implementation that have this flaw, (exhibited in different ways). In both cases, it is made worse in not being able to manually add a path to the EFI boot options from the firmware interface. With the default NixOS setup, it makes both those computers unbootable from the existing volumes. (Both implementation will boot the default ESP:/EFI/boot/boot{arch}.efi efi programs.) With rEFInd available on the installer image, this would have allowed me to plug in the drive, select the rEFInd option; rEFInd scans the volumes in the computer, finds the efi bootloader (in my case grubx64.efi) and allows me to select then boot it. This may also help a user boot NixOS if some other rude operating system decides to make itself the default option on EFI setup (and somehow break the NixOS entry).

Total cost of adding rEFInd:

  • A new entry in the boot menu (negligible)
  • 277K in the the installer image (small!)
  • Some maintenance of the rEFInd package in nixpkgs (keeping it buildable).
@andir

This comment has been minimized.

Show comment
Hide comment
@andir

andir Jan 11, 2018

Member

Are we somehow able to write an nixos test that verifies that our boot images actually work (on QEMU at least)?

Member

andir commented Jan 11, 2018

Are we somehow able to write an nixos test that verifies that our boot images actually work (on QEMU at least)?

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Jan 11, 2018

Member

Looks like there is or there was a suite of tests for the minimal iso. I'm not familiar enough with hydra and tests to confirm whether or not it would be in use anymore. It may need to be reviewed.

Member

samueldr commented Jan 11, 2018

Looks like there is or there was a suite of tests for the minimal iso. I'm not familiar enough with hydra and tests to confirm whether or not it would be in use anymore. It may need to be reviewed.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Feb 4, 2018

Member

Update ♯2

  • added grub2 support.
  • added grub2 theme.

Grub theme has been made. There is also support into the module to not use the theme and instead use a simpler background and text menu, as is currently used by default when using grub2.

The grub2 theme engine cannot render with subpixels, which is why the text is... not that great. Though the text rendering isn't the best, having the ability to render bigger text is a great help for Higher DPI devices.

Next tasks:

  • Ensure it all works using the existing tests. Tests <nixpkgs/nixos/tests/boot.nix> ran successfully.
  • Real-world tests (Test on all my machines)
Member

samueldr commented Feb 4, 2018

Update ♯2

  • added grub2 support.
  • added grub2 theme.

Grub theme has been made. There is also support into the module to not use the theme and instead use a simpler background and text menu, as is currently used by default when using grub2.

The grub2 theme engine cannot render with subpixels, which is why the text is... not that great. Though the text rendering isn't the best, having the ability to render bigger text is a great help for Higher DPI devices.

Next tasks:

  • Ensure it all works using the existing tests. Tests <nixpkgs/nixos/tests/boot.nix> ran successfully.
  • Real-world tests (Test on all my machines)

@samueldr samueldr changed the title from WIP: (Installation media) Bootloader artwork refresh to (Installation media) Bootloader artwork refresh Feb 4, 2018

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Feb 5, 2018

Member

Update ♯3

p_20180204_140735

Photo of different devices with the new bootloader configuration. (Disregard stretched out menu, side-effect of using the boot device menu in the poor UEFI implementation)

Hardware testing has been conclusive. It works on all of my UEFI devices. (Not all are pictured.)


  • VirtualBox VM
  • OVMF qemu
  • Acer c720p with Tianocore-based alternative firmware
  • ASUS Transformer Book, TP300-LA
  • CHUWI Hi10 Pro (HQ64) (CherryTrail reference platform afaik)
  • HP z420
Member

samueldr commented Feb 5, 2018

Update ♯3

p_20180204_140735

Photo of different devices with the new bootloader configuration. (Disregard stretched out menu, side-effect of using the boot device menu in the poor UEFI implementation)

Hardware testing has been conclusive. It works on all of my UEFI devices. (Not all are pictured.)


  • VirtualBox VM
  • OVMF qemu
  • Acer c720p with Tianocore-based alternative firmware
  • ASUS Transformer Book, TP300-LA
  • CHUWI Hi10 Pro (HQ64) (CherryTrail reference platform afaik)
  • HP z420
@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Feb 6, 2018

Member

Update ♯4

20180205215541

I have re-worked the grub menuentry generation to allow building new options while not repeating the code. This makes it possible to add a submenu with entries. I have added two demo entries that add video=_____ command line parameters to force the KMS resolution. It should be possible, also, to add different entries for different behaviours as needed. (E.g. black on white text, activate other accessibility options.)

Member

samueldr commented Feb 6, 2018

Update ♯4

20180205215541

I have re-worked the grub menuentry generation to allow building new options while not repeating the code. This makes it possible to add a submenu with entries. I have added two demo entries that add video=_____ command line parameters to force the KMS resolution. It should be possible, also, to add different entries for different behaviours as needed. (E.g. black on white text, activate other accessibility options.)

@FRidh FRidh added this to the 18.03 milestone Feb 10, 2018

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Mar 3, 2018

Member

The branch has been rebased on nixos-unstable

Member

samueldr commented Mar 3, 2018

The branch has been rebased on nixos-unstable

@vcunat vcunat modified the milestones: 18.03, 18.09 Apr 2, 2018

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 May 9, 2018

Member

Can we move this along? It does look much nicer.

Member

Ericson2314 commented May 9, 2018

Can we move this along? It does look much nicer.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr May 10, 2018

Member

Freshly rebased off nixos-unstable. Verified to work exactly as it did.

Member

samueldr commented May 10, 2018

Freshly rebased off nixos-unstable. Verified to work exactly as it did.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr May 13, 2018

Member

I have updated this PR with tested and working serial for grub.

This has been tested only on qemu, as I do not have the hardware to test.

This has been tested with these qemu options:

  • (none) → nothing special, no serial option
  • -serial mon:stdio → serial
  • -nographic -serial mon:stdio → serial, but no graphics

In all cases, when it made sense, serial worked, graphics worked.

Fun fact, the serial and graphics menus are synchronised! This means that selecting an option in the serial console will be visible on the screen, and selecting an option in the screen will be visible on the console.


Additionally, a new option in the Quirks menu has been added for serial. This, too, has been tested with qemu, and it works.

Member

samueldr commented May 13, 2018

I have updated this PR with tested and working serial for grub.

This has been tested only on qemu, as I do not have the hardware to test.

This has been tested with these qemu options:

  • (none) → nothing special, no serial option
  • -serial mon:stdio → serial
  • -nographic -serial mon:stdio → serial, but no graphics

In all cases, when it made sense, serial worked, graphics worked.

Fun fact, the serial and graphics menus are synchronised! This means that selecting an option in the serial console will be visible on the screen, and selecting an option in the screen will be visible on the console.


Additionally, a new option in the Quirks menu has been added for serial. This, too, has been tested with qemu, and it works.

@pSub

This comment has been minimized.

Show comment
Hide comment
@pSub

pSub Jul 15, 2018

Member

Looks great! Is there something left to do before this can be merged?

Member

pSub commented Jul 15, 2018

Looks great! Is there something left to do before this can be merged?

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Jul 15, 2018

Member

Nothing much really, though I'd love it if people could test on real-world hardware especially if they have picky hardware.

Here's a list of things that may be good to check:


HiDPI

Both checking how GRUB looks, and the hidpi helper options in the accessibility menus.

  • HiDPI options on modern laptops like the fancy new XPSes
  • HiDPI options on modern macbooks (those already supported by grub)

Serial

  • Real hardware serial for UEFI

Weird hardware?

  • Had issues with some bootloaders? please try!

Though, I don't really have any hopes to see them all tested, I mean, I'd like to have a head count of everyone that booted it on real hardware, then a head count for those that booted it on MBR and then UEFI virtual machines.

So, it's probably good to be merged, if the code quality here is fine.

Member

samueldr commented Jul 15, 2018

Nothing much really, though I'd love it if people could test on real-world hardware especially if they have picky hardware.

Here's a list of things that may be good to check:


HiDPI

Both checking how GRUB looks, and the hidpi helper options in the accessibility menus.

  • HiDPI options on modern laptops like the fancy new XPSes
  • HiDPI options on modern macbooks (those already supported by grub)

Serial

  • Real hardware serial for UEFI

Weird hardware?

  • Had issues with some bootloaders? please try!

Though, I don't really have any hopes to see them all tested, I mean, I'd like to have a head count of everyone that booted it on real hardware, then a head count for those that booted it on MBR and then UEFI virtual machines.

So, it's probably good to be merged, if the code quality here is fine.

@flokli

This comment has been minimized.

Show comment
Hide comment
@flokli

flokli Jul 16, 2018

Contributor

@samueldr tested on a T460p (2560x1440px). Grub itself is a bit slow (in terms of screen updates), but that's the same on my local grub as well.
Tested rotation and HiDPI options, worked (at stage 2).
Couldn't check serial of course ;-)

Contributor

flokli commented Jul 16, 2018

@samueldr tested on a T460p (2560x1440px). Grub itself is a bit slow (in terms of screen updates), but that's the same on my local grub as well.
Tested rotation and HiDPI options, worked (at stage 2).
Couldn't check serial of course ;-)

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 10, 2018

Member

I merged my nixos-artwork PR, made the review as needed for the grub2 theme. This is ready. I also freshly rebased ans squashed into more logical commits.

I don't want to have to press the merge button myself, would love to have someone else validate the iso works for them too.!

I don't think it was worth the effort to split the big grub2 commit into smaller parts; it's all interconnected.

Member

samueldr commented Aug 10, 2018

I merged my nixos-artwork PR, made the review as needed for the grub2 theme. This is ready. I also freshly rebased ans squashed into more logical commits.

I don't want to have to press the merge button myself, would love to have someone else validate the iso works for them too.!

I don't think it was worth the effort to split the big grub2 commit into smaller parts; it's all interconnected.

@LnL7

This comment has been minimized.

Show comment
Hide comment
@LnL7

LnL7 Aug 12, 2018

Contributor

I just tested this on a macbook, the boot menu looks great there and the HiDPI options seem work as expected.

Contributor

LnL7 commented Aug 12, 2018

I just tested this on a macbook, the boot menu looks great there and the HiDPI options seem work as expected.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 12, 2018

Member

A bit of testing was done by two members of the community

The first end-user tests hit a couple bumps from the later changes:

  • No serial interface would kind-of break the menu (but it would actually still work)

This wasn't considered at first, because it worked here, but turns out grub sees serial interfaces on my test hardware devices, and sees one with the default qemu settings! You can use -serial none to test this.

This has been fixed.

The other main pain point would be how the menu can look on really-low resolutions, like some towers/workstations can show. This has been fixed by reducing the padding (in the last PR to nixos-artwork).

The changes don't need this to be merged before this one, I'm keeping it open until this is merged, in case there are other changes; the release tarball for the theme is already fixed to serve the right files.

With the newly-done user-testing, and having had an actual refresh of its state by fixing those small issues, I'd say LGTM.

Member

samueldr commented Aug 12, 2018

A bit of testing was done by two members of the community

The first end-user tests hit a couple bumps from the later changes:

  • No serial interface would kind-of break the menu (but it would actually still work)

This wasn't considered at first, because it worked here, but turns out grub sees serial interfaces on my test hardware devices, and sees one with the default qemu settings! You can use -serial none to test this.

This has been fixed.

The other main pain point would be how the menu can look on really-low resolutions, like some towers/workstations can show. This has been fixed by reducing the padding (in the last PR to nixos-artwork).

The changes don't need this to be merged before this one, I'm keeping it open until this is merged, in case there are other changes; the release tarball for the theme is already fixed to serve the right files.

With the newly-done user-testing, and having had an actual refresh of its state by fixing those small issues, I'd say LGTM.

samueldr added some commits Jan 8, 2018

Adds refind to the installer image.
This is a 277K (as of right now) addition that can greatly help in some
last recourse scenarios. The specific rEFInd setup will not be able to
boot the installer image, but this is not why it has been added. It has
been added to make use of its volumes scanning capabilities to boot
existing EFI images on the target computer, which is sometimes necessary
with buggy EFI. While is isn't NixOS's job to fix buggy EFI, shipping
this small bit with the installer will help the unlucky few.

Example scenario: two wildly different EFI implementation I have
encountered have fatal flaws in which they sometimes will lose all the
settings, this includes boot configuration. This is compounded by the
fact that the two specific and distinct implementation do not allow
manually adding ESP paths from their interface. The only recourse is to
let the EFI boot the default paths, EFI/boot/boot{platform}.efi, which
is not a default location used by the NixOS bootloaders. rEFInd is able
to scan the volumes and detect the existing efi bootloaders, and boot
them successfully.
Use a themed grub for the installer image
This replaces systemd-boot with grub, it is at feature parity, as in it
can do everything systemd-boot did in the previous commit.
@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 24, 2018

Member

Leaving this note here if anyone needs it.

Building the image

Checkout this branch, then at the root of nixpkgs

 $ nix-build nixos/release.nix -A iso_graphical
[...]
 $ ls -l result/iso/*.iso
-r--r--r-- 1 root root 1172307968 Dec 31  1969 result/iso/nixos-graphical-18.09pre130979.gfedcba-x86_64-linux.iso
Member

samueldr commented Aug 24, 2018

Leaving this note here if anyone needs it.

Building the image

Checkout this branch, then at the root of nixpkgs

 $ nix-build nixos/release.nix -A iso_graphical
[...]
 $ ls -l result/iso/*.iso
-r--r--r-- 1 root root 1172307968 Dec 31  1969 result/iso/nixos-graphical-18.09pre130979.gfedcba-x86_64-linux.iso
@ilyaigpetrov

This comment has been minimized.

Show comment
Hide comment
@ilyaigpetrov

ilyaigpetrov Aug 24, 2018

@samueldr I get both on the minimal and the graphical ISOs on my early-EFI Nvidia notebook:

error: no suitable video mode found.
Booting in blind mode

It doesn't go away.
On other newer non-Nvidia notebook it works fine.
Some gotchas may be found on the arch wiki.
If you leave me some instructions I will follow tomorrow.
Thanks for your work.

ilyaigpetrov commented Aug 24, 2018

@samueldr I get both on the minimal and the graphical ISOs on my early-EFI Nvidia notebook:

error: no suitable video mode found.
Booting in blind mode

It doesn't go away.
On other newer non-Nvidia notebook it works fine.
Some gotchas may be found on the arch wiki.
If you leave me some instructions I will follow tomorrow.
Thanks for your work.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 24, 2018

Member

Oh, thanks for testing!

A quick note: for both isos, the behaviour is supposed to be the same as there is no distinction as far as the bootloader is concerned.

I am really glad you tested this as otherwise this would have left our users in a dire situation!

@ilyaigpetrov quick question: can you confirm that this is after successfully using grub or is the error happening before being able to select a boot option? (This changes how I will approach the issue.)

EDIT: and could you confirm that this is the same behaviour as observed in #5829; that the computer on which "Booting in blind mode" is the one which went to a blank screen; and that the other computer worked on both. I mainly want to ensure there's no actual regression...

Member

samueldr commented Aug 24, 2018

Oh, thanks for testing!

A quick note: for both isos, the behaviour is supposed to be the same as there is no distinction as far as the bootloader is concerned.

I am really glad you tested this as otherwise this would have left our users in a dire situation!

@ilyaigpetrov quick question: can you confirm that this is after successfully using grub or is the error happening before being able to select a boot option? (This changes how I will approach the issue.)

EDIT: and could you confirm that this is the same behaviour as observed in #5829; that the computer on which "Booting in blind mode" is the one which went to a blank screen; and that the other computer worked on both. I mainly want to ensure there's no actual regression...

samueldr added a commit to samueldr/nixpkgs that referenced this pull request Aug 24, 2018

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 24, 2018

Member

I have, nonetheless, applied what I think is the appropriate fix; that is, if I understand the issue correctly, and the gotchas link applies for your situation.

A quick overview of the change:

  1. Only loads the gfxterm interface if loadfont succeeds (as gotchas link does)
  2. Loads the efi modules upfront.
  3. Adds gfxmode=auto

I have doubts that adding gfxmode=auto changes anything as it is the default.

I can see that loading gfxterm only if loadfont succeeds is more appropriate than always loading it, but the switch to gfxterm was already being made conditional to loadfont succeeding.

I, though, have hope that loading the two modules that way could change the results.

Thanks again for testing, waiting for feedback (both using with the changes, and the questions asked in the previous comment),

Member

samueldr commented Aug 24, 2018

I have, nonetheless, applied what I think is the appropriate fix; that is, if I understand the issue correctly, and the gotchas link applies for your situation.

A quick overview of the change:

  1. Only loads the gfxterm interface if loadfont succeeds (as gotchas link does)
  2. Loads the efi modules upfront.
  3. Adds gfxmode=auto

I have doubts that adding gfxmode=auto changes anything as it is the default.

I can see that loading gfxterm only if loadfont succeeds is more appropriate than always loading it, but the switch to gfxterm was already being made conditional to loadfont succeeding.

I, though, have hope that loading the two modules that way could change the results.

Thanks again for testing, waiting for feedback (both using with the changes, and the questions asked in the previous comment),

@ilyaigpetrov

This comment has been minimized.

Show comment
Hide comment
@ilyaigpetrov

ilyaigpetrov Aug 25, 2018

can you confirm that this is after successfully using grub or is the error happening before being able to select a boot option?

I am able to successfully select an option from the list or edit boot parameters before the boot, after this I get the error. That's all I know.

could you confirm that this is the same behaviour as observed in #5829; that the computer on which "Booting in blind mode" is the one which went to a blank screen; and that the other computer worked on both.

Yes, the problematic notebook is the same as in #5829, the other notebook works both with the grub and the old boot loader.

I have tried your new commits -- unfortunately they don't solve the issue.

P.S. Other linux distros (Lubuntu) on the problematic notebook show the same "error: no suitable video mode found" but after some seconds boot an X11 server or a terminal in high resolution (if I remember it correctly).

ilyaigpetrov commented Aug 25, 2018

can you confirm that this is after successfully using grub or is the error happening before being able to select a boot option?

I am able to successfully select an option from the list or edit boot parameters before the boot, after this I get the error. That's all I know.

could you confirm that this is the same behaviour as observed in #5829; that the computer on which "Booting in blind mode" is the one which went to a blank screen; and that the other computer worked on both.

Yes, the problematic notebook is the same as in #5829, the other notebook works both with the grub and the old boot loader.

I have tried your new commits -- unfortunately they don't solve the issue.

P.S. Other linux distros (Lubuntu) on the problematic notebook show the same "error: no suitable video mode found" but after some seconds boot an X11 server or a terminal in high resolution (if I remember it correctly).

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 25, 2018

Member

Thanks for testing again @ilyaigpetrov! This 99% confirms my suspicions: To fix such issues, the installer image needs to use KMS (kernel modesetting) in your situation.

That's why you see no output, and then a high resolution terminal.

Since there is no regression, it doesn't block this from merging (🎉), but I have an actionable target: either default to allowing modesetting (there's probably a reason it's not enabled), and add a quirk entry to disable it; or vice versa, keep modesetting defaulted to off, and add a quirk entry to enable it (for such situations).

I'm now curious, @ilyaigpetrov, your other comment I just now read brings this to 99.99% certainty. Does using one of the resolution option under "HiDPI, Quirks and Accessibility" → "Suggests resolution @ ____", then selecting the default installer option work? It may start up without output, but after a couple seconds it could end-up showing stuff. The parameter it adds is supposed to force a resolutoin at the KMS driver level.

Member

samueldr commented Aug 25, 2018

Thanks for testing again @ilyaigpetrov! This 99% confirms my suspicions: To fix such issues, the installer image needs to use KMS (kernel modesetting) in your situation.

That's why you see no output, and then a high resolution terminal.

Since there is no regression, it doesn't block this from merging (🎉), but I have an actionable target: either default to allowing modesetting (there's probably a reason it's not enabled), and add a quirk entry to disable it; or vice versa, keep modesetting defaulted to off, and add a quirk entry to enable it (for such situations).

I'm now curious, @ilyaigpetrov, your other comment I just now read brings this to 99.99% certainty. Does using one of the resolution option under "HiDPI, Quirks and Accessibility" → "Suggests resolution @ ____", then selecting the default installer option work? It may start up without output, but after a couple seconds it could end-up showing stuff. The parameter it adds is supposed to force a resolutoin at the KMS driver level.

@ilyaigpetrov

This comment has been minimized.

Show comment
Hide comment
@ilyaigpetrov

ilyaigpetrov Aug 25, 2018

Here is the flow for you, @samueldr :
HiDPI, Quirks and Accessibility

error: no suitable video mode found.
Press any key to continue

→I press a key and select 720p
→the error above again appears, Press any key..., I press a key
→GRUB without HiDPI option, I select the default option→

error: no suitable video mode found.
Booting in blind mode

→The error doesn't go away.

P.S. The ISO image may be not of the last commit where you tried to apply arch wiki gotchats, but the previous.

ilyaigpetrov commented Aug 25, 2018

Here is the flow for you, @samueldr :
HiDPI, Quirks and Accessibility

error: no suitable video mode found.
Press any key to continue

→I press a key and select 720p
→the error above again appears, Press any key..., I press a key
→GRUB without HiDPI option, I select the default option→

error: no suitable video mode found.
Booting in blind mode

→The error doesn't go away.

P.S. The ISO image may be not of the last commit where you tried to apply arch wiki gotchats, but the previous.

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Aug 25, 2018

Member

No worries, the gotchas commit shouldn't affect the outcome since this is strictly something Linux manages.

Seeing the errors every time is... an annoying misfeature of grub; long explanation cut short: every sub-menu has to be re-initialized the same way the main menu has to be, which shows any errors while the main menu doesn't 🤷.

Improvements to do noted in #5829 could help those in the same situation in the future, though won't be part of this PR as they would be orthogonal.

I have dropped the lesser-tested last commit, in case it caused issues wherever else the other state of this PR was tested. The commit is available as a patch here for testing purposes. Anyway, AFAICT, the way grub works made most of those changes no-ops.

Member

samueldr commented Aug 25, 2018

No worries, the gotchas commit shouldn't affect the outcome since this is strictly something Linux manages.

Seeing the errors every time is... an annoying misfeature of grub; long explanation cut short: every sub-menu has to be re-initialized the same way the main menu has to be, which shows any errors while the main menu doesn't 🤷.

Improvements to do noted in #5829 could help those in the same situation in the future, though won't be part of this PR as they would be orthogonal.

I have dropped the lesser-tested last commit, in case it caused issues wherever else the other state of this PR was tested. The commit is available as a patch here for testing purposes. Anyway, AFAICT, the way grub works made most of those changes no-ops.

@grahamc

This comment has been minimized.

Show comment
Hide comment
@grahamc

grahamc Aug 29, 2018

Member

Building I get this output:

isohybrid: Warning: more than 1024 cylinders: 1102
isohybrid: Not all BIOSes will be able to boot this device

wonder the impacts of that.

reboots

  • dell xps 9560: hidpi options work nicely
  • support for rotating the screen is a nice touch
  • macbook pro late 2013 retina 15" hipdi options don't work
  • ~7yrold amd motherboard I think: booted fine but none of the fancy options showed up, which isn't problematic.
  • pcengines apu2c: couldn't get it to boot the disk image, but that isn't the disk images' fault. A shame since it is a serial-only device with no graphical capabilities. If anyone has ideas / suggestions on how to do this, please let me (cc @fpletz @globin @disassembler @andir who I believe have these too)
Member

grahamc commented Aug 29, 2018

Building I get this output:

isohybrid: Warning: more than 1024 cylinders: 1102
isohybrid: Not all BIOSes will be able to boot this device

wonder the impacts of that.

reboots

  • dell xps 9560: hidpi options work nicely
  • support for rotating the screen is a nice touch
  • macbook pro late 2013 retina 15" hipdi options don't work
  • ~7yrold amd motherboard I think: booted fine but none of the fancy options showed up, which isn't problematic.
  • pcengines apu2c: couldn't get it to boot the disk image, but that isn't the disk images' fault. A shame since it is a serial-only device with no graphical capabilities. If anyone has ideas / suggestions on how to do this, please let me (cc @fpletz @globin @disassembler @andir who I believe have these too)
@grahamc

This comment has been minimized.

Show comment
Hide comment
@grahamc

grahamc Aug 29, 2018

Member

Regarding the error message I found:

isohybrid: Warning: more than 1024 cylinders: 1102
isohybrid: Not all BIOSes will be able to boot this device

this is already true in 18.09's build of the install image:

ISO image produced: 556512 sectors
Written to medium : 556512 sectors at LBA 0
Writing to 'stdio:/nix/store/30r53g56g0gh9cpcgdv90hqqga8rgf7h-nixos-graphical-18.09pre150780.a823fb3ae5e-x86_64-linux.iso/iso/nixos-graphical-18.09pre150780.a823fb3ae5e-x86_64-linux.iso' completed successfully.

Making image hybrid...
isohybrid: Warning: more than 1024 cylinders: 1087
isohybrid: Not all BIOSes will be able to boot this device

-- https://nix-cache.s3.amazonaws.com/log/vcmba1p8gyfw49hc5p547ln25mc4xmfg-nixos-graphical-18.09pre150780.a823fb3ae5e-x86_64-linux.iso.drv

however it is not in the 18.03 image build:

ISO image produced: 518656 sectors
Written to medium : 518656 sectors at LBA 0
Writing to 'stdio:/nix/store/yc5i1bdbsqpw7h14b041d75j900yrr01-nixos-graphical-18.03.133173.561866ab17b-x86_64-linux.iso/iso/nixos-graphical-18.03.133173.561866ab17b-x86_64-linux.iso' completed successfully.

Making image hybrid...

-- https://nix-cache.s3.amazonaws.com/log/076wkccbarikn6ndbpxs867p688v0x57-nixos-graphical-18.03.133173.561866ab17b-x86_64-linux.iso.drv

We may want to see why we increased by 3k sectors and try to keep it small. However, to find out if this was a blocker, I did some research in to what this vague warning actually means.

Thanks to rcf in ##linux on Freenode, I learned this is going to be a non-issue for any system newer than MSDOS basically:

Yes, this is about booting the hybrid ISO on an old BIOS. Unfortunately
some idiot BIOS vendors decided that USB sticks didn't need EDD support.

-- https://www.syslinux.org/archives/2015-March/023306.html

and for more information: https://en.wikipedia.org/wiki/INT_13H#EDD

However, rcf noted some boot firmware vendors suck and still don't support EDD for USB disks. This means if a user has a problem with booting the graphical disk, they should try the non-graphical disk.


Overall, I would be honored to press the merge button.

Member

grahamc commented Aug 29, 2018

Regarding the error message I found:

isohybrid: Warning: more than 1024 cylinders: 1102
isohybrid: Not all BIOSes will be able to boot this device

this is already true in 18.09's build of the install image:

ISO image produced: 556512 sectors
Written to medium : 556512 sectors at LBA 0
Writing to 'stdio:/nix/store/30r53g56g0gh9cpcgdv90hqqga8rgf7h-nixos-graphical-18.09pre150780.a823fb3ae5e-x86_64-linux.iso/iso/nixos-graphical-18.09pre150780.a823fb3ae5e-x86_64-linux.iso' completed successfully.

Making image hybrid...
isohybrid: Warning: more than 1024 cylinders: 1087
isohybrid: Not all BIOSes will be able to boot this device

-- https://nix-cache.s3.amazonaws.com/log/vcmba1p8gyfw49hc5p547ln25mc4xmfg-nixos-graphical-18.09pre150780.a823fb3ae5e-x86_64-linux.iso.drv

however it is not in the 18.03 image build:

ISO image produced: 518656 sectors
Written to medium : 518656 sectors at LBA 0
Writing to 'stdio:/nix/store/yc5i1bdbsqpw7h14b041d75j900yrr01-nixos-graphical-18.03.133173.561866ab17b-x86_64-linux.iso/iso/nixos-graphical-18.03.133173.561866ab17b-x86_64-linux.iso' completed successfully.

Making image hybrid...

-- https://nix-cache.s3.amazonaws.com/log/076wkccbarikn6ndbpxs867p688v0x57-nixos-graphical-18.03.133173.561866ab17b-x86_64-linux.iso.drv

We may want to see why we increased by 3k sectors and try to keep it small. However, to find out if this was a blocker, I did some research in to what this vague warning actually means.

Thanks to rcf in ##linux on Freenode, I learned this is going to be a non-issue for any system newer than MSDOS basically:

Yes, this is about booting the hybrid ISO on an old BIOS. Unfortunately
some idiot BIOS vendors decided that USB sticks didn't need EDD support.

-- https://www.syslinux.org/archives/2015-March/023306.html

and for more information: https://en.wikipedia.org/wiki/INT_13H#EDD

However, rcf noted some boot firmware vendors suck and still don't support EDD for USB disks. This means if a user has a problem with booting the graphical disk, they should try the non-graphical disk.


Overall, I would be honored to press the merge button.

efifwsetup efi_gop efi_uga \
ls search search_label search_fs_uuid search_fs_file \
gfxmenu gfxterm gfxterm_background gfxterm_menu test all_video loadenv \
exfat ext2 ntfs btrfs hfsplus udf \

This comment has been minimized.

@grahamc

grahamc Aug 29, 2018

Member

where did these various fs modules come from?

@grahamc

grahamc Aug 29, 2018

Member

where did these various fs modules come from?

This comment has been minimized.

@samueldr

samueldr Aug 29, 2018

Member

pkgs.grub2_efi; grub-mkimage knows what to do with those.

As for which were chosen? I'd like to confirm, but I think it came from cargo-culting a list of modules from a couple sources, mostly other distro setups; used to bootstrap the process of making this work, otherwise guessing which modules are needed is hard. You're right it probably needs to be reviewed, not only for the FS, but validating others are all useful (not necessarily used), and that no other that can be used are missing.

To see a list of all modules, using nix-index:

$ nix-locate -r x86_64.*.mod | grep ^grub2_efi
@samueldr

samueldr Aug 29, 2018

Member

pkgs.grub2_efi; grub-mkimage knows what to do with those.

As for which were chosen? I'd like to confirm, but I think it came from cargo-culting a list of modules from a couple sources, mostly other distro setups; used to bootstrap the process of making this work, otherwise guessing which modules are needed is hard. You're right it probably needs to be reviewed, not only for the FS, but validating others are all useful (not necessarily used), and that no other that can be used are missing.

To see a list of all modules, using nix-index:

$ nix-locate -r x86_64.*.mod | grep ^grub2_efi
@grahamc

This comment has been minimized.

Show comment
Hide comment
@grahamc

grahamc Aug 29, 2018

Member

@andir verified it works for a serial-only device... alleyoop!

Member

grahamc commented Aug 29, 2018

@andir verified it works for a serial-only device... alleyoop!

@grahamc grahamc merged commit a141b3a into NixOS:master Aug 29, 2018

8 checks passed

grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Aug 29, 2018

Member

Thanks for you long and hard work, @samueldr!

Member

Ericson2314 commented Aug 29, 2018

Thanks for you long and hard work, @samueldr!

@samueldr samueldr referenced this pull request Aug 30, 2018

Merged

Call for Content: 2018/08 #63

@vcunat

This comment has been minimized.

Show comment
Hide comment
@vcunat

vcunat Sep 1, 2018

Member

This broke generating i686 image: https://nix-cache.s3.amazonaws.com/log/0z6fqv1dmxbngw2r65r53y0qw20h0qkp-efi-directory.drv (I know very little about EFI).

Member

vcunat commented on 41e7de4 Sep 1, 2018

This broke generating i686 image: https://nix-cache.s3.amazonaws.com/log/0z6fqv1dmxbngw2r65r53y0qw20h0qkp-efi-directory.drv (I know very little about EFI).

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Sep 1, 2018

Member

Tracked in #45908

Member

samueldr replied Sep 1, 2018

Tracked in #45908

@samueldr samueldr deleted the samueldr:artwork/iso branch Sep 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment