Consider upgrading install iso kernel to version 4.15

[philrdubois]

Qubes OS version:

4.0

Affected component(s):

kernel on install iso

---

Steps to reproduce the behavior:

Boot iso

Expected behavior:

Load installation media

Actual behavior:

Kernel panic shortly after boot

General notes:

There is a known bug in some kernel versions after 4.9 (not certain when it started) that cause a kernel panic upon boot on some hardware. This bug has been repaired in kernel version 4.15. I haven't been able to find documentation on it but I did encounter some people discussing it online. It may not be very common but it's preventing me from installing Qubes on my Dell workstation. I tried compiling my own iso using the Qubes Builder but my lack of experience lead me to give up after several days of compiling and hitting snags.

---

Related issues:

#3787 might be related

[philrdubois]

It turns out that #3787 is not related.

In addition, if this affects you: In order to just be able to install Qubes 4.0 on my system, I had to scour the internet to find an install iso that had not been updated to kernel 4.14, and I found a 4.0-rc1 iso with kernel 4.11 at https://mirrors.dotsrc.org/qubes/iso/ where I had to subsequently update dom0 after installing.

[philrdubois]

It turns out that #3787 is not related.

In addition, if this affects you: In order to just be able to install Qubes 4.0 on my system, I had to scour the internet to find an install iso that had not been updated to kernel 4.14, and I found a 4.0-rc1 iso with kernel 4.11 at https://mirrors.dotsrc.org/qubes/iso/ where I had to subsequently update dom0 after installing.

[Aekez]

@philrdubois
There are a few ways to work around this if you're still encountering this issue, or for anyone stumbling over this issue. Docs on how to work around this issue would be a good way to mitigate some issues if a new installer iso cannot be made, and/or the next release version is long into the future. We have already done some work on exactly these kind of docs over at Qubes Community Collaboration, like for example a kickstarter guide, but it's not finished yet as of this time of writing. Well it works, but it has flaws that needs correcting, like correct settings for user accounts/initial boot.

Practical approaches for immediate solutions to this issue if including potential doc/know-how

• Install Qubes with kickstarter file (if graphics doesn't work).
  • Then update kernel with an USB pen.
  • Alternatively update normally over the internet in text-mode, if available.
  • Current fedora bugs in text-mode installer require kickstarter, text mode install is not enough.
    • Additional issues require kickstarter to include disk and account information in order not to trigger text-mode bugs that otherwise do not appear in graphical or kickstarter modes.
• Install Qubes on other working Qubes hardware, and transfer Qubes after updates/modifications.
  • For example using dd or other disk-clone tools, following up using dracut for new hardware and grub2-mkconfig for bootloading.
• Install Qubes by including kernel at early boot install.
  • Essentially modifying the installer/Grub.
    • I'm a bit out of my league here, but I believe things like dracut (modules/drivers used, system hardware modifications, etc.) issue can be overcome by bootstrapping modules, drivers and settings manually, but it may quite possibly be tedious work even with a good doc/guide. However on the other hand, it may not be so bad either. This needs confirmation from someone who knows more on these involved machanics.
• Compile and build Qubes by oneself
  • It isn't too difficult if following proper instructions/guides, it's surprisingly easy to build Qubes from scratch.
  • However, perhaps better instructions on how to update the kernel used in the installer may be warranted, since this is not very obvious in either the instructions, or the builder docs/configs. Builder.conf flag for kernel branch does build the kernel in listed branch, but it does not actually use it in the boot process to the installer. Here better instructions are needed.
• Third party Qubes releases
  • Have huge trust issues, considering little or nothing may be known of this third party.

Approaches, long-term

• Wait for new Qubes release version.
• Wait for unlikely updated installers before actual new release versions are released.

Alternative solution suggestion
Customized pay-able unofficial Qubes build releases, sort of like a mixed donation/service. If a user really, really, needs this kernel, and is willing to pay for it, then it may be a win-win for developers and user together if paying for the custom release. Increasing the payment a bit so it doesn't only pay for the time used, but also a small donation to the Qubes OS project. I.e. pre-set timeXcost, and user then sets the maximum time the user is willing to pay for. Tasks could also be limited to certain types, like only kernels, or other easy to debug changes to the qubes-builder, so that paid projects doesn't run out of control in terms of needed hours/pay, but stay within predictability. Also maybe considering a re-fund if the installer still doesn't work (general extra pay for service can cover the downtime for the specific unsuccessful services leads to refunds, this way refunds won't lead to looses, and if prices spin out of control then more can be asked before a refund, like logs/pictures/etc.). Thoughts on this @andrewdavidwong ? Could this be viable? I'm not suggesting a full blown service business here, I know thoughts/work is already being put into that, but instead a small minor service station for these kind of things. It could always be integrated into a larger service offering structure later on, similar this could also serve as a "trial" for service offerings before doing the real thing on a larger scale. After all s successful businesses frequently start out small, and then scale up once succeeding.

New improved docs can go a long way, but with available docs maybe mixed with payment services could become a beneficial approach to both the user and the Qubes OS project? Giving user options to try follow docs, or alternatively pay their way out of it. This also more easily justifies asking a price for it for the service, if a doc including specific instructions is available.

Also considering this is the re-build of Qubes itself, it'd require someone with a high level of trust. For example it's not something other parties, like QCC can do so easily without trust being an issue, since we're a small unknown party hardly no one knows about, that has not earned that kind of trust on a large scale. So it'd be very beneficial if Qubes OS project did these things when it comes to whom to trust, which also could be a win-win for both sides.

[Aekez]

@philrdubois
There are a few ways to work around this if you're still encountering this issue, or for anyone stumbling over this issue. Docs on how to work around this issue would be a good way to mitigate some issues if a new installer iso cannot be made, and/or the next release version is long into the future. We have already done some work on exactly these kind of docs over at Qubes Community Collaboration, like for example a kickstarter guide, but it's not finished yet as of this time of writing. Well it works, but it has flaws that needs correcting, like correct settings for user accounts/initial boot.

Practical approaches for immediate solutions to this issue if including potential doc/know-how

• Install Qubes with kickstarter file (if graphics doesn't work).
  • Then update kernel with an USB pen.
  • Alternatively update normally over the internet in text-mode, if available.
  • Current fedora bugs in text-mode installer require kickstarter, text mode install is not enough.
    • Additional issues require kickstarter to include disk and account information in order not to trigger text-mode bugs that otherwise do not appear in graphical or kickstarter modes.
• Install Qubes on other working Qubes hardware, and transfer Qubes after updates/modifications.
  • For example using dd or other disk-clone tools, following up using dracut for new hardware and grub2-mkconfig for bootloading.
• Install Qubes by including kernel at early boot install.
  • Essentially modifying the installer/Grub.
    • I'm a bit out of my league here, but I believe things like dracut (modules/drivers used, system hardware modifications, etc.) issue can be overcome by bootstrapping modules, drivers and settings manually, but it may quite possibly be tedious work even with a good doc/guide. However on the other hand, it may not be so bad either. This needs confirmation from someone who knows more on these involved machanics.
• Compile and build Qubes by oneself
  • It isn't too difficult if following proper instructions/guides, it's surprisingly easy to build Qubes from scratch.
  • However, perhaps better instructions on how to update the kernel used in the installer may be warranted, since this is not very obvious in either the instructions, or the builder docs/configs. Builder.conf flag for kernel branch does build the kernel in listed branch, but it does not actually use it in the boot process to the installer. Here better instructions are needed.
• Third party Qubes releases
  • Have huge trust issues, considering little or nothing may be known of this third party.

Approaches, long-term

• Wait for new Qubes release version.
• Wait for unlikely updated installers before actual new release versions are released.

Alternative solution suggestion
Customized pay-able unofficial Qubes build releases, sort of like a mixed donation/service. If a user really, really, needs this kernel, and is willing to pay for it, then it may be a win-win for developers and user together if paying for the custom release. Increasing the payment a bit so it doesn't only pay for the time used, but also a small donation to the Qubes OS project. I.e. pre-set timeXcost, and user then sets the maximum time the user is willing to pay for. Tasks could also be limited to certain types, like only kernels, or other easy to debug changes to the qubes-builder, so that paid projects doesn't run out of control in terms of needed hours/pay, but stay within predictability. Also maybe considering a re-fund if the installer still doesn't work (general extra pay for service can cover the downtime for the specific unsuccessful services leads to refunds, this way refunds won't lead to looses, and if prices spin out of control then more can be asked before a refund, like logs/pictures/etc.). Thoughts on this @andrewdavidwong ? Could this be viable? I'm not suggesting a full blown service business here, I know thoughts/work is already being put into that, but instead a small minor service station for these kind of things. It could always be integrated into a larger service offering structure later on, similar this could also serve as a "trial" for service offerings before doing the real thing on a larger scale. After all s successful businesses frequently start out small, and then scale up once succeeding.

New improved docs can go a long way, but with available docs maybe mixed with payment services could become a beneficial approach to both the user and the Qubes OS project? Giving user options to try follow docs, or alternatively pay their way out of it. This also more easily justifies asking a price for it for the service, if a doc including specific instructions is available.

Also considering this is the re-build of Qubes itself, it'd require someone with a high level of trust. For example it's not something other parties, like QCC can do so easily without trust being an issue, since we're a small unknown party hardly no one knows about, that has not earned that kind of trust on a large scale. So it'd be very beneficial if Qubes OS project did these things when it comes to whom to trust, which also could be a win-win for both sides.

[andrewdavidwong]

Customized pay-able unofficial Qubes build releases, sort of like a mixed donation/service. If a user really, really, needs this kernel, and is willing to pay for it, then it may be a win-win for developers and user together if paying for the custom release. Increasing the payment a bit so it doesn't only pay for the time used, but also a small donation to the Qubes OS project. I.e. pre-set timeXcost, and user then sets the maximum time the user is willing to pay for. Tasks could also be limited to certain types, like only kernels, or other easy to debug changes to the qubes-builder, so that paid projects doesn't run out of control in terms of needed hours/pay, but stay within predictability. Also maybe considering a re-fund if the installer still doesn't work (general extra pay for service can cover the downtime for the specific unsuccessful services leads to refunds, this way refunds won't lead to looses, and if prices spin out of control then more can be asked before a refund, like logs/pictures/etc.). Thoughts on this @andrewdavidwong ? Could this be viable? I'm not suggesting a full blown service business here, I know thoughts/work is already being put into that, but instead a small minor service station for these kind of things. It could always be integrated into a larger service offering structure later on, similar this could also serve as a "trial" for service offerings before doing the real thing on a larger scale. After all s successful businesses frequently start out small, and then scale up once succeeding.

We've considered this sort of thing before and decided against it for largely the same reasons that we have this policy regarding donations:

https://www.qubes-os.org/donate/#do-donors-get-personalized-support-or-special-feature-requests

The devs already have their hands full with project milestones and, in the case of ITL devs, ITL's corporate clients.

[andrewdavidwong]

Customized pay-able unofficial Qubes build releases, sort of like a mixed donation/service. If a user really, really, needs this kernel, and is willing to pay for it, then it may be a win-win for developers and user together if paying for the custom release. Increasing the payment a bit so it doesn't only pay for the time used, but also a small donation to the Qubes OS project. I.e. pre-set timeXcost, and user then sets the maximum time the user is willing to pay for. Tasks could also be limited to certain types, like only kernels, or other easy to debug changes to the qubes-builder, so that paid projects doesn't run out of control in terms of needed hours/pay, but stay within predictability. Also maybe considering a re-fund if the installer still doesn't work (general extra pay for service can cover the downtime for the specific unsuccessful services leads to refunds, this way refunds won't lead to looses, and if prices spin out of control then more can be asked before a refund, like logs/pictures/etc.). Thoughts on this @andrewdavidwong ? Could this be viable? I'm not suggesting a full blown service business here, I know thoughts/work is already being put into that, but instead a small minor service station for these kind of things. It could always be integrated into a larger service offering structure later on, similar this could also serve as a "trial" for service offerings before doing the real thing on a larger scale. After all s successful businesses frequently start out small, and then scale up once succeeding.

We've considered this sort of thing before and decided against it for largely the same reasons that we have this policy regarding donations:

https://www.qubes-os.org/donate/#do-donors-get-personalized-support-or-special-feature-requests

The devs already have their hands full with project milestones and, in the case of ITL devs, ITL's corporate clients.

[Aekez]

We've considered this sort of thing before and decided against it for largely the same reasons that we have this policy regarding donations:

https://www.qubes-os.org/donate/#do-donors-get-personalized-support-or-special-feature-requests

The devs already have their hands full with project milestones and, in the case of ITL devs, ITL's corporate clients.

Aha I see, thanks for clarifying. The policy makes sense, and it's definitely a good idea to keep the corporate support separated.

Will be seeing if any docs related to this issue can be submitted for PR sometime in the future. The kickstarter doc should be once the remaining known problems with the doc has been resolved.

[Aekez]

We've considered this sort of thing before and decided against it for largely the same reasons that we have this policy regarding donations:

https://www.qubes-os.org/donate/#do-donors-get-personalized-support-or-special-feature-requests

The devs already have their hands full with project milestones and, in the case of ITL devs, ITL's corporate clients.

Aha I see, thanks for clarifying. The policy makes sense, and it's definitely a good idea to keep the corporate support separated.

Will be seeing if any docs related to this issue can be submitted for PR sometime in the future. The kickstarter doc should be once the remaining known problems with the doc has been resolved.