Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shim-15.8 for WAXAR #362

Open
8 tasks done
Jurij-Ivastsuk opened this issue Jan 15, 2024 · 58 comments
Open
8 tasks done

shim-15.8 for WAXAR #362

Jurij-Ivastsuk opened this issue Jan 15, 2024 · 58 comments
Assignees
Labels
bug Problem with the review that must be fixed before it will be accepted contacts verified OK Contact verification is complete here (or in an earlier submission) extra review wanted Initial review(s) look good, another review desired new vendor This is a new vendor question Reviewer(s) waiting on response

Comments

@Jurij-Ivastsuk
Copy link

Jurij-Ivastsuk commented Jan 15, 2024

Confirm the following are included in your repo, checking each box:

  • completed README.md file with the necessary information
  • shim.efi to be signed
  • public portion of your certificate(s) embedded in shim (the file passed to VENDOR_CERT_FILE)
  • binaries, for which hashes are added to vendor_db ( if you use vendor_db and have hashes allow-listed )
  • any extra patches to shim via your own git tree or as files (we use no extra patches)
  • any extra patches to grub via your own git tree or as files (no extra patches, we use Canonical upstream for GRUB)
  • build logs
  • a Dockerfile to reproduce the build of the provided shim EFI binaries

What is the link to your tag in a repo cloned from rhboot/shim-review?


https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240601


What is the SHA256 hash of your final SHIM binary?


SHA256 (shimx64.efi): 1894c7e467991c117162e1e40dd61ed85a5f790a68216fdbee93b2619b70df67
SHA256 (shimaa64.efi): 5c7238473401d791c7f376efee90cf1e9fe23e2bf1814f4795474d57920bdfbf


What is the link to your previous shim review request (if any, otherwise N/A)?


N/A

@aronowski aronowski self-assigned this Jan 18, 2024
@aronowski aronowski added the new vendor This is a new vendor label Jan 19, 2024
@aronowski
Copy link
Collaborator

Hello.

I'll start reviewing the application soon, but the first requirement is that all the checkboxes are checked and render well.

Please, fix the formatting and check the missing one - it's supposed to mean "if we have any extra patches, then they have been included."

@aronowski aronowski added the bug Problem with the review that must be fixed before it will be accepted label Jan 19, 2024
@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, many thanks for any tips!
I literally understood point 6 as "if you have extra patches for GRUB..." Since we use the GRUB software upstream from Canonical, and don't make any changes there, with patches or otherwise, I figured I shouldn't mark this point as done.

@aronowski aronowski removed the bug Problem with the review that must be fixed before it will be accepted label Jan 19, 2024
@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk, thank you!

I agree that some part in here may not be clear, but I'll try my best to address them. Some documents I've been working on improving during my free time, but haven't requested to incorporate them so far.

Reviewing also may not be easy especially when dealing with environments I'm not that familiar with or analyzing patches for correctness (took me about 8 hours to spot one major error, but I did it). If there's no activity e.g. by the end of next week, ping me.

@aronowski aronowski added the contact verification needed Contact verification is needed for this review label Jan 19, 2024
@Jurij-Ivastsuk
Copy link
Author

@aronowski, I understand and I thank you for the work you do in your spare time!

@aronowski
Copy link
Collaborator

Hello.

I did scratch the surface of the application. There are some curiosities and things that need to be fixed. I address them as some loose thoughts below.

In the meantime, I'm sending contact verification emails soon. Please, post the random words, that I'm emailing you, in the comments.


The NX compatibility bit should be enabled only if the whole bootchain is NX compatible - that's the requirement Microsoft has, as I'm aware right now. Therefore, the patch 530 shall not be used.

Furthermore, the validation function patch does its job in regard to older requirements, which have changed - there's no need to use it today.

Please, update the binaries and the building process as described - I'll then try to rebuild it and check if everything is correct.


The 626.patch PR hasn't been merged and debugging the code in my head is out of the question. It's just a one-line change, but no idea about the consequences and possible corner-cases, especially since I'm not a low-level programmer.

I'm not saying that it's unable to be approved as-is - just that I'll need some help with it and wondering, how to get some.


Linux required upstream commits - are they applied for the 5.15.1 Debian kernel?

The commit eadb2f47a3ced5c64b23b90fd2a3463f63726066 as far as I know got introduced in Linux v5.19. Is it backported in Debian?


There's no ephemeral key - is the strategy sufficient, as it's described right now?

It would be preferable to mention the usage of CONFIG_MODVERSIONS=y and CONFIG_MODULE_SIG_FORCE=y. Furthermore, is a new key being used to sign each new kernel build?


The CA certificate waxar.ceris valid for almost 100 years and it's only 2048 bit.

I don't know if Microsoft would be willing to approve this - any chances of generating one with a shorter lifespan and/or using 4096 bit public modulus?

@aronowski aronowski added bug Problem with the review that must be fixed before it will be accepted question Reviewer(s) waiting on response labels Jan 22, 2024
@aronowski
Copy link
Collaborator

Verification emails sent.

@Jurij-Ivastsuk
Copy link
Author

Jurij-Ivastsuk commented Jan 22, 2024

I have received your email. Here is the decrypted text:
summerhouses ants regionally announces unmasking fusing addling bantered
sprouted chiropractic

@Jurij-Ivastsuk
Copy link
Author

Jurij-Ivastsuk commented Jan 22, 2024

@aronowski Hi, thank you very much for the first hints!
Now I have changed the following:

  1. patches 530 and 531 (NX compatibility) have been removed
  2. docker file adapted accordingly
  3. CA certificate waxar.cer is now valid for 50 years and its size is now 4096 byte
  4. for kernel building we use CONFIG_MODVERSIONS=y and CONFIG_MODULE_SIG_FORCE=y
  5. the new certificate is already used for signing kernel build as well as for signing all modules
    Regarding commit eadb2f47a3ced5c64b23b90fd2a3463f63726066 (CVE-2022-21499) in Debian, I have found the following information:
    bullseye 5.10.197-1 fixed
    bullseye (security) 5.10.205-2 fixed
    bullseye 5.10.120-1 fixed
    bookworm 6.1.66-1 fixed
    Exactly for kernel 5.15.1 no Information could be found...
    As for 626.patch, I understand that you cannot be responsible for all matters. Maybe you can ping someone who does low-level programming and who can properly assess our PR request. Actually I explained the problem and the solution pretty well. Without this patch we can't use shim like this because it keeps giving us errors (screenshot and explanation in PR 626).

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, should we now switch to 15.8 ?

@aronowski
Copy link
Collaborator

Updating should extend the lifecycle of the review if something happened and if reviews from other people from the committee got delayed. I'd do so, especially since I just scratched the surface and will perform a further review, after the latest fixes have been applied.

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi,
the second contact (Lukas Kienbaum) have received your email (shim-words x2). Here is the decrypted text:
hauled multiplication Collins reemerge exhilarates fathers Vijayawada
stubbiest fête refile

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, did I understand you correctly, we are waiting until the last reviews from the committee are closed, then you will continue with your review with us? I still don't understand if we will stay with version 15.7 or will we logically switch to 15.8. In this case, if I understand correctly, a new review "shim-15.8 for WAXAR" will be requested. Is that the case?

@aronowski aronowski removed the contact verification needed Contact verification is needed for this review label Feb 1, 2024
@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk

Please, update all the required things to reflect shim 15.8. I'll then perform further review and ask other reviewers for help (you can ping them as well).


we are waiting until the last reviews from the committee are closed, then you will continue with your review with us?

I'll try my best when there's optimal environment for me - for instance, the comment from January 22 I posted in the middle of the night in my timezone, as I had some free time and no other things on my mind.

Still, there are other things that can make some applications being reviewed ASAP and some that wait for, let's say, a month or even more. The main thing is that I'm just more familiar with certain distros and their environments.

Some things I gathered as part of the improvements I'd like to introduce to the project, so they are blatantly visible, e.g. here.


CA certificate [...] size is now 4096 byte

4096 byte? ;-)

@Jurij-Ivastsuk Jurij-Ivastsuk changed the title shim-15.7 for WAXAR shim-15.8 for WAXAR Feb 2, 2024
@Jurij-Ivastsuk
Copy link
Author

Jurij-Ivastsuk commented Feb 2, 2024

@aronowski Hi, thank you very much for the tips and explanations! I really like your suggestions for improvements. I have to apologize for the incorrect representation of the certificate size: of course it's bits and not bytes. So the CA certificate size is now 4096 bits... ;-)))
I'll let you know when I've made all the adjustments for 15.8.

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, I have a question, can you please explain to me why the CA certificate must be 4096 bit in size and not 2048 bit?

@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk, it's not like it must be this or that, but instead a good practice to follow the recommendations from NIST.SP.800-57pt1r5 - table no. 2 on page 54 and table no. 4 on page 59.

I'll review all the updates soon, once I get more free time and some well-deserved sleep.

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, please only continue with review if you have slept enough. i have made all adjustments for version 15.8 ;-))

@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk,

Exactly for kernel 5.15.1 no Information could be found...

Apologies, that's the Ubuntu Jammy kernel, rather than Debian's as I thought earlier.

So I think our best bet is to wait for an official answer from Canonical kernel maintainers. Please see #368 (comment).


please only continue with review if you have slept enough

Let's say 6 hours is enough for now. :-P

I was checking things out and got stuck with the kernel-related thing. Let's hope it gets resolved soon.

@aronowski aronowski removed bug Problem with the review that must be fixed before it will be accepted question Reviewer(s) waiting on response labels Feb 14, 2024
@aronowski
Copy link
Collaborator

No progress on my end - hence why I myself asked that question from a newcomer point of view, as well as suggested pinging an expert.

The current implementation seems to me a bit suspicious security-wise, so I don't want to make any statements like "it's secure, period", when I have my doubts. A healthy dose of vigilance may prevent an end-user from having their system compromised.

Considering that creating the laboratory I mentioned and studying a subject I've never worked with before may take a lot of time, the best way to help is by reviewing other applications, so I have a bit less work to do and can spend the time on the lab.

Alternatively, I can help with rearchitecting the company's build system, so the CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE options can be set and the building and signing maybe being done on a single machine. Although I don't know if that will be possible due to things like NDAs and me not being an employee there.
(Note: that's just the beginning of things to come - the mandatory part, but not everything)

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, I think we can solve this problem differently. Please correct me if I am wrong. I could change the config file so that the two parameters (CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE) are set to "Yes", recompile the kernel and all modules and then re-sign the kernel and all modules with our certificate. What do you think, will such a procedure solve the problem? Then you don't need to do any more research on these parameters.

@aronowski
Copy link
Collaborator

aronowski commented Apr 3, 2024 via email

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hello, we have done everything as announced in the previous message. How can we move forward?

@aronowski
Copy link
Collaborator

aronowski commented May 1, 2024 via email

@THS-on
Copy link
Collaborator

THS-on commented May 2, 2024

A few comments before looking at the at rest of the submission:

  • Can you include a description of your current kernel module signing setup in the submission itself? As you are not using ephemeral key signing and it's easier to read then the comments on this issue. Similarly as @aronowski suggested adding your kernel config and sources makes it easier to accept for us.
  • What kind of lockdown patches for the kernel are you using? Please add this to the submission
  • Regarding Improving the robustness of value retention for the variable second_stage shim#626: I would like at least see a comment from a shim developer that this is fine. On first sight it looks ok, but any custom patch makes it harder for us to accept. Because this is a new vendor submission it might make sense to submit a vanilla shim first.
  • Regarding GRUB: I assume you then take the upstream binary and add your own signatures to it? Ideally rebuild it with your own SBAT entry added, as this allows easier revocation.

@Jurij-Ivastsuk
Copy link
Author

@aronowski , @THS-on Thank you for the further review!
Please see the current commit tagged as: waxar-shim-x86_64-aarch64-20240506

  • The description of our current kernel modul signing setup is described in the file Setup_for_module_signing.txt. Kernel sources you can find in archive linux-source-515.tar.gz. The config file .config_WRK_5_15_SIG_Yes and the config file .config in the surce are identical.
  • Unfortunately, the file with compressed kernel sources linux-source-515.tar.gz could not be pushed because the file size (193.46 MB) exceeds the GitHub limit of 100 MB. The error message: remote: error: File linux-source-515.tar.gz is 193.46 MB; this exceeds GitHub's file size limit of 100.00 MB
  • We are currently using the sources of linux kernel 5.15, the lockdown patches have been merged into the mainline Linux since kernel 5.4. For this reason, we have not applied any lockdown patches in our kernel version.
  • What concerns the patch #626 (Improving the robustness of value retention for the variable second_stage #626), I think the review by @Metabolix is exactly what you want to see. Please read the post from Feb 15 by @Metabolix.
  • What concerns GRUB: We use the latest official grubx64.efi file from Canonical, replace the SBAT section with a new one where our line is included and then sign this file additionally with our certificate. The SBAT section can be replaced with:
    objcopy --set-section-alignment '.sbat=512' --remove-section .sbat --add-section .sbat=ubuntu_waxar.csv --adjust-section-vma .sbat+10000000 grubx64.efi Therefore, we do not need to recompile the original grubx64.efi file from Canonical.

@aronowski
Copy link
Collaborator

Please see the current commit tagged as: waxar-shim-x86_64-aarch64-20240506

There's something wrong - I can't see any reference to such name.

Unfortunately, the file with compressed kernel sources linux-source-515.tar.gz could not be pushed because the file size (193.46 MB) exceeds the GitHub limit of 100 MB. The error message: remote: error: File linux-source-515.tar.gz is 193.46 MB; this exceeds GitHub's file size limit of 100.00 MB

Please, don't push such blobs directly if there's no need to showcase something strictly related to them, like their metadata (Source RPMs come to mind). I'd instead recommend creating a separate repository and pushing only the contents of that archive.

@Jurij-Ivastsuk
Copy link
Author

@aronowski , @THS-on Hello, this is very strange, I can see this commit under this link:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240506

Please try again and let me know if it doesn't work for you. Actually, everyone can see this repository (at least public read
access should work).
As you suggested, I created a new repository and uploaded the Debian-Linux 5.15 kernel sources there. You can access them here:
https://github.com/Jurij-Ivastsuk/Linux_kernel_source-5.15

I hope you have enough information. If there is anything else missing, please let me know.

@Jurij-Ivastsuk
Copy link
Author

@aronowski , @THS-on Hi, yesterday together with @Metabolix friendly help we found the way how to make shim compilation without the patch I suggested earlier (Improving the robustness of value retention for the variable second_stage #626). For more information see the last posts under #626.
I would suggest, as soon as I get information from you what else should be changed or adapted, I make the new compilation of Shim without applying patch #626, adapt all other areas like Docker-File, SHA256 sums accordingly, make any adjustments you may suggest and create a new commit that should contain all changes. Please let me know when you are ready.

@Jurij-Ivastsuk
Copy link
Author

@aronowski , @THS-on Hello, I wanted to kindly ask if there is any progress in our case ?

@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk,

Hello, I wanted to kindly ask if there is any progress in our case ?

I suppose it will be more helpful if I start raising both positive feedback, as well as potential issues immediately, rather than doing one big review. The latter is tough for several reasons, one of them being the lack of having a comfortable office with air conditioner - doing an analysis in over 30 degrees Celsius does not help at all.

The curiosity I noticed:

Next, we set the parameter: CONFIG_MODULE_SIG_KEY="certs/signing_key.pem" (see config file).
The string provides the path to a file containing both a private key and its corresponding X.509 certificate in PEM form.
For security reasons, these files are not included in current Sources (linux-source-515.tar.gz).

According to https://www.kernel.org/doc/html/v5.15/admin-guide/module-signing.html:

File name or PKCS#11 URI of module signing key (CONFIG_MODULE_SIG_KEY)

Setting this option to something other than its default of certs/signing_key.pem will disable the autogeneration of signing keys and allow the kernel modules to be signed with a key of your choosing.

Was this intentional? Is it possible to check the compiled modules through static analysis, if they were signed with the appropriate signing key?

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hello, and thank you for your questions.
I can well understand that it is indeed difficult to work at temperatures above 30 degrees Celsius.
I hope that the temperature conditions will be a little more bearable for you in the next few days.

To your questions, during the compilation with CONFIG_MODULE_SIG_ALL=y we noticed that if there are no pub- and priv-keys
in the certs directory, they are not created automatically as the documentation says, no idea why.
But if we generate both keys beforehand and then insert them into the certs directory, then all modules are signed.
That is the reason for this procedure.

To check whether modules are actually signed we have 2 possibilities (as an example on the ext4.ko module):

  1. We see a line in the building log: SIGN debian/linux-image/lib/modules/5.15.1-devimg-waxar/kernel/fs/ext4/ext4.ko
  2. With the help of modinfo:
    modinfo ext4.ko
    filename: /usr/src/Test_Modules_Signature/ext4.ko
    softdep: pre: crypto-crc32c
    license: GPL
    description: Fourth Extended Filesystem
    author: Remy Card, Stephen Tweedie, Andrew Morton, Andreas Dilger, Theodore Ts'o and others
    alias: fs-ext4
    alias: ext3
    alias: fs-ext3
    alias: ext2
    alias: fs-ext2
    depends: mbcache,jbd2,crc16
    retpoline: Y
    intree: Y
    name: ext4
    vermagic: 5.15.1-devimg-waxar SMP mod_unload modversions
    sig_id: PKCS#7
    signer: Build time autogenerated kernel key
    sig_key: 30:00:4F:89:A6:9D:0E:78:07:FB:35:C5:A8:07:17:E7:8E:C4:39:C1
    sig_hashalgo: sha256
    signature: 04:16:59:88:3C:30:C2:19:12:CB:74:DE:AF:B8:25:49:4A:EE:F0:6A: .....

I hope I have answered all your questions. If you have any further questions, please do not hesitate to contact me.

@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk,

To your questions, during the compilation with CONFIG_MODULE_SIG_ALL=y we noticed that if there are no pub- and priv-keys
in the certs directory, they are not created automatically as the documentation says, no idea why.
But if we generate both keys beforehand and then insert them into the certs directory, then all modules are signed.
That is the reason for this procedure.

As far as I can see the process does seem to work like the kernel documentation says. The modinfo listing says:

[...]
signer: Build time autogenerated kernel key
[...]

This matches the process described in the "Generating signing keys" section of the v5.15 kernel documentation. In particular the x509.genkey file should be generated by your certs/Makefile.

For extra safety I'd also check your kernel's keyring on a booted system. Please see, if anything is printed by running:

# grep "Build time autogenerated kernel key" /proc/keys

on the current compilation of your system. If the same happens, then the usage of an ephemeral key is actually present here, most likely accidentally, as your shim application says otherwise (No worries - things like that may happen and that's why the peer-review process is here to help).

What I'd also recommend is to edit the kernel's config file, change the current line:

CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"

to something like

CONFIG_MODULE_SIG_KEY="certs/waxar_signing_key.pem"

and try again by renaming the file with your private key and its corresponding certificate to waxar_signing_key.pem. Once the kernel's compiled, see if modinfo prints something different, as intended in the application.

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hi, we have checked your suggestions:
# grep "Build time autogenerated kernel key" /proc/keys
there is no output, the string does not exist

After new compilation we have checked the same module (modinfo ext4.ko)
[...] signer: Build time autogenerated kernel key [...]
No changes in this line.

@steve-mcintyre steve-mcintyre added the contacts verified OK Contact verification is complete here (or in an earlier submission) label May 29, 2024
@Jurij-Ivastsuk
Copy link
Author

@aronowski Hello, I have now created a commit with the current Shim version 15.8.
I have compiled this Shim variant without using the improvement I suggested earlier (Improving the robustness of value retention for the variable second_stage #626). The main change: in this variant I use the parameter DISABLE_REMOVABLE_LOAD_OPTIONS for compiling (see https://github.com/rhboot/shim/blob/main/BUILDING).
With this parameter I can solve my problem without using the patch #626.

Changes in this commit:

  1. current link to the current tag: waxar-shim-x86_64-aarch64-20240531
  2. current SHA256 hashes for both files shimx64.efi and shimaa64.efi
  3. Dockerfile now reflects the changes in the building process
  4. build.log file now reflects the changes in the building process
  5. both files shimx64.efi and shimaa64.efi are now up to date

You can access this commit under the following link:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240531

@Jurij-Ivastsuk
Copy link
Author

@aronowski Please don't be surprised, I have forgotten something:

  1. Inserted changes and adjustments to the README.md file to reflect the latest changes in relation to recompiling without patches

Now all the documentation is complete... I think :-)
You can access this commit under the following link:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240601

@aronowski
Copy link
Collaborator

@Jurij-Ivastsuk, I'll take a look at the updated shim no sooner than this Sunday. Please let me know everything I need to know by then.

@Jurij-Ivastsuk
Copy link
Author

@aronowski Hello, I'm sure you are responsible for many application reviews and each application has its own special features. Therefore, in order to simplify things and time savings for you with our application, so that you have a better overview, I am writing a short summary of what we have done together so far.

  1. We have checked email contacts they are OK
  2. We have created a new certificate with 4096 bit and a validity period of 50 years (instead of 2048 bit, 100 years) and implemented it for the compilation of the current Shim version (15.8) and signing of our boot components. You have checked this certificate and it was OK
  3. I had quite a long discussion with @Metabolix about the usefulness of the PR 626 (Improving the robustness of value retention for the variable second_stage shim#626). In the end, although the solution we proposed in PR 626 turned out to be viable, we found a solution that solves our problem and does not change the code (using the parameter DISABLE_REMOVABLE_LOAD_OPTIONS=y). So I think using this parameter will make the review process easier for you, because there are no more code changes to discuss.
  4. According to your recommendation, we have changed the config file for kernel compilation and activated the parameters CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE. So that the modules are automatically signed with a certificate during kernel compilation. This increases the security that the kernel cannot load any foreign modules. We can also prove that all modules are signed accordingly after kernel compilation.

All changes and adjustments are now included in the last commit (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240601)
Please let me know if something is missing.

@THS-on
Copy link
Collaborator

THS-on commented Jun 10, 2024

By just looking at the provided kernel config and setup description this contradicts the statement made in the README.
If you have the private key in a file, it is not secured by an HSM. Please clarify the following:

  • How are your signing keys stored for the kernel modules?
  • Why are you not able to use ephemeral key signing?
  • How do you ensure that kernel modules from older kernels cannot be loaded on newer ones?

@aronowski if I see it correctly the cert is only newly generated if the x509.genkey changes. See https://github.com/torvalds/linux/blob/83a7eefedc9b56fe7bfeff13b6c7356688ffa670/certs/Makefile#L52-L53

@Jurij-Ivastsuk
Copy link
Author

Jurij-Ivastsuk commented Jun 16, 2024

@THS-on , @aronowski Hi, thank you for questions.
In HSM only our private key for our certificate is stored which we only use for our CA certificate, which in turn is used for signing kernel and grub.

  • If we compile a new Linux kernel with the options CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE, the signature keys are handled as follows:
    During the kernel compilation process (or, in our case before compilation), a new key pair (private and public key) is generated for signing the kernel modules. The private key is used to sign the modules during the compilation. The public key is integrated into the compiled kernel. This is then used when loading the modules to check their signature. The generated keys are not stored persistently. A new key pair will be generated with each new kernel compilation. We delete the private key immediately after kernel compilation to prevent misuse. The public key is contained in the compiled kernel and does not need to be stored separately.
  • As I wrote earlier in this chat, we can't use the ephemeral key because building and signing are done on different machines and we cannot rule out the possibility that there will be a need to build out-of-tree modules.
  • In order to ensure that kernel modules from older kernels cannot be loaded on newer ones, We always remove all old modules after upgrading to a new kernel version.
    The simplest way to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case, as I already mentioned, we cannot use them.

@steve-mcintyre steve-mcintyre added the bug Problem with the review that must be fixed before it will be accepted label Jun 17, 2024
@steve-mcintyre
Copy link
Collaborator

@Jurij-Ivastsuk Simply deleting old kernel modules from a system is not enough for revocation. You need to have technical measures in place to block them from loading.

There's a few issues here where it's not clear from your submission that you understand how things are meant to work.

@Jurij-Ivastsuk
Copy link
Author

@THS-on , @aronowski
Dear @steve-mcintyre thank you for your comments. Our Linux environment is located on the external boot device like CD, USB stick or USB drive (like live-CD). The boot process is initiated by selecting Boot Device from the boot menu.

  • When we upgrade our entire system, we have full control over kernel upgrade, modules upgrade in system and modules upgrade in initramfs. Therefore, when we upgrade - we upgrade the entire system, starting from bootloader stage1 and stage2, kernel, initramfs up to all system components and modules. All modules in initramfs and in the system tree are linked to the corresponding kernel. Therefore, I think the treatment of modules from the security perspective is slightly different in our case.

  • As I have already mentioned, the simplest mechanism to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case we cannot use them. Theoretically, there is another possibility to block the loading of old modules, namely to use the blacklisting procedure for modules. But in our case this procedure would not be applicable because the names of most modules remain the same during the transition to a new version.

  • Finally, we have a mechanism designed to exclude unwanted modules from loading - signing the modules with a unique license key at compile time. This is the purpose of signing. All modules that do not have the correct license key are ignored.

There's a few issues here where it's not clear from your submission that you understand how things are meant to work

I can't say that I know everything, but unfortunately I can't do anything with this statement of yours. I would be very happy to learn something new from you. If you think you know something more in this context, please share your knowledge with everyone here who is in a technically similar situation and would like to improve their solutions.

@Jurij-Ivastsuk
Copy link
Author

Hello dear reviewer @aronowski @THS-on @steve-mcintyre, to answer once again your question about the prevention of loading unwanted modules: we use our own certificate to check the signatures of the modules to be loaded. Our kernel can also be delivered with new keys, if we need it, to prevent old modules from being loaded.

@aronowski
Copy link
Collaborator

I'll raise the issue during the next call.


If you think you know something more in this context, please share your knowledge with everyone here who is in a technically similar situation and would like to improve their solutions.

I myself was thinking about writing a guide, how to start the shim-related venue from scratch and implement appropriate measures for reasonable security, but taking into account each company's distinct situation, but there's no way I'll be able to account for so many different cases and limitations. Especially doing this in a free time after job hours and considering my own knowledge, experience and limitations.

I could think of something similar to the venue from this thread about smaller companies (one referred there as a fictional MyCompany) and their products deriving from the larger ones, but in the context of shim implementation and the whole bootloader chain (referred as Using downstream implementations from Canonical in your application). Still, as said earlier, I can see too many distinct cases to make it reasonable.

@THS-on
Copy link
Collaborator

THS-on commented Jul 22, 2024

@Jurij-Ivastsuk regarding the kernel module signing:

  • If you need the way to build kernel modules after the initial build, you can include additional kernel specific certificates per each build. Note that it is highly recommended to protect those keys in an HSM or similar. The main reason right now for accepting submissions without ephemeral key signing, is already products that have been signed in the past and when they have a sufficient mitigation strategy.
  • How your build is currently documented (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240601/Setup_for_module_signing.txt) it looks like the signing keys are in plain text on some build machine and emphermal, which is not sufficient.
  • Please elaborate on the implementation with the license key and module signing. I currently don't see the connection to the kernel module loading mechanism

@Jurij-Ivastsuk
Copy link
Author

@THS-on Hello, thank you very much for your comments. In our current kernel-mudule signing-setup we currently use two different certificates: 1. our own CA certificate and 2. kernel specific certificate for each compilation.
Can you please explain what you mean by “license key”. Is this a third variant of the certificate?

@THS-on
Copy link
Collaborator

THS-on commented Jul 25, 2024

In our current kernel-mudule signing-setup we currently use two different certificates: 1. our own CA certificate and 2. kernel specific certificate for each compilation

Can you update your documentation to reflect that? And also describe how your procedure with those keys are in detail?

Can you please explain what you mean by “license key”. Is this a third variant of the certificate?

This refers to your third point from this comment #362 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Problem with the review that must be fixed before it will be accepted contacts verified OK Contact verification is complete here (or in an earlier submission) extra review wanted Initial review(s) look good, another review desired new vendor This is a new vendor question Reviewer(s) waiting on response
Projects
None yet
Development

No branches or pull requests

5 participants