Provide IFD, Gbe and ME blobs

[osresearch]

The 12 MB x230 ROM image isn't flashable since it doesn't include the IFD and ME sections. This isn't normally a large problem for updates since flashrom-x230 won't touch the ME region, but it does lead to problems for initial install.

---

tlaurion hijacking top post here to give actual status in a glimpse.

• @osresearch accepted that heads distribute extracted IFD from an original Lenovo's rom, lower down in this issue here. The goal of this is to not generate the IFD ourselves and just unlock the one being present in the repository, on which BIOS region has been extended resulting of freed ME region, where the ME region is reduced to the minimal size possible. This results in a BIOS region of 11.5mb instead of 7mb, mitigating Quest to reduce firmware size #590 issue until help joins forces for boards with limited SPI flash space without the need of soldering...
• 3 collaborators decided to support the adventure of not waiting for legal approval, nor expecting Intel reply, and going forward and include the ME cleaned binary. Some efforts are still needed so the blob is downloaded directly from Lenovo, on which innoextract needs to be applied to remove the blob itself from local blob directory and do the required magic in a script having downloaded the blob from Lenovo website directly. Meanwhile the cleaned blob is present in tree with manual reproducibility instructions in xx30's blobs's directory README. Read the xx30 blobs README here for more details. Please show interest for being paid fixing this.
• Along the adventure, it was discovered and documented upstream that the GBE region needs to be present to have a functional ethernet card. This GBE blob could be totally generated. But would need some low level people involved. I'm not good at reading reference manual and implementing what is required. Please show interest for being paid fixing this.

Mitigations from dropping those binaries would be to:

• Claim right to repair laws?
• Claim Fair use laws?
• Generate those blobs from source code (see below)

So the result of the ongoing #703 PR is a PoC of that, which requires Heads to fake MAC address on boot to not have a MAC collision on the same LAN network segment if many Heads xx30 empowered machines are present.

Todo:

• Generate GBE blob to remove the GBE blob from tree. Seperate issue: Generate GBE blob from coreboot's bincfg tool #796
• Extract ME from Lenovo website (innoextract) and run me_cleaner on top of it to put resulting blob locally to remove it from tree. Seperate issue: Download, innoextract Lonovo website's downloaded ME blob and put it under blobs/xx30 from a module  #797

[merge]

This and #375 is really only one issue IMO :)

[tlaurion]

I think it might a false problem.

For both x220 and x230 and other 2 spi chips boards, the IFD should be provided inside of Heads (./initrd/etc/x230-layout.txt), accompanied with a flash script (/initrd/bin/flashrom-x230.sh) for Heads to be able to flash internally, the IFD containing the ME region to not touch, though that region could be reduced a lot, no?

@osresearch : would flashtool be able to deal with reading the IFD in flash dynamically and determine the real used ME region? I mean. On the x230, there is plenty of space when ME modules are trimmed and the free space is concatenated:

./me_cleaner.py ~/Documents/Firmwares/ME/x230/MeRelocated.rom -S  -O x230_me_cleaned.rom -r -t --extract-me extracted_me.rom
Full image detected
The ME/TXE region goes from 0x3000 to 0x500000
Found FPT header at 0x3010
Found 1 partition(s)
Found FTPR header: FTPR partition spans from 0xd00 to 0xcad00
ME/TXE firmware version 8.1.40.1416
Public key match: Intel ME, firmware versions 7.x.x.x, 8.x.x.x
The AltMeDisable bit is NOT SET
Reading partitions list...
 FTPR (0x00000d00 - 0x0000cad00, 0x000ca000 total bytes): NOT removed
Removing partition entries in FPT...
Removing EFFS presence flag...
Correcting checksum (0x2f)...
Reading FTPR modules list...
 UPDATE           (LZMA   , 0x04d207 - 0x04d3c5       ): removed
 ROMP             (Huffman, fragmented data, ~2 KiB   ): NOT removed, essential
 BUP              (Huffman, fragmented data, ~56 KiB  ): NOT removed, essential
 KERNEL           (Huffman, fragmented data, ~135 KiB ): removed
 POLICY           (Huffman, fragmented data, ~91 KiB  ): removed
 HOSTCOMM         (LZMA   , 0x04d3c5 - 0x05419f       ): removed
 RSA              (LZMA   , 0x05419f - 0x0593f5       ): removed
 CLS              (LZMA   , 0x0593f5 - 0x05eb8a       ): removed
 TDT              (LZMA   , 0x05eb8a - 0x065280       ): removed
 FTCS             (Huffman, fragmented data, ~18 KiB  ): removed
 ClsPriv          (LZMA   , 0x065280 - 0x065661       ): removed
 SESSMGR          (LZMA   , 0x065661 - 0x073f8b       ): removed
Relocating FTPR from 0xd00 - 0xcad00 to 0xd00 - 0xcad00...
 Adjusting FPT entry...
 Adjusting LUT start offset...
 Adjusting Huffman start offset...
 Adjusting chunks offsets...
 Moving data...
The ME minimum size should be 98304 bytes (0x18000 bytes)
The ME region can be reduced up to:
 00003000:0001afff me
Setting the AltMeDisable bit in PCHSTRP10 to disable Intel ME...
Extracting and truncating the ME image to "extracted_me.rom"...
Checking the FTPR RSA signature of the extracted ME image... VALID
Checking the FTPR RSA signature... VALID
Done! Good luck!

[tlaurion]

In this case, following the previous output from me_cleaner:

The ME region can be reduced up to:
 00003000:0001afff me

/etc/x230-layout.txt ME region could be modified from

00000000:00000FFF Descriptor     
00001000:00002FFF GBe                   
00003000:004FFFFF ME              
00500000:00BFFFFF BIOS

to: 00003000:0001AFFF ME

So we could use the output of me_cleaner to adapt the /etc/x230-layout.txt

Is there something i'm missing?

[merge]

no, I think you're right. The question is mostly whether we need the space and want to become dependent on me_cleaner being applied. Am 20. April 2018 22:41:23 MESZ schrieb tlaurion <notifications@github.com>:

…

In this case: ``` 00000000:00000FFF Descriptor 00001000:00002FFF GBe 00003000:004FFFFF ME 00500000:00BFFFFF BIOS ``` ME could be reduced to:` 00003000:0001AFFF ME` Is there something i'm missing?

-- Martin Kepplinger http://martinkepplinger.com sent from mobile

[merge]

@osresearch yes the 12m isn't flashable which is dangerous when flashing externally (first-time install) and people get it wrong (and the head wiki might even be misleading a little).

FWIW, specifically (only) for the X230, there's now a tiny project named Skulls that basically tries to hide as much as possible and simplify external flashing / unlocking the IFD (and applying me_cleaner). There is a script to install Heads (from Linux) too.

[merge]

this comment actually belonges in here more: #424 (comment)

[tlaurion]

@osresearch @merge @flammit :
The more I think about it, the more I start to believe that the same approach as Purism should be taken to download and neutralize ME binary blob before inserting it into Heads's 12MB rom, with a valid IFD being generated for that ROM.

The x230 flash.sh specifics could then overwrite the whole SPI instead of just the bios region.

What do you think? Modifying the scripts.

[tlaurion]

@merge @flammit @osresearch

Missing some pieces of the puzzle.

[merge]

why? what's the benefit when flashing internally on the system?

[tlaurion]

@merge more space, the possibility to update ME internally following probable changes in the ROMP and BUP modules and to have a valid IFD instead of only being able to flash the bios region since the layout file is not provided anymore. Flashrom 1.0 upgrade and related flash.sh modifications wiped it.

But basically, more usable space in SPI flash. Maybe a false need, again.

[tlaurion]

@merge said in #424 which is now merge:

We need a plan here :) we don't even know what to do yet

    First decision to make: we should agree that generating the 4M and 8M images of the x230 board is wrong in any case. We'll never be able to safely generate a flashable thing of 12M. And there's no point in having the splits other than flashing externally, which is always wrong.
    Also, flashing externally is only done the first time. And that's what the x230-flash board is for, which makes the x230 4M/8M split even more misleading. But then, x230-flash is not sufficient. At least touching the other chip's IFD is necessary.

    So, second decision to make: Do we want to support "bootstrapping" explicitely inside of Heads?
        If so, let's add scripts that unlock the IFD on the one chip and flash x230-flash on the other, and guide users through all of it.
        If not, drop x230-flash completely and simply add "unlocked IFD" to the requirements and add a script or (even better?) live-USB-image that flashes x230 from outside of Heads, which makes "bootstrapping" quite neat.

thoughts?

I think the x230-flash should deal with it internally. If i get things right from skulls, it means that x230-flash misses ifd tool to unlock the IFD from within. @merge?

[merge]

ifdtool's "unlock" has to be applied to the 8M chip's region. x230-flash (just like a skulls image) uses 4M (the 4M chip) and thus makes it possible to hardware-flash "read, unlock, write" to the 8M chip, which makes the process as easy as possible (as of now). That documentation is what's important.

x230-flash or x230 shouldn't include ifdtool in initrd. For future write-protection solutions we won't use IFD. We just need it to be unlocked once.

[tlaurion]

Interesting points posted in #564

@merge

I haven't really looked into the current state of the x230-flash "board". IMO it shouldn't need gpg at all. It's for bootstrapping only, right?

gpg is not there anymore, since GPG2 is too big and x230-flash is used only to bootsrap.
@shamen123 : the 12mb, internally flashed version is used for everything else then initial flashing of Heads.

Once the 4M "board" itself is in a nice shape (flash-gui, ... can be tested independently of the bootstrapping problem), we can think about how to help users with IFD unlocking. I can imagine a similar solution to https://github.com/merge/skulls/blob/master/x230/external_install_bottom.sh and good documentation. That script can directly be used actually... If it wasn't for "x230-flash" (a flash-only solution independent of the hard disk), we could even say "install Skulls first, and then come back here and flash Heads for your running Linux distro", but it's of course nicer to have a flash-only solution.

Unfortunately, following #517 and 4mb limit, having a flash-gui displayed in FBWhiptail or Whiptail only, won't fit in 4Mb.

I will look again on the Skull's external_install_bottom.sh script, since I can't still wrap my head about how we could deal with this one shot from the external, flashing system. Never actually played with Skull. Will try to find some time.

@shamen123

i did make write up notes at the time and they are lurking around on one of my machines somewhere, which may be of use to others. but that was based on 0.2.1 generic init, so its likely already out of date.

The process hasn't change a lot since that time. Chip that script in?

[merge]

Interesting points posted in #564

@merge

I haven't really looked into the current state of the x230-flash "board". IMO it shouldn't need gpg at all. It's for bootstrapping only, right?

gpg is not there anymore, since GPG2 is too big and x230-flash is used only to bootsrap.
@shamen123 : the 12mb, internally flashed version is used for everything else then initial flashing of Heads.

Once the 4M "board" itself is in a nice shape (flash-gui, ... can be tested independently of the bootstrapping problem), we can think about how to help users with IFD unlocking. I can imagine a similar solution to https://github.com/merge/skulls/blob/master/x230/external_install_bottom.sh and good documentation. That script can directly be used actually... If it wasn't for "x230-flash" (a flash-only solution independent of the hard disk), we could even say "install Skulls first, and then come back here and flash Heads for your running Linux distro", but it's of course nicer to have a flash-only solution.

Unfortunately, following #517 and 4mb limit, having a flash-gui displayed in FBWhiptail or Whiptail only, won't fit in 4Mb.

not yet at least :) but alright. There's no need to have a GUI for that one job.

I will look again on the Skull's external_install_bottom.sh script, since I can't still wrap my head about how we could deal with this one shot from the external, flashing system. Never actually played with Skull. Will try to find some time.

It's nothing special. Just a different 4M coreboot build, and thus it has all the bootstrapping-benefits that we need equally need to take advantage of for "x230-flash" here. Basically, you can even directly replace a pre-built "skulls" image with an "x230-flash" image and directly use skulls' "external" scripts - one per flash chip.

@shamen123

i did make write up notes at the time and they are lurking around on one of my machines somewhere, which may be of use to others. but that was based on 0.2.1 generic init, so its likely already out of date.

The process hasn't change a lot since that time. Chip that script in?

[tlaurion]

#605

[tlaurion]

@merge : There is not enough space anymore in the BIOS region (coreboot cbfs defined space) as it is to do anything else.

Playing around and applying this, including ifd and me in rom (while modifying flash.sh for it to reflash whole 12mb, not just BIOS region), there is now around 5mb flash that just appeared magically.

Looking at how Purism distributes its binaries through whole rom download for their board, there doesn't seem to be any problem redistributing ifd and me, which are basically included in their downloadable image, and extracted by their scripts.

@kylerankin ? @zaolin : Would there be a legal problem redistributing me.bin (98kb shrinked bin) and resulting ifd.bin directly from a x230 blobs directory on the repo?

What I propose:

• Create x230 blobs directory.
• Put cleaned_me.bin and reduced ifd.bin there
• Modify coreboot config so it builds with those dependencies included.
• Remove x230-flash.rom (unneeded. There will never have enough place in 4Mb to do something useful there anyway.)
• Modify x230 board config to generate top.rom(4mb) and bottom.rom(8mb) ROMs.

This way, reading roms would only be useful to have backups of them, but there would be no real reason to even run me_cleaner anymore. Just download GitlabCI artifact, and flash both chips.

x230 me_cleaner command generating blobs would ressemble this:
python me_cleaner.py -S -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin original_down_dump.bin
This neuter+deactivate(AltMEDeactivateBit) Intel ME leaving only BUP and ROMP, trim resulting space, remove IME access to other flash regions, and export both ifd.bin and me.bin that would be distributed and included in expended CBFS BIOS region under x230 coreboot config file.

What do you think? Playing around to include thin-provisioning-tools right now to measure OEM distributed image directly from distributed /boot with dm-verity at reownership. More space is needed and my quest to reduce firmware size has not gone very far.

@marmarek: That would be useful too if fwupd subproject goes farther in providing firmware upgrades from QubesOS over sys-whonix from dom0 (so not a Heads task, but QubesOS task.) Heads could look into directory for signed firmware file, validate against distributing key inside of rom, and compute new measurements based on firmware parts to implement forward sealing of firmware upgrades.. Any thoughts on that are also welcome.

Playing with this here

[osresearch]

The ifd.bin should be included; I don't think there are any licensing issues. me.bin I'm not sure about.

Intel has a very reasonable redistribution clause for their microcode updates that came about from the Debian pushback as well as for the FSP.

However, the ME team has not had any similar sort of large scale push them to change their license. Lots of us have asked, but I've never received a response.

Fetching it from a separate blobs repo would at least avoid tainting the main Heads tree, which seems to be the coreboot approach. This also avoids bloating up the tree with how ever many devices end up being supported.

[merge]

@merge : There is not enough space anymore in the BIOS region (coreboot cbfs defined space) as it is to do anything else.

Playing around and applying this, including ifd and me in rom (while modifying flash.sh for it to reflash whole 12mb, not just BIOS region), there is now around 5mb flash that just appeared magically.

I guess we should just do that. Might be the easiest way to keep the x230 supported. I might think that through sometime... (we need more space for vboot later anyways).

(not now but for the future, I'd be interested: would we then need a new vboot fmd file in coreboot too? how would that look like?)

[merge]

I tested this too and it works nicely: merge@4cbab95 5M of free space. we need to be careful how to make the transition:

We change the IFD, so flashing from inside of Heads won't (conveniently) work! I guess before we do a Heads-release, we can and should do this and force flashing from outside of Heads (ifd, me and bios regions (we can write a script) from userspace or 4M/8M from external hardware).

do you see any way we could avoid that "break" for users? IMO the state of the project would allow us to do it and say "please flash from outside of Heads, once now", when doing a release?

[tlaurion]

#590 can be closed when this is resolved.

[tlaurion]

Following #700 conclusions, without gbe blob, no ethernet.

[MrChromebox]

the GBE blob is necessarily system unique though, right? since it contains the MAC address?

[tlaurion]

(EDIT) Tracking of attempts:

• Generalize Gbe blob
• Analysis of Gbe

[Thrilleratplay]

@MrChromebox @tlaurion I came here via the me_cleaner ticket opened. There is a lot going on between a number of repos and tickets but did want to mention something relevant to the gbe blob.

Yes, the gbe blob from the x230 contains the mac address. Using hexyl on the gbe blob outputted from iftool , the address is at offset 00000000 and 00001000. Something I wanted to test was to modify the mac in the firmware and see if it still worked, like a low level mac spoofing. Also, quite a bit of the blob looks like it is padding and wondered how difficult it would be to reverse engineer the remainder.

EDIT: Oops, looks like I a little slow. @tlaurion seemed to have figured this out in #700

[tlaurion]

@Thrilleratplay : no its not fixed (not randomized). Its the actual mac address provided in gbe.bin blob that is exposed and present directly in the blob in that issue update, showing its addresses. I haven't tried to randomize it yet. Any success on your side?

[Thrilleratplay]

@tlaurion I'm sorry I don't understand what you mean by

no its not fixed (not randomized)

The MAC addresses seems to be in a fixed position at those hex offsets. My offsets match the ones you posted in #700 and they are also the same on my x220. I haven't tried modifying the addresses, I assumed there was a crc32 checksum or something that verified the mac's validity apart from the vendor prefix.

[tlaurion]

I meant I didnt try to modify it.

On May 12, 2020 10:20:06 PM UTC, Tom Hiller ***@***.***> wrote: @tlaurion I'm sorry I don't understand what you mean by > no its not fixed (not randomized) The MAC addresses seems to be in a fixed position at those hex offsets. My offsets match the ones you posted in #700 and they are also the same on my x220. I haven't tried modifying the addresses, I assumed there was a crc32 checksum or something that verified the mac's validity apart from the vendor prefix. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #307 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[tlaurion]

GBe attempts can be followed here

[tlaurion]

on IGB: missed that and looking at it https://libreboot.org/docs/gnulinux/grub_cbfs.html#changeMAC

[tlaurion]

@MrChromebox @PatrickRudolph

the GBE blob is necessarily system unique though, right? since it contains the MAC address?

Well, Leah seems to have resolved that 2 years ago: https://notabug.org/libreboot/libreboot/src/master/resources/utilities/ich9deblob/src/gbe/gbe.c

[tlaurion]

So, to be able to generate gbe blobs (ich9gen wast obsoleted in favor of bincfg), one would need to adapt the x200 bincfg spec file to ICH6 series

@9elements @3mdeb anyone?

[tlaurion]

#796 is fixed. So We have extracted and modified IFD in repo, GBE generated per bincfg, and ME neutered+deactivated. Now time to automated it #797 and this issue wil be resolved. Progress under #703

[tlaurion]

From #307 (comment)

The MAC addresses seems to be in a fixed position at those hex offsets. My offsets match the ones you posted in #700 and they are also the same on my x220. I haven't tried modifying the addresses, I assumed there was a crc32 checksum or something that verified the mac's validity apart from the vendor prefix.

@Thrilleratplay BTW it seems there is no verification made on GBE checksums.

Dunno if you are going to be quicker then me, but it seems that if we find a quick way to extract GBE from flash, modify at offset MAC defined in /etc/config (or under board config at compilation) we could hack the MAC, and reinject it prior of flashing, we could have either user defined MAC and persistent randomized MAC in Heads from a new menu entry.

[Thrilleratplay]

@tlaurion You are saying that the checksum within the GBE blob is not checked? And it does not effect OS level drivers? Huh.

I want to create a python script that is more flexible than bincfg that allows a command line parameter or template to set the MAC. If a patch file is generated for the gbe-82579LM config the MAC could be set on compilation with correct checksum. Although, if the checksum is not utilized, a small C program probably could do a binary replacement of the MAC in HEADS as it already has Flashrom. However, I am not a C developer and cannot write it quick/efficient/safe.

[tlaurion]

@Thrilleratplay my idea was to bash script around busybox' hexedit from output flashrom -p --ifd -I gbe -r /tmp/GBE.ROM following this insight.

I tried to manually hexedit only de:ad:c0:ff:ee at both offsets, and then flash /tmp/GBE.ROM back

But then, bringing up Ethernet by loading e1000e results in:
The NVRAM Checksum Is Not Valid.

I haven't checked further then this. But checksum is confirmed to necessary at least on x230 from that little test. Commented PR and crosslinked here.

[Thrilleratplay]

@tlaurion NVRAM Checksum Is Not Valid sounds familiar. I am trying to remember the errors I saw when testing/debugging the generated GBE blob. It is possible the checksum used to test that matched the original checksum. This is why I thought a small C program could do this and also calculate the checksum if needed.

[tlaurion]

@Thrilleratplay well i just modified de:ad:c0:ff:ee at both places with a valid one without changing the checksum here, so the error seems pretty valid to me.

Of course, if the whole region is computed in the checksum (on both regions which are exactly the same match the real checksum of what is expected, the error is legit here. I just had hopes that as reported in the PR referred above, it wasn't checked. But it clearly is, following reporte error.

I still think its possible to script something to extract the region, modify it and append checksum. Then duplicate as expected and reinject in GBE in backuo and flash back. But of course, a C program could do it as well.

Just thought a quick hack could be made and confirmed it can't as easily as expected. Checksum is needed.