WiP (NOT FUNCTIONAL): xx20 blobs extraction scripts fixes

[tlaurion]

me_cleaner called to neuter, deactivate, trim ME and output reduced ME under blob dir, modify both ME and BIOS regions accordingly to be able to accept CONFIG_CBFS_SIZE=0x750000 defined under coreboot configs (attempt to fix #870)

[tlaurion]

@userongihu does this fixed the initial issue you had under #870?

Per testers defined here : @SebastianMcMillan @techge @eganonoa @BlackMaria @shamen123 @Thrilleratplay @userongihu

Can at least one x220 and one t420 owner/tester report working state so we can merge change?

[snmcmillan]

I do not have access to my T420 for testing until around the 25th of November. However, if it works on the X220, I highly doubt that it'll not work on the T420, since they use the same flash layout and they can use identical blobs.

…

On Sun, Nov 1, 2020, 15:21 tlaurion ***@***.***> wrote: @userongihu <https://github.com/userongihu> does this fixed the initial issue you had under #870 <#870>? Per testers defined here <#692> : @SebastianMcMillan <https://github.com/SebastianMcMillan> @techge <https://github.com/techge> @eganonoa <https://github.com/eganonoa> @shamen123 <https://github.com/shamen123> @Thrilleratplay <https://github.com/Thrilleratplay> @userongihu <https://github.com/userongihu> Can at least one x220 and one t420 tester report working state so we can merge change? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#877 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNTUNHLASFRY2G7XPFUVI3SNXGPPANCNFSM4TEF5V5A> .

[techge]

I currently have no access to my x220, but should get it end of the week. I'll try to test it until end of week. If somebody is faster, feel free to test it :)

[ghost]

The x220 works now flawlessly.It boots into heads and then choosing Qubes
the OS starts as supposed.Great internet connection.Thanks to everybody that helped me to get the x220 up and running with Heads.

[ghost]

I also tried to boot some other distros,that I put on an USBdrive along with the
.sig or .asc file.They are all booting,connecting to the internet and working as supposed.

[tlaurion]

@userongihu : was that linked to the changes outlined here which tries to replicate the commands you typed manually?

This is what is at stake here, trying to figure out if that changeset is mergeable upstream.

[ghost]

@tlaurion,what did the trick for me was:

1. The new flashrom command which was
   "flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/"
   before using this command I couldn't get the GPGkeys to persist reboots.

2)Using this command to disable ME and liberate the space to be reused by Heads and shrinking IFD also:
python me_cleaner.py -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin original_dump.bin
3)Making an "OEM reset",because it took care of each and every step as far as the steps are concerbed which need to be done after flashing Heads onto the chip.

I would like to suggest to emphasize in future install-documentation,having in mind inexperienced users like me,that an "OEM reset" makes Heads do all the setup-work,so that you end up with everything setup and working out-of-the-box.Because only after the "OEM reset" I could get the OS to boot.

[tlaurion]

@userongihu @alexmaloteaux I need someone to test the extract scripts in this pull request, please.

I do not have x220 nor t420 boards and we want boards requirements to be filled with the lesser friction for end user, so doing those steps manually is not really acceptable, so please test and report.

I will add another commit now so that the t420 blobs dir extract.sh contains those changes.
I also understand that the same steps will impact #830 @SebastianMcMillan

[tlaurion]

Testing build right now for the xx30 ME blob being downloaded from Lenovo: https://app.circleci.com/pipelines/github/tlaurion/heads/598/workflows/202ad9c3-7e06-45d2-ac7e-cd3292ea596b/jobs/646

Pertinent changes (not yet clean):
master...tlaurion:x230-external-flash

1. My approach is lazy for now and just asks the CI to call the new script

2. The script that downloads the blob, extract and neuters is here

3. While GBE and IFD are provided in blobs dir.

For the extraction script to operate on local backup: here is my take. Tested and working.

@SebastianMcMillan @techge @eganonoa @BlackMaria @shamen123 @Thrilleratplay @userongihu : please take, adapt, and do a PR for xx20. I can't test for you guys.

[tlaurion]

@SebastianMcMillan @techge @eganonoa @BlackMaria @shamen123 @Thrilleratplay @userongihu @MrChromebox :
Transposable PoC to xx20 devices result is showed successful #703 (comment)

[tlaurion]

• ME latest executable to which apply and redo process is here

• Instructions and process I took is here

  • binfcg patch to generate GBE with binfcg is here - should be the same for xx20 if I understood well. (Thanks to @Thrilleratplay !)

  • download_clean_me.sh script to be modified is here

  • extract.sh working script is here (should be near drop in replacement to xx20 (x220, t420 and others) which should probably combined under xx20 blobs dir since probably common to xx20's.)

• ifd.bin extracted from extract.sh can be dropped in in tree in PR.

• gbe.bin generated by bincfg per instructions can be dropped in tree per PR.

@SebastianMcMillan @techge @eganonoa @BlackMaria @shamen123 @Thrilleratplay @userongihu @MrChromebox :
Voila!

[tlaurion]

@techge the post above shows steps to have neutered me and expended maximal BIOS region. Consequently coreboot cbfs region can be set to maximal size freed in meand redistributed in BIOS region just like in #703.

Does that makes sense?

[tlaurion]

corna/me_cleaner#343 (comment)

[Thrilleratplay]

@tlaurion There is something odd about the ME file in xx20 series Lenovos verse the xx30. Given the name, I think ME7_5M_UPD_Production.bin is an incomplete update file. ME8_5M_Production.BIN has the $FPT header (hex value 24 46 50 54 found at offset 0x10) that me_cleaner looks for. However, the closest value in ME7_5M_UPD_Production.bin to this is $MN2 (hex value 24 4d 4e 32 at offset 0x1d).

EDIT: yeah, this is not a full ME file. ME_cleaner may need some modifications to strip down an ME update.

-------[ ME Analyzer v1.30.0 r100 ]-------

File:     ME7_5M_UPD_Production.bin (1/1)

Family:   ME
Version:  7.1.91.3272
Release:  Production
Type:     Update
SKU:      5MB
Date:     2017-04-07
Platform: CPT/PBG
BList 0:  <= 7.0.10.1203
BList 1:  <= 7.1.13.1088
Latest:   Yes

I'll do some more digging.

[tlaurion]

This is informative:

FWUpdate Updates (UPD): FWUpdate images (UPD) are partial RGN/EXTR firmware regions which contain only ME CODE without any DATA. They are created and used only by Intel's FWUpdate tool. Thus, they can neither be opened nor configured by Intel Flash Image Tool (FIT). Never flash UPD images via anything other than Intel FWUpdate tool. UPD images are not needed for 7-series (ME 8) or newer systems. However, all 6-series (ME 7) or older systems must use UPD images in order to initiate a ME firmware update. Thus, at section B1 below, only RGN/EXTR images are provided for 7-series (ME 8) or newer systems and only UPD images are provided for 6-series (ME 7) or older systems.

I will go into making extract.sh script working with backups and will personally put aside download_clean_me.sh for xx20.

[tlaurion]

I tested the 3 7.1 images available here (which are all UPD, so capsules, not full ME regions, which needs to be flashed with proprietary tool FWUPD)

corna/me_cleaner#152 (comment)

No joy!

[Thrilleratplay]

@tlaurion I am curious if there is enough information in the ME updates to piece together a functional shrunken ME partition. The ME_cleaner output suggested that the remaining partitions may not have data. That will take some time and testing. I want to get IFD generation working first and I'll attempt a "Frank-ME-nstein".

[tlaurion]

@Thrilleratplay i'm not convinced generating ifd is really a priority.
I have experienced flashing xx30 ME for which version mismatches exist between IFD/ME without any error.

An alternative approach, would be to host ifd.bin and generated gbe.bin (as currently done under #703).
The real problem, for xx20 right now, is to be able to extract/download ME from an online source (since it cannot be downloaded from Lenovo website as for xx30) so that that boards (x220, t420 ,etc) can be added to CI without hosting the ME binary under heads repo, which was a no go.

#703 has two scripts, one to extract blobs from backup (gbe.bin, me.bin and ifd.bin) while generated gbe.bin and modified ifd.bin is already in PR. Another script from #703 permits the user to not loose his MAC from laptop (extract gbe.bin) and extract me.bin and ifd.bin to replace the ones provided in repo. From my perspective, right now, this is more then enough to dodge legal problems while permitting CI reproducibility.

[Thrilleratplay]

@tlaurion Ok, I'll see what I can do with ME on the xx20 series.

[ghost]

@tlaurion ,haven't been on github couple of days.If you still need me to test something,just leytme know.The x220 works really great.Qubes is very RAM demanding,so I changed to Debian 10 ,signed the /boot files and the x220 runs like supposed.

[Thrilleratplay]

@tlaurion I have a proof of concept to automate the generation a cleaned minified me.bin using the ME7 update from Lenovo. It can be found here. A few notes, it requires python3, innoextract and wget installed. Before compiling, run blobs/xx20/download_parse_me.sh. I have tested and booted on an x220.

I need to document what it does, break down of the partitions, sources, where I stole code from, etc. I'll eventually create a repo under my name with all of this information. The script itself is created only to be used with Heads so let me know if you want any changes to the script. The branch was modeled after #703 as that is still a work in progress, I just committed the basics.

[tlaurion]

@Thrilleratplay awesome!

[tlaurion]

Continues over #912