Talos II: flashrom, flashtools, PNOR handling #1222
Conversation
|
Regarding access from host OS, I think that nothing blocks accesses to PNOR through LPC space (0x80060300F0000000, + 64MB), but software would have to explicitly use cache inhibiting instructions. You would still get the image with ECC and would have to parse partition headers, though. That being said, I think "the proper way" would be to use OPAL calls. |
I thought maybe it's reserved, but I probably confused it for HOMER.
A user-space software can use such instructions on a mmap of
Yes, that's what mtd driver does on PPC. |
Fair point, these are hypervisor instructions so in some cases even OS wouldn't be able to do it. I guess this should be doable through storage control bits (WIMG) in page tables, but I haven't looked at that - it wasn't necessary for coreboot stuff 🤷 |
BMC: Disconnection from BMC at this point. Reconnection: Since on workstation with Dasharo/dasharo-issues#193 unresolved here, there is output on Workstation's VGA, but no usb keyboard support. So each keypress here with commit 73bc621 generates a Basic testing: Agreed that Testing: I guess better safe then sorry here, but really not sure of a case where verifying the rom internally is really useful.... Testing will be continued with server board, since testing interactions from menus is not really possible on workstation as of now. |
|
Coming back from Dasharo/dasharo-issues#193 to continue tests of flashrom/pnor addition under flashtools. From my understanding, coreboot calls zImage.bundle with hardcoded boot options from .config compiled in zImage.bundle. So just pflashing the server board's zImage.bundle should do. Let's see.... So here we have bootblock and coreboot artifacts from workstation, and zImage.bundled from server. Let's push only the required changes with rsync over BMC: Disconnects here. Applying precedent patch Attempting injection of my public key The flashing prompt on screen stays at 0%, leading to think that what is being written to flash is way more then just the added keyring and trustdb... The log under /tmp/flashrom-xxxxx.log contains a log of strings, linked to dynamic output produced with flash.sh dynamic progress output. It ends with So let's see, I guess.... Issuing reboot through Heads with Leads back to Heads menu, which gives different output. Seems like there is a public key in keyring. Let's redo without patch, again from recovery shell so we can have error output to console again, and without my patch. Manually testing flashrom Hmmm.... Ok... Will redo and review the scripts, not sure where flashrom exits 1 |
|
For good measure, since I think about it, output of cbeme and dmesg |
initrd/bin/cbfs-init
Outdated
| # We're running before /tmp/config exists, but flash.sh needs it, so generate its initial version | ||
| combine_configs | ||
|
|
||
| echo "Reading ROM..." |
There was a problem hiding this comment.
@SergiiDmytruk That won't fly.
Even if flashrom was meant to read only once to /tmp (major TOCTOU for everything else that relies on that backup later on) this would mean adding ~30 seconds at each boot?!
~ # time flashrom -p linux_mtd -r /tmp/tst
flashrom on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org
Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
real 0m 26.32s
user 0m 0.08s
sys 0m 25.91s
Since flashtools is already modified, why cbfs cannot be modified to be able to search for base and not rely on flashrom and then pnor to get cbfs content?
There was a problem hiding this comment.
By the way, scripts read ROM 3 times and reading 64 MiB from flash 3 times takes noticeable amount of time.
Adding 30 seconds at each boot (if flashrom calls under flash.sh were replaced to be called only once for reading backup instead of the current 3x which is currently a bit less than 90 seconds now) cannot happen at each boot. 30 seconds to backup rom, add content in cbfs section and then flash back would be acceptable. But extracting cbfs at boot should not take 30 seconds.
Can't cbfs be modified under flashtools to find/interpret content through pnor, combining the functions?
@SergiiDmytruk what are the alternative approaches ?
|
I forgot to mention that
See comments at the top, reading flash through
Well, I was also thinking of abusing fmap of No, wait. We can probably just use |
SergiiDmytruk
force-pushed
the
flashrom
branch
from
331ef89
to
50ca595
Compare
|
Reading PNOR using If this is functionally OK, I'll clean up commit history in |
|
@SergiiDmytruk here is my patch against flash.sh to have good rom size detected (with progress bar), modifying logic to that reading the logs don't break prior of having Reuse and adapt as needed. EDIT: added under #1230 |
|
@SergiiDmytruk tested #1230 successfully, and https://app.circleci.com/pipelines/github/tlaurion/heads/1223 as well under #1230 (comment). Waiting for better implementation/review there but this works with flash.sh fix to be able to inject public key in firmware without flash.sh failing. |
|
@SergiiDmytruk : It still seems that Heads internal firmware upgrades won't be possible, though, even if Heads was stitching the files together, and consequently, that no fwupd support will be possible in the future? For example, from within Heads: |
|
@SergiiDmytruk |
|
It is not possible to use the Options->GPG options-> Replace gpg key option giving the same error: Note also the repetitive additional mtd_size messages above. |
|
BMC Heads: Reboot. Heads again. Gpg Options-> List GPG keys in your keyring: OK |
|
@SergiiDmytruk Also note that after key injection upon reboot, the following is showed on console: |
|
@SergiiDmytruk Sending you 2 files under single tarball over matrix:
Impossible to use talos.gpg as mboxctl file backend to replace keys through Heads. Anything attempting to flash, keeping states would result in |
Not sure about fwupd, it will need some handling of PNOR/ECC to work, but otherwise it should work as long as there are no bugs.
I don't think using MTD should have such an effect.
That's debug messages I forgot to remove.
Don't see anything unusual there except for Thanks, I'll see what might be going wrong on updating flash more than once. |
Flashing the same flash image content backed up before (-r for backup, -c flag for clean flash without persistence of settings in cbfs) took 1m49 here. @SergiiDmytruk Everything leads into thinking that the whole flash gets overwritten, even if flashing exactly the same ROM image that was backuped previously, as seen in output (and why I provided that output), questioning mtd. If each read and verify takes around 30 seconds each, flashing and verifying around 30 seconds each, then we are at around 1m49s adding the operations. Where if the flashing operation of the same ROM was to be applied only on blocks that changed, it would be expected that the operation be reduced to less than a minute and no flashing happening at all, since the write operation would not happen since the output from flashrom comparing backup and to be flashed image would see that the to be flashed image is identical. Then there is no explanation for the additional 30 seconds for the flashing operation. Try in your tests. You will get the same results. There is a flash operation happening, even though flashrom reports that content is identical, after the fact, as opoosed to internal programmer. Also, writing into cbfs being the result of a rom backup, cbfs injection of a keyring, trustdb and possiblity a config.user overlay is supposed to be a change of a couple of kilobytes. So if mtd was only applying the changes, as the internal programmer does, the flash operation itself should take maximum 2-3 seconds, while verifying should take another 30 seconds. But this is not what is observed, hence me asking if mtd writes each blocks, not only the differences. Bug or feature: I do not know. |
First thing I witness : |
Question here is not just about fwupd, but more about how users are supposed to update Heads without loosing their settings (keyring, trustdb, config.user overlay which is copied to new firmware image under Heads when persisting setting option is used under Heads). As of today, we are testing and this is ok. But every other board under Heads produce a final ROM image that Heads configuration states FLASHROM_OPTIONS to properly handle regions to be flashed from a whole firmware image. For "Legacy boards" (ex: x230-flash+ x230 vs x230-maximized), the FLASHROM_OPTIONS clearly states that only the BIOS region (coreboot+linux payload) is to be written in IFD region, where maximized boards contain everything stitched together to be wholly flashed (with flashrom only applying block differences throug erase+write). As of today, this is done through BMC to flash a clean ROM, overwritting bootblock, hostboot and petitboot regions, leaving other regions untouched, without setting persistence for testing. But the desired outcome, as for fwupd prior work, is to drop a firmware under /boot where Heads as direct access to flash. Which means that the current use case is missing a final packed image (just like legacy roms, faking ifd, not containing ME nor GBE, and only specifying BIOS region to be flashed, giving BMC flashing equivalent: HBB HBI and BOOTKERNEL regions ( Otherwise we are otherwise saying here that future fwupd in OS will be able to write directly to mtd (equivalent of passing iomem=relaxed to kexec'ed kernel at all time?) where past work and pending PR under Heads is for Heads to be able to deal with ROMs that landed under /boot, and there, the mechanics of settings persistence through cbfs is supposed to copy those settings into the ROM to be flashed (coreboot region inside of cbfs) to be able to flash the final combined result. Either a properly packed Heads ROM + proper FLASHROM_OPTIONS are expected to instruct flashrom to flash specific mtd regions (partitions) with bootblock, coreboot and zImage.bundled from flashrom obtained backup (just like pflash is doing it from bmc), or as I understand it, Heads internal upgrade functionality (settings persistence of gpg keyring, trustdb and config.user overlay) will not be possible as for other boards? |
That's the usual thing as far as I can tell. I see it all the time. I see the same:
MTD is not accessed through memory directly, kernel communicates with skiboot using OPAL API and asks it to read/write flash.
flashrom isn't capable of dealing with PNOR on its own, either need to wrap it to read/write needed sections and post/pre-process them or write a separate tool that can do it. Part of this PR reads and writes via
time flash.sh -c /tmp/backup.romtime flashrom $CONFIG_FLASHROM_OPTIONS -w /tmp/backup.rom
|
SergiiDmytruk
force-pushed
the
flashrom
branch
from
50ca595
to
a183b29
Compare
Two more places needed |
SergiiDmytruk
force-pushed
the
flashrom
branch
from
a183b29
to
8a2cebf
Compare
hashes from tested x230-htop-maximized hashes.txt file: |
|
@SergiiDmytruk #1230 just got merged, please rebase on master. |
SergiiDmytruk
force-pushed
the
flashrom
branch
from
8a2cebf
to
2e766c2
Compare
|
Rebased.
I think I didn't do this because
How about |
The idea here is more related to consistent testing and knowing what is being deployed in case of Talos. For other boards, the images being self contained, it is not really an issue. But for Talos, 3 files are required, two of which are versioned and zImage.bundled is not. It is easy to report issues while testing a older zImage.bundled. They should all have the same file versioning scheme.
Not against. If hashes.txt is being bundled in tarball, next steps could be to use the hashes.txt in the tarball to further verify firmware integrity as well. Similar changes exist in Nitrokey's fork, creating npf firmware upgrade packages and having flash.sh modified to deal with those here: master...Nitrokey:heads:nitropad. Could be upstreamed if that is useful to this usecase as well. That is the easiest path vs padding those files into correctly constructed and interpretable flashrom images? |
Even if we assume that offsets of PNOR partitions are fixed and provide an appropriate fmap file/ifd |
@SergiiDmytruk : then the above would be my recommendation. Maybe its time to provide compressed, versionned firmware packages for all boards if we go into that direction. I might generalize what will be done for Talos II to other boards as well. Having the hashes.txt included in the tarball with localized hashes might be ideal to go forward as well, providing an easy way to validate hashes from within Heads against /boot dropped firmware package (see #613 ), and then that firmware package being part of detached signed digest would also be a nice step forward; permitting him to validate integrity of binaries/libraries/scripts against a known latest firmware package dropped under /boot, while also permitting reflashing the same firmware, expecting the sealed measurements to be exactly the same (TOTP/HOTP being still congruent) on next reboot, producing the same TPMTOTP/HOTP challenge. The flash.sh script could even check for expected checksum of images under hashes.txt prior of permitting flashing. |
|
Just to remind that this MR originated from issue reporting lack of flashrom support: Dasharo/dasharo-issues#190 After some struggle we implemented some solution, even though from the start it was said that the internal programmer would not work for the POWER9. Now we are discussing some versioning, even though currently no boards in heads implements that (?). I'm already confused what's left here to mark it as I take this comment as the final needs.
Looks merged already.
Looks like we agreed on how it can be done and @SergiiDmytruk will proceed?
I'm not sure where we are standing there |
Sure.
Packing 3 files into a tarball and doing things like for nitropad. Also need to update new |
@macpijan: if we look at artifacts currently produced by board configs under master https://app.circleci.com/pipelines/github/osresearch/heads/457/workflows/52255961-041d-41af-a1db-d73ff7772ffa/jobs/4964/artifacts Talos boards currently produces BMC-only flashable artifacts:
My request here is to have consistent versioned filenames so that parts are clear to belong to the same board and same parent commit, as all other boards produced ROM artifacts. Minimally:
I was thinking of something like the following to match/ease understanding of dasharo flashing instructions and local filenames being easy to target for testing reports/reproducibility of issues:
As a result, a user could easily upload a "heads-tests" directory containing firmware parts to be flashed through BMC, understanding what are HBB HBI and BOOTKERNEL files and quickly be able to tell from which commit they are flashing the parts from:
Correct. With #1230 merged, Heads doesn't prevent Talos to use flashrom successfully through flash.sh, as expected from this PR content without it merged. Otherwise, flashing failed after verification of flash content.
It is agreed that coreboot+heads/hostboot+heads/coreboot+petitboot/hostboot+petitboot can be tested without flashing that initial testing from end users per https://docs.dasharo.com/variants/talos_2/installation-manual/#testing-firmware-images-without-flashing. Heads support always required persistence of pubkey+trustdb+config injection/extraction through cbfs, which always required flashrom+cbfs integration, otherwise Heads is not Heads. With TPM, Heads measures and seals those measurements as part of TPMTOTP/HOTP. I'm still confused today on how Heads can be considered Heads without flashrom+cbfs. Without a TPM to measure, seal and remote attest firmware and having a setuped TPM released disk encryption key to boot OS, how a platform can be considered fully supported through Heads. Going back to the part of this MR into that fulfilling Heads support: coreboot.rom, coreboot.bootblock and zImage need to be packed somehow together to be downloadable and flashable from within Heads
@SergiiDmytruk : If Heads/dasharo produced artifacts/documentation per board configuration are consistent to BMC memory/physical flashing/Heads internal flashing, then this PR can be merged and Dasharo/dasharo-issues#190 closed as complete. As of today, there is no internal upgrade possible from within Heads. So no real Heads support yet. |
SergiiDmytruk
force-pushed
the
flashrom
branch
from
2e766c2
to
ba98b2d
Compare
|
Seems to work. Loaded server board, changed configuration, then successfully updated to workstation board retaining configuration. |
|
SergiiDmytruk
force-pushed
the
flashrom
branch
from
ba98b2d
to
fff474a
Compare
Logs again. Made the message more sensible and made it appear only with |
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
This makes output suitable for use via Heads' menus. Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
SergiiDmytruk
force-pushed
the
flashrom
branch
from
fff474a
to
472ca6f
Compare
|
@SergiiDmytruk par submitted to flashtools? |
|
I thought you'll approve this one, then I'll send flashtools PR to be able to update |
Power is different from x86 in several ways:
flashtoolsmodule here uses 3mdeb's fork, will send changes there after some review.With these changes settings are loaded on startup, settings can be stored from the menu.
By the way, scripts read ROM 3 times and reading 64 MiB from flash 3 times takes noticeable amount of time.Edit: fixed under #1230