QPMR header not found, unable to IPL

[madscientist159]

We're running into an issue on our Talos systems with DD2.2 and the latest hostboot / HCODE master trees. Our systems will not IPL, and throw errors in ISTEP 15 related to the QPMR header. Our internal diagnostics and @dcrowell77 are in agreement that the HCODE in this repository is either out of sync with the rest of the firmware, or simply broken.

Example output:

43.84925|================================================
 43.86459|Error reported by hwpf (0x0900) PLID 0x900000A6
 43.86611|  HW Procedure generated error. See User Data.
 43.86611|  ModuleId   0x01 MOD_FAPI2_RC_TO_ERRL
 43.86612|  ReasonCode 0x090f RC_HWP_GENERATED_ERROR
 43.86613|  UserData1  RC value from HWP : 0x0000000000cb102a
 43.86614|  UserData2  <unused> : 0x0000000000000000
 43.86911|------------------------------------------------
 43.87778|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 43.87785|  HWP Error description      : Failed to update QPMR Header region of HOMER. Note: 0xFFFFFFFF in field MAX_ALLOWED_SIZE is an invalid size suggesting that image section is not found
 43.87786|------------------------------------------------
 43.89450|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 43.89656|  FFDC                       : EC_LEVEL
 43.89657|     22
 43.91391|------------------------------------------------
 43.91393|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 43.91394|  FFDC                       : CHIP_TYPE
 43.91394|     05
 43.91397|------------------------------------------------
 43.91399|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 43.91400|  FFDC                       : MAX_ALLOWED_SIZE
 43.91401|     0000001D
 43.91404|------------------------------------------------
 43.91406|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 43.91660|  FFDC                       : ACTUAL_SIZE
 43.91660|     00000128
 43.91664|------------------------------------------------
 43.91816|  Callout type             : Procedure Callout
 43.91816|  Procedure                : EPUB_PRC_HB_CODE
 43.91817|  Priority                 : SRCI_PRIORITY_HIGH
 43.91818|------------------------------------------------
 43.91818|  Hostboot Build ID:
 43.91819|================================================
 44.73707|================================================
 44.73708|Error reported by hwpf (0x0900) PLID 0x900000A6
 44.73709|  HW Procedure generated error. See User Data.
 44.73710|  ModuleId   0x01 MOD_FAPI2_RC_TO_ERRL
 44.73710|  ReasonCode 0x090f RC_HWP_GENERATED_ERROR
 44.73711|  UserData1  RC value from HWP : 0x0000000000cb102a
 44.73712|  UserData2  <unused> : 0x0000000000000000
 44.73713|------------------------------------------------
 44.73714|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 44.73720|  HWP Error description      : Failed to update QPMR Header region of HOMER. Note: 0xFFFFFFFF in field MAX_ALLOWED_SIZE is an invalid size suggesting that image section is not found
 44.73721|------------------------------------------------
 44.73723|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 44.73724|  FFDC                       : EC_LEVEL
 44.73724|     22
 44.73727|------------------------------------------------
 44.73729|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 44.73730|  FFDC                       : CHIP_TYPE
 44.73731|     05
 44.73733|------------------------------------------------
 44.73735|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 44.73736|  FFDC                       : MAX_ALLOWED_SIZE
 44.73737|     0000001D
 44.73740|------------------------------------------------
 44.73742|  HwpReturnCode              : RC_QPMR_HDR_BUILD_FAILURE
 44.73742|  FFDC                       : ACTUAL_SIZE
 44.73743|     00000128
 44.73746|------------------------------------------------
 44.73747|  Callout type             : Procedure Callout
 44.73747|  Procedure                : EPUB_PRC_HB_CODE
 44.73748|  Priority                 : SRCI_PRIORITY_HIGH
 44.73749|------------------------------------------------
 44.73749|  Hostboot Build ID:
 44.73750|================================================
 45.69712|System shutting down with error status 0x900000A6
 45.69715|================================================
 45.69716|Error reported by istep (0x1700) PLID 0x900000A6
 45.69903|  IStep failed, see other log(s) with the same PLID for reason.
 45.69903|  ModuleId   0x01 MOD_REPORTING_ERROR
 45.69904|  ReasonCode 0x1703 RC_FAILURE
 45.69905|  UserData1  eid of first error : 0x900000a60000090f
 45.69906|  UserData2  Reason code of first error : 0x0000000200000f01
 45.69907|------------------------------------------------
 45.69907|  host_build_stop_image
 45.69908|------------------------------------------------
 45.69909|  Callout type             : Procedure Callout
 45.69910|  Procedure                : EPUB_PRC_HB_CODE
 45.69911|  Priority                 : SRCI_PRIORITY_LOW
 45.69911|------------------------------------------------
 45.69912|  Hostboot Build ID:
 45.69913|================================================

This is blocking our DD2.2 demos at the Summit. Migrated from open-power/hostboot#129

[dcrowell77]

Just as an experiment can you try using the hcode from hostboot-binaries?

[madscientist159]

@dcrowell77 We'd have to bring up a new build tree on a different server, as the HCODE build is integrated into our version of op-build. Also that won't help us with the demo, as one of the key points we're making is "this is running on full open source, no binary-only firmware components.".

If it'll help narrow things down, I can try to run a test against the binary versions after the Summit.

[madscientist159]

@dcrowell77 One thing I did notice, picking through the HCODE binary versus the binary we generated, was that the internal versions in the two binaries don't line up at all. The binary-only version is "hw022318a.911", while the source version on GitHub is "hw030918a.910". Without access to the internal HCODE tree I have no way to know if a breaking change was implemented after "hw022318a.911".

[dcrowell77]

I can tell you that hw022318a.911 is at 46c42b6f3bbf59bfa28d1231d4ef383f375fa28c but it looks like the commit numbers aren't quite matching up... (obviously still working out some kinks here).

commit 46c42b6f3bbf59bfa28d1231d4ef383f375fa28c
STOP: Fix NDD20 dual cast workaround lose second core wakeup

[dcrowell77]

It looks like the version of hcode that made its way up to github is missing some support for DD2.2. @jahunsbe is working on making and testing a fix now.

[rjknight]

as a quick test you could update tools/build/rules.dir/chips.env.mk in the hcode repo
you would need to modify line 31 to look like this p9n_EC += 20 21 22

[madscientist159]

@dcrowell77, @rjknight With some adjustments for the Talos plaforms, we can now IPL on DD2.2. There are some new issues that have appeared (probably due to the hostboot, etc. updates required), most notably open-power/op-build#1974, but at least this is a good starting point.

Thanks all for tracking that down so quickly!