Clarification on TigerLake S3 suspend support with old FSP and ME Firmware

[fnerdman]

@jackpot51 In response to your in depth comment on the S3 suspend situation on TigerLake-U (see below) I would like you to clarify the following:

B0 Stepping + old FSP and ME Firmware = S3 support
B1 Stepping + new FSP and ME Firmware = no S3 support
B1 Stepping + old FSP and ME Firmware = ?

Have you tested that? I think a lot of customers would be fine with outdated Intel firmware and be able to use s3+disable me

The answer for the TigerLake delays is highly technical. S3 suspend mostly worked on TigerLake-U B0 stepping and an earlier FSP/ME. We were on track to ship product with disabled ME and S3 suspend. Eventually we ran into S3 suspend hangs with updated FSP and ME, and with CPU's with B1 stepping. These hangs were intermittent and I spent about a week investigating them. It was discovered, deep in one of the Intel documents, that TigerLake-U no longer officially supported S3 suspend. References to S3 suspend were still prevalent in many other documents, making it take longer to realize this.

When it was realized, work was done to implement S0ix suspend. After the initial implementation, it was realized that a disabled ME reduces S0ix residency. We sent a communication indicating to customers that the ME would be enabled and sometime later an optional firmware would be offered disabling the ME but increasing suspend power draw.

A hang was then identified with certain Linux distributions, related to switching to S0ix. This hang can be difficult to replicate, and is such a serious hang that debugging information is not printed when it happens. Last week the cause was identified, a single bit set in the Thunderbolt Runtime D3 ACPI code from coreboot that is also no longer valid on CPU's with B1 stepping. This bit was undocumented, but a careful inspection of the FSP revealed the difference between coreboot and the reference BIOS.

We now had firmware that successfully replaced S3 suspend for S0ix suspend, and would work on all CPU steppings of TigerLake. Another communication was sent identifying that units would start shipping this week.

TLDR: We believed that the TigerLake products were ready to go with disabled ME and S3. Updated firmware from Intel and a new CPU stepping forced us to enable the ME and implement S0ix suspend. This work took two weeks, during which we sent communications to any delayed orders. I expect customers to be mad at us for this, and hope to make it up to them with quality product and good support. In my nearly four years at System76, this has been an incredibly strange situation that we hope will never happen again.

[fnerdman]

For those interested, I've compiled system76 coreboot with s0ix disabled and s3 reenabled, with both the latest firmware blobs as well as the oldest I could find in the models/lemp10 directory. On a tiger lake clevo l140mu, stepping 0x01 (B1), both resulted in a system freeze after wakeup from s3 suspend.

[jackpot51]

@lead4good there is no firmware configuration that can get S3 functional on B1 stepping, as you saw.

[fnerdman]

@jackpot51 do you have any hopes of S3 becoming functional again in further FSP/ME updates? Do you think it's a hardware issue or rather only software related? The lemp10 is such a great device, I even got TB4 running now but without S3 its just not valuable to me.

[WatchMkr]

Intel is sun-setting S3. S0ix is its replacement. AMD looks like they’re doing the same. We all have to transition.

[pbaykalov]

I am debugging my problems with S3 suspend on a Tiger Lake U laptop not made by System 76 with stepping 2 as displayed in CPU-Z (it probably means it's a B1 stepping, right?).
Do I understand it correctly that if I have any kind of problems with S3 I should just accept it and set my OS up for S0ix instead?

S3-Hot is not supported by the processor

Is it this quote which Jack was talking about when saying that S3 is not supported on Intel?