Crust Wakeup issue

[kmsgli]

Type of issue

Question:

I am not entirely sure if this is a crust issue but I can not seem to trouble shoot it so I figured asking here may point me in the correct direction;
I am using an original Pinephone and generally crust works fine to wake my phone for calls and text messages. I use my Pinephone as a daily driver so it suspends and wakes a lot throughout the day as any phone does. At least once but sometimes upwards of four or five times a day when I hit the power button while the phone is in suspend my phone will not wake. I know its still on because holding the power button does not turn it on. I have to hold the power button to turn it off then hold the power button again to turn it on again.

Ssh does not allow me in obviously because it is in the deep sleep state so I can not get into the OS to trouble shoot what the issue is. This has never happened on the charger as my phone is set to never sleep on the charger.

I posted in the Pine64 forums and found another person having the same issue with a PInePhone Pro which I assume has the same crust implementation, he has tested on multiple OS's as well so its not specific to a distribution.

Any help trouble shooting this would be greatly appreciated as this is the only thing that really makes daily driving my phone a pain, otherwise it works 100% as a regular phone.

[gamelaster]

PinePhone Pro have different SoC which is not supported by Crust, so the issue on forums is not related to your problem.

[kmsgli]

Well I will report that back to the guy who posted on the forums. Am I correct to assume my issue is related to crust as far as the standard pinephone?

[smaeul]

Yes, it is possible that your issue is related to Crust.

What distro are you using? Some distros build Crust with a few debugging options already enabled. Otherwise, you will need to rebuild Crust with those options enabled. Then, after a failure to wake you will be able see what Crust was last doing and if it finished its part of the resume process with hexdump /sys/devices/platform/soc/1f00000.rtc/nvmem0/nvmem. You will also need to temporarily disable cpuidle (if your distro has it enabled) with cpuidle.off=1 on your kernel command line, or else that will overwrite the data stored in the RTC.

Please also provide a full dmesg taken after a successful resume.

With that information, I can suggest further troubleshooting steps.

[kmsgli]

I am using manjaro phosh is that built in or do i need to rebuild crust?

[smaeul]

It does not have those options enabled. You can enable them by changing that block of lines and rebuilding the package:

  echo -e "\nBuilding CRUST for Pine64 PinePhone...\n"
  echo CONFIG_DEBUG_RECORD_STEPS=y >> configs/pinephone_defconfig
  echo CONFIG_DEBUG_VERIFY_DRAM=y >> configs/pinephone_defconfig
  make CROSS_COMPILE=or1k-elf- pinephone_defconfig
  make CROSS_COMPILE=or1k-elf- build/scp/scp.bin
  cp build/scp/scp.bin ../u-boot-${pkgver/rc/-rc}

You can verify that it's working if you see nonzero values in the nvmem file after a suspend/resume cycle.

[kmsgli]

Ok just so I am clear because I have only built AUR package I would do this;
git clone https://gitlab.manjaro.org/manjaro-arm/packages/core/uboot-pinephone.git
change the bit of code you listed above in the PKGBUILD file then;
makepkg -si

test with
hexdump /sys/devices/platform/soc/1f00000.rtc/nvmem0/nvmem

then report back?

[smaeul]

Yes, that should work. It looks like installing the updated package will try to flash the new U-Boot binary.

[kmsgli]

Yes, that should work. It looks like installing the updated package will try to flash the new U-Boot binary.

Is that an issue? or should it be fine to flash the new U-Boot binary?

Also I am reluctant to do this now as I am out of town and if something gets trashed I wont have a phone so I will wait until monday to do this and report back.

[smaeul]

Crust is inside the U-Boot binary, so you have to flash U-Boot to get the updated Crust.

There's no rush; install the debug firmware and try to collect the logs whenever is convenient for you.

[kmsgli]

I believe I have successfully installed U-Boot with the debug options.

Running hexdump /sys/devices/platform/soc/1f00000.rtc/nvmem0/nvmem after a wake from suspend I get:

0000000 0000 0000 0000 0000 0000 0000 0503 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
*
0000040

At this point I should wait for a freeze then reboot the phone and runhexdump /sys/devices/platform/soc/1f00000.rtc/nvmem0/nvmemagain and report back?

[smaeul]

Yes, that's the plan.

The NVMEM output you provided looks good (but see below). Examining the nonzero word, the first two digits confirm this is output from Crust v0.5. The last two digits come from this list of steps. 0x0503 means STEP_SUSPEND_CSS, so you caught Crust in the middle of turning off a CPU core. Because your kernel has cpuidle enabled, Linux tells Crust to power off the CPU cores 100+ times per second. This is normally good because it saves a bit of power, but it will unfortunately also overwrite the crash data before you have a chance to read it.

So you have two options:

1. If you have a UART adapter available, you can grab the data from the U-Boot prompt with md.l 1f0010c 1. Since you would be running this command before Linux has a chance to overwrite the data, you would get good data from the crash.
2. Otherwise, you will need to temporarily disable cpuidle by adding this kernel command line argument: cpuidle.off=1. You will also need to build a new version of Crust (sorry!) that contains commit e624ba0. (This commit keeps you from needing to disable other things to get good data.) You should be able to do this by setting _scpver=master in the PKGBUILD, and then fixing the URL to accept a branch instead of a tag:

"crust-${_scpver}.tar.gz::https://github.com/crust-firmware/crust/archive/refs/heads/${_scpver}.tar.gz"

[kmsgli]

I don't have UART adapter as an option so number 2 is it.
I am with you on rebuilding the software as its exactly what I did before by changing the initial echo lines I have already done and changing the two parameters at the very beginning of the PKGBUILD (that should be no problem).

I am a but fuzzy on turning cpuilde.off=1 how to do I make that happen?

Interesting piece of information. Since rebuilding crust (yesterday at midnight) I have yet to get hung up in suspend. This could very well be a coincidence however its very rare that I have gone a full day without a hang up in suspend. I am going to try to stay off the charger as much as I can today to see if this is maybe more then coincidence and there is simply something wrong with the crust build that comes on the latest Manjaro betas ( seems unlikely as arch did the same thing and I have tried flashing OS fresh a few times).

[kmsgli]

As of today I have not had a single hang up in suspend, could this problem have resolved itself with the recompile of crust? I tried fresh OS installs in the passed with no luck?

Does debug need to be on or was it simply manually rebuilding the package that fixed my issue?

[smaeul]

I am a but fuzzy on turning cpuilde.off=1 how to do I make that happen?

From the Manjaro packaging, it looks like you would need to add the option to the setenv bootargs line in /boot/boot.txt and then run pp-uboot-mkscr as described in that file.

Does debug need to be on or was it simply manually rebuilding the package that fixed my issue?

That's very interesting. Either one is possible. It's also possible (though unlikely at this point) that the issue still exists, but it has happened not to be triggered. You could try reinstalling the official package, or building a new package without the debug options, to see if the crashes come back. If they do, then likely the tiny additional delay introduced by the debugging code is allowing suspend/resume to complete.

[kmsgli]

I just reinstalled using the git clone method without changing the debug options which in theory should be the same as the stock uboot setup when the OS is installed. I will report back in a day or two with if I have any suspend resume problems.