4.12.0-rc4 Kernel: Reboot does not work

[sghazagh]

Kernel 4.12.0-RC4 is compiled successfully and boots the Tinker Board just fine.
Thank you for adding WIFI to, so WIFI is working just fine as well.

However, REBOOT does not restart the board and after turning off the device, cannot get it up again!
I have a same issue if I use "echo b > /proc/sysrq-trigger" as well.

[Miouyouyou]

Alright, I'll see if I can find the main reason of this issue and resolve it.

[Miouyouyou]

I catched that in rockchip's kernel miniarm DTS file :

        /*
         * NOTE: vcc_sd isn't hooked up on v1.0 boards where power comes from
         * vcc_io directly.  Those boards won't be able to power cycle SD cards
         * but it shouldn't hurt to toggle this pin there anyway.
         */
        vcc_sd: sdmmc-regulator {
                compatible = "regulator-fixed";
                gpio = <&gpio7 11 GPIO_ACTIVE_LOW>;
                pinctrl-names = "default";
                pinctrl-0 = <&sdmmc_pwr>;
                regulator-name = "vcc_sd";
                regulator-min-microvolt = <3300000>;
                regulator-max-microvolt = <3300000>;
                startup-delay-us = <100000>;
                vin-supply = <&vcc_io>;
        };

This might be related to the reboot bug.

[sghazagh]

I didn't get your point? can you explain please?

At the moment yous rk3288-miniarm.dtsi file has a same code.
and, in official image, he reboot works fine and only sysrq does not work!

[Tonymac32]

The supplies are separated on later boards, although you may be correct in a way, power cycling the SD card on shutdown kills it, something that wasn't an issue on 1.0 boards. Also, the system does not bring the SD card back to 3.3V despite returning to the lower speed operating point.
TinkerBoard/debian_kernel@c6ad239
TinkerBoard/debian_kernel@5883709

[Tonymac32]

So, @sghazagh , the reason it works on the official image and the Armbian 4.4 is due to a straight up hack of the driver code. I attempted to do the same thing on 4.11 and it did nothing.

[sghazagh]

Why this Tinker Board has this much issue then???!
If pros like you guys and community have found hard to resolve the issues, AUSU team definitely can't do much to improve it too for sure!

ASUS had to provide a better alternative to RPi3 if they really were looking to hit the market!!! Disappointing!

[Miouyouyou]

Hmm, @wzyy2 and @hizukiayaka might have a clue about how to handle power cycle issues. I'll try to ping them in IRC.

[sghazagh]

That would be good if that can be fixed first as it's very annoying when you're working remotely and you cannot reboot due to this issue!

[wzyy2]

00:22 That issue in tinker board will happend in warm rest.
00:22 and WIFI module are not reset.
00:22 ASUS have fix it in their kernel.
00:23 let me search which commit are used to fix it......
00:29 TinkerBoard/debian_kernel@3308568
00:30 = =I think it will be this commit.
00:31 BTW, TinkerBoard/debian_kernel@5883709. You might also need this commit when run mainline or rockchip 4.4 kernel in tinker board.

[Miouyouyou]

Alright, most of these patches look small enough to give them a try.

I'll try to add all these patches this evening, see if that resolve the reboot issue on 4.12 kernels.

[Miouyouyou]

New release integrates TinkerBoard/debian_kernel@5883709 patch. The DTS patches were already added previously.

See if that resolves the issue. I don't think this will be enough, as @Tonymac32 already tried that.

[sghazagh]

No, the reboot issue hasn't resolved!!!

[Tonymac32]

I'm somewhat at a loss on this one, is this due to the tinker board hardware, or is it something driver related? I would have assumed the option was there to keep the vmmc active (device tree or otherwise), also shouldn't switching vqmmc back to 3.3 V from 1.8 be standard? I just want to make sure some funny joke isn't being played on everyone involved... ;-)

[wzyy2]

"I'm somewhat at a loss on this one, is this due to the tinker board hardware, or is it something driver related? "

Rockchip bootrom code only allow normal sd mode, so if the vqmmc is 1.8v, it will failed to load firmware from sd-card.
It's necessary to set it back to 3.3v during warm reset.

[Tonymac32]

Thanks for the reply, and that I completely understand the mechanism of failure. I'm just surprised to see it show up as a bug that is only addressed via a "hack" or two. The power supply is also apparently switched off before the warm reset. Is there a bootrom fix, or an expected bugfix to the driver?

As for the patch, I think the patch to restart the vmmc is also very important, however it has a delay built into it which appear to break compilation on 4.12. I don't have time tonight to dig into that one.

[Tonymac32]

OK, put the hack inplace, can verify it is being executed, but it is not having the desired effect. Not having appropriate schematics or documentation for this board I'm having a near-impossible time checking physical voltages.

[wzyy2]

Did it still fail to boot?
Could you try to disable uhs mode by remove "uhs" prop in sdmmc dts node?

[Miouyouyou]

I'll try to remove it today for the next release, see if that changes anything.

[Tonymac32]

I was hoping to not have to disable one of the more attractive features of the board, Obviously that's the trigger to drop to 1.8V. Perhaps now it needs a delay after telling it to set, the delay position in the code was somewhat of a mystery to me from the beginning, it's the first thing done in the subroutine rather than the last. I would have thought you'd need to restart vmmc, set vqmmc back to 3.3V, then wait for it to stabilize before triggering the SoC to reset. I compiled a test of that last night, but had to get to sleep before it was finished compiling for test. I'll check it later today.

[Tonymac32]

No Dice, removed the uhs props from it to no effect. Also added delay after setting supplies, also no effect.

[wzyy2]

So it might be another reason make it failed to boot.
Could you connect otg port with host PC and try reboot?

If "lsusb" in host PC show maskrom device, then it means bootrom failed to load firmware from SD-Card.
If there is no maskrom device, then it means kernel have hang in reboot.

[Miouyouyou]

Alright, so, the MMC hack introduced reboot issues with the MiQi so I removed these hacks for now. I've commented 'uhs' references in the sdmmc driver.

Meanwhile, another way to test if the machine reboot or not is to ping the device. If the device still answers a few seconds after closing the SSH session, it means that it could not reboot and is still running...

[Miouyouyou]

Meanwhile if you can:

• Get a serial console output using the USB<->Serial communication system
• Reboot the machine and look at the logs and see if there's anything useful, like crashes, OOPS messages or things like that

If that's the case, paste these messages there.

[Tonymac32]

I'm afraid I almost work exclusively though the serial output on the board, there are no messages, it successfully shuts down, and simply does not restart. This is definitely related to the SoC bootloader (not U-boot). I will plug in a USB cable to see if I can see the maskrom device or not. I will have to check with the team to make sure the Armbian MiQi build is still rebooting properly.

Is it possible supply assignments in the kernel changes between 4.4 and 4.12?

[Miouyouyou]

It's entirely possible. Meanwhile, it might be interesting to open a thread in mainling list consulted by Rockchip enthusiasts. I'll try find the URL on this mailing list again.

[Miouyouyou]

I'm still surprised that regulator_get_voltage would generate a NULL dereference kernel panic. But whatever.

Removed the logs in the patch, just in case sending printk with an error level in the logs fucked up the reboot on the ASUS SugarBoard. Redefined the variables outside the if branches when possible and added a printk during the emergency reboot, just in case the Out-Of-Order execution model of the ARM processors might have generated issues. (And also to see if the issue comes from the printk call).

Still, just to give you an idea, this is the Armbian patch, this is my last rework of the reboot patch. You can compare them with meld if you want.

Here's a comparison anyway. This patch is known to work.

I compared the DTS too, there's no differences beside indentation and nodes relative position.

So, yeah, if that does not work, just create a service that does echo b > /proc/sysrq-trigger after the filesystems get unmounted and call it ASUS-Expertise.service.

[elar-systems]

I just see your difference and I see that there are couple of things are not same.
For instance the struct *mmc has not default value in your verision (you set it later inside 'if' !!!) but Armbian one has (line 77 red: *mmc = mSdhost->slot->mmc; and line 98 red: static int dw_mci_init_slot(struct dw_mci *host, unsigned int id) has 3rd parameter green one doesn't)

Does this something you should know about?
also see some other difference between you and Armbian.
Those might be the case! Would you please check again and see if the patches are identical between your version and Armbian one?

[Tonymac32]

"ASUS-Expertise.service" 🤣

Honestly this and the Bluetooth shenanigans nearly had me considering pulling the plug on my support for this board, and since I'm the only contributor to it on Armbian... Add the completely unnecessary Audio Codec and the fact you cannot power it over microUSB without a brownout if anything is plugged into the USB ports...

Beautiful board with exceptional build quality. But it ends there.

Now off to look at my new shiny RK3328 and A905X boards for a while and cool down on the Tinker Toy before I do something rash. I bought it to be a media center/gaming box, wound up buying a Ugoos UT3S instead...

[Miouyouyou]

the line 77 will crash any board that could not set the mSdhost value, with a NULL dereference.

The unsigned int id is actually removed in 4.13 kernel
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/mmc/host/dw_mmc.c?id=e4a65ef7687b6aaf36bedb497d3fd1480163d2d5

[Tonymac32]

Oops, I should probably switch that then, although the flood of RK3288 boards hasn't happened, has it? Probably the 32-bitness, even if it is very powerful.

[elar-systems]

@Tonymac32 , I understand your frustration as there is no documentation for this board after all this times!
But regarding USBMicro power, honestly I have not any issue as I am using a 5V, 3A adapter and it never complains for power!
I always connect a USB dongle for Bluetooth mouse and keyboard and it also can handle that as well!
You might need to check your power source.

[Miouyouyou]

Well, if you can try to set the value inside the if branch and see if it changes anything when you do an emergency reboot, I'd appreciate that.

That said, it only concerns the emergency reboot system, which @sghazagh said was working so...

[elar-systems]

@Miouyouyou ,
The interesting thing is that, the reboot shuts-down the power completely as all LEDs goes off, however, the echo b> ... one keeps the LED lights open and can restart!
Is there anything forcing that to power off the board? Do you wanna check that one?

[Miouyouyou]

Well, first give a try to the latest zImage here https://github.com/Miouyouyou/RockMyy-Build . Just grab the zImage in /boot and the rk3288-tinker.dtb too. Boot them, try a reboot and if that does not work, try an emergency one.

Meanwhile, I'll try to see what are the major differences between the emergency reboot and the standard one.

[Tonymac32]

@elar-systems I'm an electrical engineer, I actually did a write-up specifically concerning powering this board, measuring the voltage drops/etc. If your supply is a 5.25 Volt unit and your usb peripherals are comfortable at 4.5 V you can get away with it, assuming 18 AWG and a shorter cable. The amp rating is irrelevant if the series resistance causes too large a voltage drop.

The "factory" failed reboot causes the light to stay on, so the mmc supply is being disabled, but not re-enabled. Why the light is affected I actually don't know...

[elar-systems]

Sure you know better mate, just was saying... ;)
I am not an Electronic guy as I am a Mechanical Engineer.
The Single Board computers and ARM world is what we offer at industrial level for Automation. I design the Mechanical part of that based on the client requests.
Now that the market is so quiet, I am learning the Linux more and like to understand the ARM based system more and more.
I wish I could understand the electronic part of the system but I think it's a little bit late for me (45 now ;))
anyway...

[Miouyouyou]

Well, the difference between simple restart and SysRQ restart seems blurry. That's the kind of code where I'll need some ways to get call graphs of any code that can call the machine_restart function.

Now, for the record, here's how emergency_restart is implemented

#ifndef _ASM_GENERIC_EMERGENCY_RESTART_H
#define _ASM_GENERIC_EMERGENCY_RESTART_H

static inline void machine_emergency_restart(void)
{
	machine_restart(NULL);
}

#endif /* _ASM_GENERIC_EMERGENCY_RESTART_H */

There's a special version of emergency_machine_restart, but only on X86 systems it seems.

That said, it seems that any driver can set up special callbacks when SysRQ mechanisms are triggered, so that mean that drivers could behave completely differently when calling the SysRQ reboot mechanism.

The Rockchip systems seem to register at least 2 reboot handlers in :

• syscon-reboot-mode.c
• clk.c

That said, that does not clearly tell me why the LEDS are shut off during simple reboot, and why they're not with the SysRQ one.

Also, machine_restart can be called with a special command. It might be the case here, I don't know.

[Tonymac32]

I tried it, nothing changed... :-/ echo b > /proc/sysrq-trigger went off without a hitch...

[Miouyouyou]

Thought so. Well, at least RK3288 boards that doesn't have a support-sd property on the SDMMC node won't panic on emergency reboot at least.

Anyway, I'll leave this bug opened for a week, just in case someone either :

• got it working with RockMyy-Build zImage and latest DTB. Grab the DTB seriously;
• got a very bright idea.

And then, I'll close this bug if that's not the case.

The patch has been applied. If that doesn't work for you, I don't know what will. I mean, really !

It's not a patch issue.
It's not a DTS/DTB issue.
The ARMbian rockchip-dev also apply roughly the same kernel patches as me.

ARMbian users got it working, so try ARMbian. Maybe they applied additional power supply fixes in their distro that make this patch work fine. They've been fed up of power supply issues on their forum so it's highly possible they did things to lower the probability of such problems occurring. If their kernel work, try this one just in case. If it works too, leave a comment ! Re-open the bug if needed !

You got the compilation script here. You got the patches. The script prefers local patches over net patches anyway, so just clone the repo and get to work !
You can add any patch easily by just putting the patch in the right folder, and adding it to the list in the script. If you find any combination of patches that works for you, meaning that the board reboots fine with your combination, leave a comment too ! Re-open the bug too if needed !

If the emergency reboot works, meaning echo b > /proc/sysrq-trigger works, create a service like specified earlier, that will be executed before shutdown and trigger the emergency reboot. I'm not very familiar with creating services but that should not be very hard on systemd.

You can try various reboot=X arguments, as specified in an earlier post, and see if that works. Maybe Soft Reboot might work, even though they're extremely ill advised on SMP.

You could also try to get kexec working on your board, this will avoid doing a power cycle and just load up a new kernel directly, by-passing the whole boot cycle. Good luck though, as this is the most untested way of loading up kernels.

At last, if you really want to get this bug fixed, and all the alternatives failed, either :

• ask ASUS to send me a sample;
• pop 60$ on my Patreon or my Pledgie and I'll buy one;
• hone your electronic skills and debugging skills and get to work !

[Tonymac32]

As a last thought, I have CONFIG_SYSCON_REBOOT_MODE=m as the system is supposed to support it according to the dtsi, no idea if that would have an impact.

ASUS never got armbian any boards to my knowledge, despite promising them.

[resipat]

@Miouyouyou I think you got it. The reboot works. Attached the logfile.
Tinkering_4.13_reboot.txt

Great work!

[sghazagh]

Congratulation everyone!
the reboot works fine for me too

Well done Miouyouyou.
all are working:

• GUI "reboot"
• terminal "reboot"
• shutdown -r now
• and "echo b > /proc/sysrq-trigger"

We are waiting for merging this to master and adding a new fresh branch.
Thanks for all your efforts...

[Miouyouyou]

Well, I'll port the patch to the 4.12, so that it's also usable on MyyQi. Then I'll close this bug.

Thank you everyone for your efforts.

Glad this bug is solved.

[Miouyouyou]

Patch backported on MyyQi. If you really want to use a 4.12 for some reasons, give it a try.

In any case, I'll close this bug tomorrow if I don't have any more reports.

[elar-systems]

@Miouyouyou ,
The latest stable 4.12 release is 4.12.3.
I do recommend you make a tag for 4.12.3 and apply the patches. Let us check and if all was good make it as release.
This way we have working 4.12.3 and you can have a master as ongoing work for 4.13.x series.

I personally will stick to 4.12.3 as 4.13 is release candidate and is not stable.
Please update us if will you do this...
Thanks

[Miouyouyou]

I'll see if I sync with yet another kernel Git repository. Remember that the 4.12.3 does not exist in Torvalds GIT repository, it's maintained in yet another repo, which I don't the address of currently.

I'll look around.

[elar-systems]

No,
It's in stable kernel release at main kernel.org https://www.kernel.org/

You can also find it here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

[Miouyouyou]

Alright, I'll try with this one, see if the patches apply. But I'm not going to maintain two kernel branches.

The script will modified once to use the stable git repository, then I'll wait until it breaks.

[Miouyouyou]

Here's the https://github.com/Miouyouyou/MyyQi/tree/stable .

I won't maintain this branch though, unless I'm rolling in cash.

So if it breaks, it breaks.

[sghazagh]

@Miouyouyou,
I have tried the kernel and DTB files from your new stable 4.12.3 release.
There are two DTB files relevant to Tinker board.
"rk3288-tinker.dtb" and "rk3288-miniarm.dtb"

the second one did not work, but the first one, "rk3288-tinker.dtb", was ok and "reboot" was working fine.

I think, as the official ASUS DTB file named "rk3288-miniarm.dtb", keep the file name consistence or somewhere in README file, explain which DTB file is for Tinker Board.
I am not sure if any other devices like MiQi is using "rk3288-miniarm.dtb" or not. But if not, then I think at least you can remove it from the list as it is confusing.

Anyway...
Thanks for all your effort and well done.

[Miouyouyou]

Thanks.

Glad to know that the DTB was essential to get this fix working.

Now, again, as stated before, the official Tinkerboard DTS used in the official kernel (the one hosted and maintained by Linus Torvalds) is named rk3288-tinker.dts and will stay like that until the maintainer of that file think it should change.

Seriously, it's there : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/rk3288-tinker.dts

The rk3288-miniarm.dts I provided during the 4.12 version was an improved version of the rk3288-tinker.dts provided by @Tonymac32 . @Tonymac32 called it like this because during the 4.11 there were no Tinkerboard DTS mainlined, so he took the name used in the semi-official Rockchip linux tree on github.com. However, the improvements added to rk3288-miniarm.dts have now been ported to the official rk3288-tinker.dts.

So, yes, it's VERY confusing, and thus the unofficial rk3288-miniarm.dts have been removed in the new kernels scripts.
The rk3288-miniarm.dts is removed from the list starting from the 4.13 version.

So, forget about rk3288-miniarm.dtb. It never was official. It's not available anymore on 4.13.x kernels. And no, I don't consider the semi-official Rockchip kernel tree on GitHub an official source for that matter.

Conclusion : use the rk3288-tinker.dtb

---

Now, the reboot issue is solved on MyyQi and RockMyy.

YAY !

So I'm closing this issue.

If you get this issue again on future versions :

1. Don't reopen this bug if it happened on a stable kernel version like 4.12.92, unless you want to pay me. Even if you pay me, check step 4. . As specified in the stable branch of MyyQi, I don't care about the linux-stable repository. I don't use it and don't have the financial support to start caring about it anyway.
2. If it happens while using a RockMyy-Build, open a new issue on RockMyy.
3. Be sure that you're using the latest rk3288-tinker.dtb provided in the RockMyy-Build repository.
4. Don't bother opening a bug if you can't provide exhaustive kernel logs of the bugged reboot procedure, or if you don't have a clear idea about what's going on and don't know a patch that fixes it on some kernel version, as I won't be able to help you. Seriously, if there's NOTHING in the logs that state the issue, I won't guess it.

---

Alright.

Thank you everyone for your participation and efforts.

This issue is now closed.

You can now reboot mainline kernels on your Tinkerboard.

Congratulations.

[sghazagh]

Ok, that make sense.
By the way, as far as it's not there, des not matter if the name is tinker or something else.
Actually, Tinker is much better.

Meanwhile, that's really good you see the latest kernel is loading and wifi is working as well.
But wish the Bluetooth also was working as it's symbol is showing disconnection and make you feel disappointed.

I am sure you will fix that too.
Thank you