Linux kernel 5.8 cannot control fan speed

[zegranix]

Hello,

Hardware: MacBookPro 6,1 mid-2010. Using latest version of mbpfan. Have used mbpfan for long time on Arch Linux across many upgrades with no problems.

After recent upgrade to 5.8.1 kernal, from 5.7.12, mdpfan does not work,
with error

ehost mbpfan[374]: Could not set fan speed: Input/output error
Aug 18 21:01:21 ehost kernel: applesmc: send_byte(0x40, 0x0300) fail: 0x00
Aug 18 21:01:21 ehost kernel: applesmc: F1Tg: write data fail

--This seems similar to issue #155...

data cannot be written.

From looking in /sys/devices/platform/applesmc.768

one can see that even root can only write to fan1_manual, fan1_min, and
fan1_output, for example.

-r--r--r-- 1 root root 4096 Aug 18 21:00 fan1_input
-r--r--r-- 1 root root 4096 Aug 18 21:01 fan1_label
-rw-r--r-- 1 root root 4096 Aug 18 21:00 fan1_manual
-r--r--r-- 1 root root 4096 Aug 18 21:00 fan1_max
-rw-r--r-- 1 root root 4096 Aug 18 21:00 fan1_min
-rw-r--r-- 1 root root 4096 Aug 18 21:00 fan1_output

The above permissions have not changed from earlier kernels when mbpfan worked. They are exactly the same on the systems where mbpfan works.

As a test, I upgraded another system with a working mbpfan (kernel 5.7.9) to 5.8.1, and the same no write problem appears.

I can write to fan1_min using something like

sudo sh -c "echo 2500 > /sys/devices/platform/applesmc.768/fan1_min"

Of course, it has rw.

--It seems that something in the kernel that affects mbpfan has changed between
5.7.12 and 5.8.1.

--I'd appreciate any suggestions, and I'll try to help troubleshoot this.

[Crawdadius]

Hi there.

Not sure whether there is much that I can contribute to help troubleshoot, but I'd just like to share that I experienced this same issue on my MacBookPro12,1 (13-inch, Early 2015). After upgrading Linux 5.7.12.arch1-1 => 5.8.1.arch1-1 my fans no longer spin and I get the following output in dmesg.

applesmc: send_byte(0xa8, 0x0300) fail: 0x40
applesmc: F0Tg: write data fail

I had actually previously been using macfanctld (which seemed to perform well enough to me) from the AUR when the issue first arose, but after troubleshooting I realized that it hasn't been updated in ages. Unfortunately switching to mbpfan gave me the same result; no fans and the same dmesg output.

After downgrading to 5.7.12.arch1-1 my fans are working properly.

Let me know if there is any other info I can provide or testing I can do. Thanks!

[zegranix]

Hi,

Thanks for your reply. I appreciate it. --It's useful information to know that I'm not the only one with this issue, which would suggest that it was something in my configuration.

I continue to think and experiment.

One further thing I experienced, oddly. In my testing, I just happened to try

sudo sh -c "echo 2500 > /sys/devices/platform/applesmc.768/fan1_output"

not expecting it to work at all.

Well, I got the usual i/o error (Could not set fan speed... and the rest) but the change worked! Conky showed me that the speed of fan 1 changed to 2500, seconds after the command. I waited a minute or so then tried it again with 2000. Yep, it went back to 2000.

So, I get the error message, but the command works. --Further observation suggests that mbpfan may actually be functioning as expected --in spite of the error messages. (I'm skipping the details.) Hard to believe. This is an interesting
puzzle. I'm going to stick with 5.8.1 for a while longer and see what develops.
--If fan speed is going to be an issue for my model in future releases, I'd kind of
like to get a sense of that now.

[zegranix]

--Sorry, clicked the wrong button.

[pobetiger]

Hi, just a passerby, ran into the same issue. From the error message

F%dTg is target speed (manual rw), according to: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/hwmon/applesmc.c?h=v5.8.5#n80

So it looks like both of the fan control services return this error that claims it fails to set the target speed.
Recent changes according to the git log:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/hwmon/applesmc.c?h=v5.8.5

Age | Commit message (Expand) | Author | Files | Lines
-- | -- | -- | -- | --
2020-05-27 | hwmon: (applesmc) avoid overlong udelay() | Arnd Bergmann | 1 | -3/+9
2019-11-06 | hwmon: (applesmc) switch to using input device polling mode | Dmitry Torokhov | 1 | -20/+18

Suspected kernel patch that introduces the bug: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/hwmon/applesmc.c?h=v5.8.5&id=fff2d0f701e6753591609739f8ab9be1c8e80ebb

call chain:

• fan daemon -> sysfs -> applesmc_store_fan_speed() -> applesmc_write_key() -> applesmc_write_entry() -> write_smc() -> send_bytes().

And the fan%d_manual is the last thing on the list for the fan group (probably explains why it's always F0Tg or F1Tg that's failing in most the logs).

tl;dr: udelay() were switched to usleep_range() due to clang -O3 doing some funny business (from commit message above). My guess is that usleep_range() is probably too lenient and results in enough jitter that some of the commands are failing/timing out on the spi bus...

This will explain why it sometimes works but the error is spamming our syslogs.

I currently do not see a patch for this fix on patchwork: https://patchwork.kernel.org/project/linux-hwmon/list/?series=&submitter=&state=*&q=applesmc&archive=&delegate=

We will likely have to submit a patch or file a bug on the LKML.

EDIT: upon further inspection, it looks like the code in question also reduces the numbers of times retries is allowed since now it's doing usleep_range(us, us*16), and in each loop, it's doing us<<1 (doubling). The old code used to just wait us and double. So the new code also tries less times and shows the error message more times.

My prognosis is that this patch went in and nobody thought about the fact that the underlying code is going to spi bus... :( or that they've reduced the number of times to retry on that access.

EDIT 2: opened bug: https://bugzilla.kernel.org/show_bug.cgi?id=209063

[SudosFTW]

Adding in my 2 cents, Debian 10 with backports kernel 5.8.0-0.bpo.2-amd64 (5.8.10-1~bpo10+1) from 2020-09-26, on a MacMini3,1 (Early 2009 C2D 2GHz/GF9400M) and I'm getting these same errors, yet fan control is working just fine. Since this is a headless box I'm not too worried about it but I thought I'd drop this here to let you know it's not a distro-specific issue and getting updated code out to fix this, preferably to get into buster-backports/testing/sid, will save a lot of users a lot of headaches.

This is especially true since a number of macs are about to hit the used market when the ARM macs hit retail, and honestly, this is the only daemon that reliably, every time, keeps the fan running at a reasonable speed to keep temps down, and I thank you greatly for it.

[fnarfbargle]

This should fix it :
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7e908b7461ec395293335852485a183c16765303

[gaul]

Confirmed fixed in 5.10.

[hamishmb]

Presumably doesn't need to be pinned any more, now its fixed.

[gaul]

Removed the pin; thanks for the reminder!