Oryx Pro Fan Curve Improvement

[jackpot51]

I bought an Oryx Pro last fall and I've written System76 support about this exact issue because the fan curve on my machine is infuriating. Sometimes my fans sound like a rocket ship when I just refresh a web page or open up a random app like Git Kraken. I'll be in random zoom meetings and the fans will spike for absolutely no reason - its literally become a meme with my coworkers now.

System76 Support wrote: "After updating your firmware, you should have manual control over the fans' turbo speed with the Fn + 1 shortcut, however at this time we still do not have fan curves exposed from within the firmware."

Is this custom firmware something I could use on my Oryx Pro? lm-sensors and fan-control are not going to work here right?

These are the two links the System76 support person told me to look at:
https://support.system76.com/articles/system-firmware/
https://github.com/system76/firmware-open

I don't want to hear any fans when my CPU is <15%.

Originally posted by @barthvader13 in #139 (comment)

[jackpot51]

@barthvader13 please provide the output of sensors when the fans are running but the CPU is <15%.

[jackpot51]

Also, which graphics mode do you typically use? Do you use external displays?

[bartlebee13]

First of all, thank you for your help.

I use NVIDIA Graphics mode with the High Performance profile.
I use two external displays: HDMI @1920x1080 and USB-C @3440x1440

The reason I use the NVIDIA Graphics mode is because I get screen tearing on the external monitors while using any of the other modes. I fix screen tearing with a startup script that runs
nvidia-settings --assign CurrentMetaMode="nvidia-auto-select +0+0 { ForceFullCompositionPipeline = On }"
which only works in NVIDIA Graphics mode.

The biggest culprit for fan noise is obviously Zoom and any video conferencing software like Slack Video or Skype. I keep my mic muted as much as possible in almost every meeting I'm in even 1on1s because of the fan noise. I will get a sensors output for you next time I'm in a zoom meeting.

My main concern is I wish I had more control over the fan curve because 30% CPU should not cause the fans to be loud enough that my coworkers can hear it. Though I think this issue may be amplified due to the design and placement of the mic on the Oryx Pro.

The other issue I noticed in general is that the fans turn on randomly for about 20-30 seconds at a time with seemingly no correlation to CPU usage/temps - this is the <15% CPU I initially posted about.

I wanted to note that I did try following this post when I initially "discovered" this problem a couple months ago but the fancontrol package did not seem to work correctly for me so I removed both fancontrol and lm-sensors.

On a normal day I usually have these applications open:
VSCode, Postman, DBeaver, Konsole, Slack, Discord, and Chrome where I'm very diligent of keeping my # of tabs open to <10.

FInally here's sensors output at ~10% CPU - basically idle not opening any new programs or tabs or anything:

[jackpot51]

Thanks for the information. If you can get the sensors output when the fans spin up for no apparent reason it will help a lot. Using the information under the system76_acpi sensors, I will be able to figure out why the fans are running at whatever speed they are running at.

[bartlebee13]

Sorry for the confusion - as far as I could tell - the above sensors output was the fan spin up "for no apparent reason".

IDK anything about the System76 implementation but just from my experience, the fans seems to only be correlated with CPU Percentage instead of both CPU percentage as well as temps?

I keep system monitor and htop open often so I can peep at what's going on and in the process of writing this post I was able to capture two more "random" fan spikes:

random fan spike:

random fan spike:

It seems like the culprit may be the GPU fan? 🤔
Could using a different graphics mode fix this?

Again, thank you for your swift responses!

[bartlebee13]

Another one for good measure hehe

[jackpot51]

Thanks for these. The fan speeds are determined based on the CPU temp and GPU temp as listed in sensors. There is an additional algorithm on top, but for the purposes of this issue, the temperatures are all that is really important. Having the NVIDIA GPU on definitely causes the fans to spin up more often, it produces a lot of heat. But from what I am hearing it seems the NVIDIA graphics mode is louder than it needs to be, spinning the fans higher than they need to.

[curiousercreative]

@jackpot51 maybe e should standardize the default fan curve? galp5 doesn't spin up until 70C for example. Combined with the fan smoothing, I bet this use case would improve.

[jackpot51]

@curiousercreative the chassis are very different. Plastic bottom on the galp5 doesn't get hot easily. Skin temperature is the concern on the oryp6

[curiousercreative]

@jackpot51 yeah, in that case, a higher resolution fan curve at low temperatures would likely help with these constant but mild workloads. Here is the fan curve I run with something like that: https://github.com/curiousercreative/ec/blob/galp5/src/board/system76/galp5/board.mk#L53

[jackpot51]

Yeah, I was thinking very low speed at around 50C to 60C would help. Going straight to 40% at 60C is probably the issue

[jackpot51]

@barthvader13 firmware update 2021-03-08_a065fdb will update the fan curve to handle this use case of low to medium utilization with NVIDIA GPU on a little better.

[jackpot51]

I closed this since under my own testing it sounded much better, running in NVIDIA mode with an external monitor and doing a video call. Feel free to re-open if it is not adequate.

[bartlebee13]

@jackpot51 thank you! I installed the firmware update and its definitely working

[fulara]

apologies if this is not a good place to post this.

Hmm, after installing latest firmware update, my fan is making those buzzy sounds, it didnt happen before.. could that be related to this change?

the buzzy sounds happen when I am not in fully idle-mode but doing a tiny little bit of something.
When its in idle, there is no buzzy sound.
When its fully spinning there is no buzzy sound.

Can that be related to the firmware upgrade? Or is that a hardware malfunction?
I am using Oryx pro.

[leviport]

@fulara Can you post a recording of the sound? Both the oryp6 and oryp7 I tested the update on quieted down quite a bit as a result of the optimized fan curves, so my first thought is that one of your fans is getting long in the tooth. Since the fans can now run at slower speeds than they did before, it might just be more obvious at these new slower speeds.

[fulara]

hello @leviport here we go
https://user-images.githubusercontent.com/5104907/110733636-a23cc280-8269-11eb-83cd-9cf5eabf18b7.mp4

At the end buzz is gone ( its back to idle ).

[jackpot51]

@fulara 15 inch or 17 inch variant?

[leviport]

Thanks for sending that recording. That sounds almost exactly like the noise my darp5 fan started making about 4 months ago. I was pretty sure it's due to wear, so I preemptively ordered a replacement fan to install in case the fan kicked the bucket for good. So far it still seems to be running strong despite the noise so I've just left it alone (I usually have headphones on, so the noise doesn't bother me). I can make the swap today and see if it makes the noise go away. If it does, I'd bet that one of your fans is in the same situation, and either swapping it or cleaning it will make the noise go away.

[curiousercreative]

@leviport you already gave it a good shake and a shot of compressed air?

[leviport]

Not yet, but the fan is easy enough to get at on the darp5. I'll certainly inspect the old fan closely when I have it out.

[jacobgkau]

While previously testing on the 17" model, I noticed a buzzing similar to @fulara's when I picked the machine up and turned it upside down while the fan was spinning at the lowest speed. (I wouldn't normally recommend doing that, but I was specifically looking through the vents to check if the fans were spinning.) The buzzing was worst when I had the machine partially flipped over, at an angle the fans wouldn't normally be sitting at. I'd also be curious if this noise is happening on a flat surface (desk), or if it's while the machine's being used on a lap or on any kind of angled dock/stand.

[fulara]

Hmm, the noise is present on flat surface and is no longer there when i angle laptop at around 90 degree angle.
this is a 17 inch one.

[leviport]

Ah, I believe the 17" has different fans, so the situation might be different there.

Since I said I would: I got the fan changed in my old Darter and it cleared up the buzzing noise completely. There was no significant dust or debris buildup in the old fan or heatsink. This laptop is over 2 years old with a pretty heavy pattern of usage though, so we could be hearing similar noises for different reasons.

[fulara]

I find it puzzlying that this stsrted occuring right after i installed firmware, although i did wake up and installed the firmware oretty much soon after booting up, so in theory something dust up in there since my day usage the day before.
looking at the change it seems pretty harmless, maybe it uncovered some hardware issue not being balanced at certain speeds.as this is only heard when i fans are at low speeds,

i assume there is no way to rollback firmware to verify whether without it there is no sound?
i can try to clean this up but that will not happen for next few days.
working now in home with laptop often leada to not ideal env for laptops tidiness wise i guess. ( often on the knees, couch rather than sitting at the desk )

[fulara]

Just small update:
the sound is present when the fans are like this:
EDIT: had wrong snippet in first example

system76_acpi-acpi-0
Adapter: ACPI interface
CPU fan:      965 RPM
GPU fan:        0 RPM
CPU temp:     +50.0°C  
GPU temp:      +0.0°C   

There is no sound if fans are spinning at:
max:

system76_acpi-acpi-0
Adapter: ACPI interface
CPU fan:     5310 RPM
GPU fan:        0 RPM
CPU temp:     +90.0°C  
GPU temp:      +0.0°C  

or silent:

system76_acpi-acpi-0
Adapter: ACPI interface
CPU fan:        0 RPM
GPU fan:        0 RPM
CPU temp:     +49.0°C
GPU temp:      +0.0°C

So yea Looking at the change: 66a970f
I think previously fans were never spinning at such low RPM - therefore this firmware update has possibly uncovered an underlying hardware problem.

There doesnt seem to be a way to rollback the changes, so I'll have to probably contact with warranty people. Will see.

[jacobgkau]

@fulara I'm not sure how old your machine is, but if it's still in-warranty, it should be very easy to get a replacement if you open a support ticket describing the issue you're experiencing.

It would be possible to run older firmware on your machine by checking out an older EC version and flashing manually (see https://github.com/system76/firmware-open), but that wouldn't be a good solution long-term. And while it might make sense to roll back these fan curve improvements if this was a widespread issue, you're the only one so far who's reported buzzing at the lower speed, which makes this sound like an issue with the fan in your unit and not necessarily with the fan curve or design of the fan.

The tech docs website covers fan replacement for oryp7 and oryp6, whichever you happen to have. (It would probably be a replacement fan/heatsink assembly if you get one through our support team.)

[sd650]

My Oryx Pro 17" (model oryp6) also exhibits buzzing in the CPU fan at low speed (around 1000 RPM), and it started immediately after applying the firmware update. Mine is between 3 and 4 months old.

[Zacxen]

My less than a month old oryx pro 17” (oryp6) exhibits the same buzzing when the CPU fan kicks in at 983 rpm. This happened right after the latest firmware update.

[jackpot51]

We will take a look into raising the lowest fan point. 20% may be too slow for some fans.

[jackpot51]

It would be helpful if one of the affected users would take a look at this PR: #167

[jackpot51]

@fulara @sd650 @Zacxen I would really appreciate one of you taking a look at #167 and seeing if it fixes the buzzing. It increases the lowest fan point from 20% to 25%.

[Zacxen]

@jackpot51 I am planning on re flashing my firmware to see if #167 fixes it however I have never flashed the firmware from source before and wanted to make sure that I did things correctly and not mess something up.

I cloned the tree oryp6-buzz and ran ./scripts/deps.sh.

Do I just need to run make BOARD=system76/oryp6 flash_internal or would I need to re flash the entire firmware not just the embedded controller firmware?

[jackpot51]

@Zacxen that is correct, it will shut down when you run that command so please close everything but the terminal.

[Zacxen]

@jackpot51 This seems to have fixed the buzzing. With the old firmware I could hear it as long as the fan was below 1010 RPM and now the fan never runs that slowly and so I cannot hear it.

[jackpot51]

@Zacxen thank you for testing!

[fulara]

I have also flashed my firmware, i will report if I hear anything, just installed it 30 minutes ago.

[jackpot51]

Great. We will release the firmware tomorrow if testing goes well.

[sd650]

I tested it, too. It appears to work for me as well; no more buzzing. For me, the buzzing zone was below around 1015 RPM, similar to what Zacxen reported. Just a peculiarity of this particular fan model, I suppose.

[jackpot51]

Good to know. I think 20% was on the edge of acceptable and hardware variability made it not fully drive some fans. The new point is 25% which is not noticeably louder and should be more reliable

[fulara]

Yep, the fans now go from 0 RPM to ~1425 - and there is no buzzying at 1425.

thanks.

[ggobbe]

Just a big thanks to the team for this firmware update.

I was also feeling the pain because of the fan noise on my oryp6; since the a065fdb update my days have become much quieter and less stressful 🙇

[bartlebee13]

Just a big thanks to the team for this firmware update.

I was also feeling the pain because of the fan noise on my oryp6; since the a065fdb update my days have become much quieter and less stressful

I'd like to echo this sentiment. Seriously, huge thank you! The customer support that I've received from the team at System76 has been exceptional. I tell all my friends about Pop!_OS and your products. Keep up the great work!!!

[MilesBHuff]

The fix for this issue (#163) seems to have directly caused system76/firmware-open#215.