System freeze on MacOS Mojave

[ameyah]

Hello,

Santa (update 0.9.28) causes random occasional system freezes on Mojave beta 10 and 11.

From the end-user perspective, it feels like a kernel level lock up and happens within a half hour or so of boot on beta 10 and 11. Seems to especially be triggered by WindowServer context switches (wake from sleep, switch virtual desktops, etc.).

[tburgin]

Hello, thanks for the report. I have not seen this behavior on B10 or B11. A few clarifying questions.

Are you running in Monitor or Lockdown mode?
Do you experience these lockups with Santa uninstalled?
Do you have any crash reports?
Can you share logs from the time of the lockup?

/usr/bin/log show --info --debug --predicate 'senderImagePath CONTAINS "/Library/Extensions/santa-driver.kext"' --last 5d

[tburgin]

Okay, I have been able to reliably reproduce the problem. The process trustd that runs as root, is used somewhere in the code sign checking chain. For some reason on Mojave this process falls over. If this coincides with Santa purging its in-kernel whitelist cache, a ~deadlock will occur. trustd is trying to exec, Santa is trying to check trustd's code signature, but this requires trustd. Classic.

Anyways, we have a plan to fix this. A fix will be up soon.

[arubdesu]

I was experiencing something like this issue (Mojave GM, in monitor mode, versions .9.26 and .9.28 of santad), is there more of a 'smoking gun' log message to know that .9.30 fixes the issue?

[headmin]

we too experiencing something like this issue from client reports using 10.14(18A391) machines. fully remove santa as short term remedy stopped lockups at affected client machines.
atm we run few in-house tests to trace in more depth.

[tburgin]

@arubdesu Binaries that are in the critical path of code signature verification are pre-evaluated by santad and allowed to execute thereafter. We include the following in the exec log: explain=critical system binary. These binaries are executed a lot during normal usage. There is no differentiator between an exec that would have caused a deadlock and an unrelated exec attempt.

[tburgin]

@headmin Sorry about that. Are you still experiencing lockups with version 0.9.30?

[AndersTao]

I still see clients freez after 0.9.30
MacOS 10.14 and 10.14.1

So far I have only seen the problem on upgrated clients.

[tburgin]

During a lockup can you grab a spindump?
sudo spindump santad
You can also do it from Activity Monitor (click the gear and choose spindump).

[AndersTao]

I will try to catch the next one, but not sore its posible

Computer is all locked up when it happens, but will try with ssh, before the user reboot

Can I get any usable debug data out, after the user has rebooted?

[tburgin]

Logs might help.
/usr/bin/log show --info --debug --predicate 'senderImagePath CONTAINS "/Library/Extensions/santa-driver.kext"' --last 1d

The execution logs are somewhat sensitive. Feel free to send them to me directly bur@google.com.
/var/db/santa/santa.log*

Somewhat related, do you use any FUSE filesystems? They also caused similar lockups for me.

[AndersTao]

Thanks Tom

My own computer just crashed with the spinning wheel, I will colllllect and send them to your mail.

About FUSE, it looks like Transmit use it for local mounts, but I don't use the function

[headmin]

@tburgin on a t2 MacBookPro first time santa intalled running on 10.14.2 later upgraded to 10.14.3 the freezes happend almost once per day. I have log files to send you

[tburgin]

@headmin When do the freezes occur? Can you describe what you experience?
Most important, do you have a spindump captured during the freeze?

[headmin]

sure both time I have been using Chrome in foreground, about 3-4 pages each with 3-6 tabs opened inside and only few other apps open: CodeRunner.app(nonMAS), Terminal.app, Messages.app, Slack (non MAS), Ulysses.app(MAS) . on today hang I also had VMware Fusion 11.0.2 open but this was not the case yesterdays hang so I doubt fusion involved directly

[headmin]

I removed Santa on this work box atm but keep all /var/db/santa and var/log/santa*files of course. our TLS server is Zentral, mobileconfig auto-created by zentral, KEXT whitelisting via MDM