Use a non-deprecated API for Keychain access

[NickBorgersOnLowSecurityNode]

Bug report

What operating system and version are you using?

 version = 12.6
   build = 21G115
platform = darwin

What version of osquery are you using?

Latest, the issue is in master.

What steps did you take to reproduce the issue?

We are seeing occasional corruption of Login Keychains across a fleet of 1000s of Mac OS devices. We have not observed any correlation (or significant occurence) of System Keychain corruption.

Our baseline was to be running SELECT * from certificates every 60 seconds as part of a device health compliance system. With this in-place as a query_pack we see ~0.3% of assets experience Login Keychain corruption every week, with the systems experiencing the bad event tending to experience it on an ongoing recurrent basis. There is a correlation with older hardware experiencing the brunt of the impact; newer systems are less likely to be affected.

What did you expect to see?

We did not expect osquery interacting with the keychain at any frequency to be correlated with corruption of the keychain.

What did you see instead?

When osquery interacts with the keychain frequently, we see occasional corruption of user keychains.

Attempt at RCA

Now that we have determined a correlation between osquery interacting with the keychain and corruption, we went looking at osquery source code and identified function calls like this:

osquery/osquery/tables/system/darwin/keychain_utils.cpp

Line 50 in b1ac296

auto status = SecKeychainOpen(keychain_path.c_str(), &keychain);

osquery/osquery/tables/system/darwin/keychain_utils.cpp

Line 68 in b1ac296

status = SecKeychainGetPath(keychain, &path_size, keychain_path);

Both of those functions, SecKeychainOpen and SecKeychainGetPath, are part of a set of MacOS APIs for Keychain management that were deprecated by MacOS 12.0 at the latest.

There do appear to be currently supported MacOS APIs for interacting with the keychain, but I'm not currently prepared to build and test those changes and haven't touched C/C++ since high school. We have a support contract with a vendor who supports our deployment and will be contacting them as well.

Sharing here so that:

• Someone on the project can tell me I've misinterpreted the code
• Someone else who's encountered this issue can find this and know they're not alone
• Collect facts in a shared place if/when this can be addressed

[directionless]

@sharvilshah you tend to be the most knowledgable here, any thoughts?

[directionless]

Hi @NickBorgersOnLowSecurityNode Thanks for reporting this.

My gut sense, is that the older APIs and the corruption are separate issues. While it would be nice for osquery to use the current APIs, I would expect any shipped apple API to cause corruption.

What kind of query are you running, and at what frequency?

Do you see this on any particular macOS versions?

Have you reported this to Apple? (no worries if not, I will eventually, and I'm trying to understand what's up)

[sharvilshah]

The certificates table uses OpenSSL APIs to parse the X509/ASN1 data from the keychain. AFAIK the only Keychain API that table uses is to enumerate and get the path I believe.

We updated that table to use OpenSSL APIs because there was a leak in the Keychain API, and it was slow (this was I believe 5 years ago), and also to encourage porting this to Linux.

@NickBorgersOnLowSecurityNode what does a corrupted keychain look like? Does it not open with Keychain access?

[directionless]

One of Kolide's customers is bumping this. I got a bit more info from them that I can share:

They see corruption on the login keychain. It is corrupt in macOS. The user gets weird errors, and on reboot macOS tells them Your login keychain is damaged and cannot be used. A new keychain has been created for you. This causes loss of any stored passwords.

They report some unified logs like:

osqueryd: (Security) [com.apple.securityd:security_exception] CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
osqueryd: (Security) [com.apple.securityd:security_exception] CSSM Exception: -2147413759 CSSMERR_DL_DATABASE_CORRUPT
osqueryd: (Security) [com.apple.securityd:integrity] dbBlobVersion() failed for a CssmError: -2147413759 CSSMERR_DL_DATABASE_CORRUPT

This appears on both Big Sur and Monterey. It is sporadic, but appears more often if they have minutely queries that involved the certificates table.

They do have an issue open with apple. And somewhat worrying is that they report Apple saying that "an abnormal amount of reads may lead to corruption of the keychain"

[NickBorgersOnLowSecurityNode]

What @directionless has shared is exactly what we have been seeing. In addition to losing saved passwords, the private keys for certificates that were stored in the Login Keychain are lost. This interferes with some authentication use cases where we would like to be using the Login Keychain of the user to provide maximum protection to credentials stored on the device.

What kind of query are you running, and at what frequency?

The part I can share now is SELECT * from certificates and we are running it every 60 seconds. I need to double-check appropriateness before I share the full query pack. The certificate information is used so the results of the query we collect include the identifying information of the certificates which makes lookups easier.

Do you see this on any particular macOS versions?

This appears on both Big Sur and Monterey. Across thousands of assets no clear version association pattern has emerged, though older hardware manifests the problem more frequently.

Have you reported this to Apple?

Yes, we have an open ticket with Apple. All we have gotten back is aligned with

"an abnormal amount of reads may lead to corruption of the keychain"

Probably silly idea based on this mention

We updated that table to use OpenSSL APIs because there was a leak in the Keychain API, and it was slow (this was I believe 5 years ago), and also to encourage porting this to Linux.

Would it make any difference to create a copy of the keychain and then interact with that copy?
Another layer would be whether you could inspect the "real" keychain to see if it has been changed, and keep the copy up to date based on that check. Then, regardless of how often you read from the copy you aren't needing to interact with the real keychain.

Quite possible this is anti pattern for how you want osquery to work, and I have no idea if there's a reliable way to see if the keychain has been updated. Perhaps as simple as checking a filesystem timestamp, but I'd generally use a checksum. I am still new to Mac so am not clear on whether the keychain is really a file or something that just gets represented that way a la /proc in Linux.

[directionless]

If we accept these apple comments at face value, I think we don't have good options. We could implement a caching layer, or inspect mod times. But that won't eliminate the possibility, just reduce it.

I'm not sure if we can copy the files -- I feel like there's a lot of weird apple frameworks? Maybe? TBD I guess.

I think Sharvil is trying to reproduce.

[zwass]

Some links Eric Kaiser provided in office hours:
Keychain file format: https://github.com/libyal/dtformats/blob/main/documentation/MacOS%20keychain%20database%20file%20format.asciidoc
Keychain data protection: https://support.apple.com/en-ca/guide/security/secb0694df1a/web

[directionless]

Chatting a little in office hours today, we wondered:

• what is the security tool doing?
• Can we parse these files directly? seph thought they were sqlite somewhere. (see Zach's links above)
• What if we did the silly thing and copied the files to a tmp dir first? (Or even in memory)

[getvictor]

Did some research into this issue today. Simply using SecItemCopyMatching without the deprecated API calls does not return certificates from SystemRootCertificates.keychain, which is most of the certificates on a typical system. I believe this is Apple's intention -- to not have API access to root certificates.

@zwass @directionless I will attempt to fix this by copying the files to tmp dir first.

[directionless]

Did some research into this issue today. Simply using SecItemCopyMatching without the deprecated API calls does not return certificates from SystemRootCertificates.keychain, which is most of the certificates on a typical system. I believe this is Apple's intention -- to not have API access to root certificates.

@zwass @directionless I will attempt to fix this by copying the files to tmp dir first.

I tried implementing with the new apis (#7822), but we had access to significantly less data. I don't think anyone has tried the copy to tmp approach.

[getvictor]

@zwass @directionless I opened a PR #8189 that fixes certificates and keychain_items by coping keychains.

The keychain_acls table doesn't seem to work with copies, so we'll need another approach there.

This is my first osquery PR. Early feedback will be appreciated.

[directionless]

Worth noting that we're still using the deprecated APIs -- we don't see any other way to get at the data. But #8192 adds a cache layer in which, hopefully, alleviates the corruption

[NickBorgersOnLowSecurityNode]

Worth noting that we're still using the deprecated APIs -- we don't see any other way to get at the data. But #8192 adds a cache layer in which, hopefully, alleviates the corruption

As original reporter, I believe my organization's pain point will be resolved with the caching solution. Thank you to all involved in the project and @getvictor for writing the fix.