p11-kit-proxy as libnssckbi replacement breaks HTTPS in Firefox

[fdelapena]

Hi,

I'm using p11-kit-proxy in Fedora 28 to make Firefox work with system wide pkcs11 proprietary modules. I have a proprietarylib.so in a proprietaryconfig.module file and works nicely in all p11-kit supported apps. For NSS is well known that Firefox is not using the pkcs11.txt from /etc/pki/nssdb, then so far I was replacing libnssckbi.so with the p11-kit-proxy replacement as the documentation was suggesting:

sudo alternatives --install /usr/lib64/libnssckbi.so libnssckbi.so.x86_64 /usr/lib64/p11-kit-proxy.so 50
sudo alternatives --auto libnssckbi.so.x86_64

With the latest p11-kit update in Fedora, which adds the disable-in: p11-kit-proxy change in p11-kit-trust.module, Firefox won't trust any HTTPS site anymore. It works by commenting out this line as a workaround in the meanwhile. However, is this issue on purpose or an unexpected side effect? I'm aware of upcoming improvements targeted for Fedora 29, if they are related somehow.

Thanks.

[ueno]

That was actually for this Fedora 29 Change, where I am thinking a different way of registering p11-kit-proxy in NSS databases:

1. create /etc/crypto-policies/local.d/nss-p11-kit.config with the following content (be aware of the last empty line):

library=p11-kit-proxy.so
name=p11-kit-proxy

2. run update-crypto-policies

Then p11-kit-proxy.so should be loaded by all new/existing NSS databases. Would you be able to check if that works?

[fdelapena]

Thanks, after restoring the symlink and trying the crypto-policies configuration provided, it loads as expected, p11-kit-proxy module is shown above trust modules in Firefox security devices configuration.

However, when I try to add the .module as usual in /etc/pkcs11/modules/ or /usr/share/p11-kit/modules/, Firefox tabs will crash. When run from the console it displays seccomp-based sandbox violations:

Sandbox: seccomp sandbox violation: pid 7323, tid 7323, syscall 64, args 19755611 1 1974 438 0 140585357805504.
Sandbox: seccomp sandbox violation: pid 7323, tid 7323, syscall 64, args 19755611 1 0 438 0 140585357805504.
Sandbox: seccomp sandbox violation: pid 7323, tid 7323, syscall 64, args 19755611 1 1974 0 0 140585357805504.
Sandbox: seccomp sandbox violation: pid 7323, tid 7323, syscall 64, args 19755611 1 0 0 0 140585357805504.
terminate called after throwing an instance of 'COsUtils::GeneralError'
Redirecting call to abort() to mozalloc_abort

[Parent 7210, Gecko_IOThread] WARNING: pipe error (56): Connection reset by peer: file /builddir/build/BUILD/firefox-60.0.2/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[Parent 7210, Gecko_IOThread] WARNING: pipe error (57): Connection reset by peer: file /builddir/build/BUILD/firefox-60.0.2/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353

###!!! [Parent][MessageChannel] Error: (msgtype=0x150066,name=PBrowser::Msg_RealMouseMoveEvent) Channel error: cannot send/recv


###!!! [Parent][MessageChannel] Error: (msgtype=0x2C0105,name=PContent::Msg_AsyncMessage) Channel error: cannot send/recv


###!!! [Parent][MessageChannel] Error: (msgtype=0x15007F,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv

abrt also sends a FAF report with the segfault backtrace.

[nmav]

Does the module work if you load it manually from firefox? It looks to me that the semaphore creation failed due to firefox sandbox requirements.

A hardware card may have a legitimate reason to use a semaphore to prevent multiple uses of the hardware, and if I understand well this page, firefox in general breaks these use cases.

A long shot, but if my speculation above is correct, using p11-kit server and putting your PKCS#11 module in a separate process may work.

[fdelapena]

Loading p11-kit-proxy.so manually from Firefox loads the proprietary module without issues, same when trying to load the proprietary .so directly.

I can confirm setting security.sandbox.content.level to 0 in about:config makes tabs work. The module however crashes Firefox when trying token login.

The sandbox tab crash also happens when trying to set the proprietary .so in the nss-p11-kit.config directly (instead of the p11-kit-proxy.so).

[ueno]

It's interesting that the behavior changes when the proxy module is loaded manually. At the NSS database level, the only difference I can see is that the policy configuration is processed a bit too late.

Are you able to check if the following scratch build improves the situation?
https://koji.fedoraproject.org/koji/taskinfo?taskID=27746619

[fdelapena]

I'm sorry, but it seems my test in the first reply was flawed. I didn't do a system restart to see more effects.

After restarting, not just firefox tabs were crashing, but every p11-kit based app: a simple p11-kit list-modules crashes (invalid pointer), firefox not even starting and returning the same message and gnome settings daemon's smartcard plugin crashes on startup too. I've also tried with the scratch build and restarted with the same effect.

Maybe the p11-kit FAF report helps with this. It looks similar to the tab crash backtrace.

[ueno]

I am afraid I can't tell anything from the FAF report: the call sequence in p11-kit seems fine.

• Are you able to access the proprietary module with p11tool (from the gnutls-utils package) or pkcs11-tool (from the opensc package) directly? For example, pkcs11-tool --module /usr/lib64/module.so -L
• Does this happen if you de-register the proprietary module? i.e., by removing it from /usr/lib64/pkcs11

[fdelapena]

Applying crypto policies (and system restart) having the proprietary .module file with module: libASEP11.so, it makes both p11tool and pkcs11-tool crash, maybe is related with semaphores as suggested by @nmav:

p11tool (trace)

$ p11tool --provider=/usr/lib64/libASEP11.so --list-tokens
munmap_chunk(): invalid pointer
Aborted (`core' generado)

pkcs11-tool (trace)

$ pkcs11-tool --module=/usr/lib64/libASEP11.so -L
munmap_chunk(): invalid pointer
Aborted (`core' generado)

It also may crash the whole GNOME session few seconds after login (with both Wayland and Xorg).

When I comment out the line module: libASEP11.so from my /usr/share/p11-kit/modules/proprietary.module (needs system restart for crypto policies), p11-kit does not crash as expected and p11-tool and pkcs11-tool work without issues:

$ p11tool --provider=/usr/lib64/libASEP11.so --list-tokens
Token 0:
	URL: pkcs11:model=IDProtect;manufacturer=Athena%20Smartcard%20Solutions;serial=XXXXXXXXXXXXXXX;token=IDProtect%23XXXXXXXXXXXXXXX
	Label: IDProtect#XXXXXXXXXXXXXXX
	Type: Hardware token
	Flags: RNG, Requires login
	Manufacturer: Athena Smartcard Solutions
	Model: IDProtect
	Serial: XXXXXXXXXXXXXXX
	Module: (null)

$ pkcs11-tool --module=/usr/lib64/libASEP11.so -L
Available slots:
Slot 0 (0x0): Athena ASE IIIe [CCID Bulk Interface] 00 00
  token label        : IDProtect#XXXXXXXXXXXXXXX
  token manufacturer : Athena Smartcard Solutions
  token model        : IDProtect
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 1.0
  firmware version   : 1.0
  serial num         : XXXXXXXXXXXXXXX
  pin min/max        : 4/16

I've moved the lib from /usr/lib64/pkcs11/ to /usr/lib64/ just to discard related issues, before I was using a symlink, but it does not change any behaviour.

[nmav]

My guess is that when the module is registered globally, it remains initialized by a system process and that makes it crash on subsequent uses preventing any other process initializing it. I'd suggest to try some smart card which has a decent driver.

[fdelapena]

Thanks, it looks like that. Unfortunately there is no way to replace the smart card in my case, they're electronic IDs provided by public institutions and they don't work with OpenSC or other decent drivers. There are no driver updates as far I know neither for this particular model (module middleware seems to be under license). Sadly, there is also risk to happen with other vendors with broken modules. However, I wonder why it's working with p11-kit-proxy as nssckbi replacement without colliding while using other tools (p11tool, pkcs11-tool) just after initializing and using it in Firefox or Evolution.

[ueno]

What do you see if you run: lsof /usr/lib64/libASEP11.so when you see the crash?

[fdelapena]

Thanks for the hint, after confirming lsof was showing the gsd-smartcard using it (used for pam_pkcs11 login I guess), I've tried killing the involved process (uid 42, gdm), then lsof was clean. However, it was not the issue, processes were being killed anyways.

After diving a bit deeper with strace for all crashing apps, there was a permission denied issue while trying to read/write files in /tmp just before every crash, not happening when it works. Due to the proprietary module library design, it creates the same lock file names in /tmp, so when gdm user calls the proprietary library, it creates files there with user gdm and files remain there. Then, when trying to login, another gsd-smartcard service instance, this time for the user (e.g. uid 1000) crashes. When gsd-smartcard process crashes, GNOME session terminates. This happens because the proprietary .so module is unable to overwrite lock files created by gdm user. That's why other programs (p11tool, pkcs11-tool) run by the user are crashing too. After doing chown to these files in /tmp to be owned by the desktop user, desktop login and all apps (except Firefox tabs, different issue) work as expected. Also, lsof displays multiple apps accessing to the proprietary .so without issues.

If I recall correctly, modern kernel security measures disallow other users (including root) work with files created by other users in /tmp, even if they explicitly use 0666 in permissions (the proprietary module tries this). I'll try to contact with the vendor (tried years ago, no luck yet) for the issue with /tmp.

Firefox tabs issue still persists, perhaps yet another proprietary module problem, I'll try to add feedback later. Sorry for the noise so far. Probably I'll stick with the nssckbi workaround if still possible with Fedora 29.

[ueno]

Thank you for the detail investigation; indeed, that explains the issue. I have opened a MR in gdm for that:
https://gitlab.gnome.org/GNOME/gdm/merge_requests/10

[fdelapena]

Thank you, it looks like there may need another approach to be accepted.

I've found it does not crash when those files are readable by the current user. Files in /tmp created by the smartcard module have rw- --- --- permissions, however adding read access for other users seems to be enough for this particular smartcard driver. Maybe calling umask() somewhere before calling NSS_InitContext in g-s-d smartcard plugin is possible. Something like | S_IROTH to umask flags might allow the NSS call initialize the lib with readable permissions.

	current_umask = umask(077);
	umask(current_umask | S_IROTH);

[ueno]

Another idea might be to isolate the card access in a separate chroot, using the remote: option with tools like bwrap or unshare. For example, I can wrap the access to opensc-pkcs11.so by modifying opensc.module:

# module: opensc-pkcs11.so
remote: |bwrap --unshare-all --dir /tmp --proc /proc --dev /dev --ro-bind /usr /usr --symlink /usr/lib64 /lib64 --ro-bind /var/run/pcscd /var/run/pcscd p11-kit remote /usr/lib64/opensc-pkcs11.so

[fdelapena]

Thank you for the sandbox approach, definitely looks a safer a better way. I've tried with bubblewrap and worked nicely:

remote: |bwrap --unshare-all --dir /tmp --ro-bind /etc/Athena /etc/Athena --proc /proc --dev /dev --ro-bind /usr /usr --symlink /usr/lib64 /lib64 --ro-bind /run/pcscd /run/pcscd p11-kit remote /usr/lib64/libASEP11.so

Firefox tabs are not crashing anymore. However, token login still crashes NSS based apps (Firefox, Evolution, certutil). At first sight, it seems to communicate properly with the card, because when trying with invalid PIN it prompts again for another retry. I've also tried with the scratch build from above with the same result.

Here's a backtrace while trying login, at nssTrustDomain_UpdateCachedTokenCerts():

$ gdb --args certutil -d sql:/etc/pki/nssdb -L -h IDProtect#XXXXXXXXXXXXXXXX
GNU gdb (GDB) Fedora 8.1-15.fc28
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from certutil...Reading symbols from /usr/lib/debug/usr/bin/certutil-3.37.3-1.1.fc28.x86_64.debug...done.
done.
(gdb) r
Starting program: /usr/bin/certutil -d sql:/etc/pki/nssdb -L -h IDProtect\#XXXXXXXXXXXXXXXX
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Detaching after fork from child process 7777.

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Enter Password or Pin for "IDProtect#XXXXXXXXXXXXXXXX":

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6dea5e7 in PK11_DoPassword (slot=0x5555558f0d00, session=17, loadCerts=<optimized out>, wincx=0x7fffffffd520, alreadyLocked=0, 
    contextSpecific=0) at pk11auth.c:640
640	            nssTrustDomain_UpdateCachedTokenCerts(slot->nssToken->trustDomain,
(gdb) bt f
#0  0x00007ffff6dea5e7 in PK11_DoPassword (slot=0x5555558f0d00, session=17, loadCerts=<optimized out>, wincx=0x7fffffffd520, alreadyLocked=0, 
    contextSpecific=0) at pk11auth.c:640
        rv = SECSuccess
        password = 0x5555558f4290 ""
        attempt = 1
        loadCerts = <optimized out>
        contextSpecific = <optimized out>
        wincx = <optimized out>
        slot = <optimized out>
        rv = <optimized out>
        attempt = <optimized out>
        alreadyLocked = <optimized out>
        session = <optimized out>
        password = <optimized out>
        password = <optimized out>
#1  0x00005555555622fa in listCerts (pwarg=0x7fffffffd520, outfile=0x555555782850, extensionOID=0x0, ascii=0, raw=0, slot=0x5555558f0d00, email=0x0, 
    name=0x0, handle=0x5555558e13f0) at certutil.c:548
        newrv = <optimized out>
        rv = SECFailure
        certs = <optimized out>
        node = <optimized out>
        rv = <optimized out>
        certs = <optimized out>
        node = <optimized out>
        newrv = <optimized out>
        the_cert = <optimized out>
#2  ListCerts (handle=0x5555558e13f0, nickname=0x0, email=0x0, slot=0x5555558f0d00, raw=0, ascii=0, extensionOID=0x0, outfile=0x555555782850, 
    pwdata=0x7fffffffd520) at certutil.c:650
        rv = <optimized out>
#3  0x0000555555564e74 in certutil_main (argc=<optimized out>, argv=<optimized out>, initialize=<optimized out>) at certutil.c:3342
        certHandle = 0x5555558e13f0
        slot = <optimized out>
        subject = 0x0
        inFile = 0x5555557827e0
        outFile = 0x555555782850
        certReqDER = {type = siBuffer, data = 0x0, len = 0}
---Type <return> to continue, or q <return> to quit---
        certDER = {type = siBuffer, data = 0x0, len = 0}
        slotname = <optimized out>
        certPrefix = 0x555555571575 ""
        sourceDir = 0x555555571575 ""
        srcCertPrefix = 0x555555571575 ""
        upgradeID = 0x555555571575 ""
        upgradeTokenName = 0x555555571575 ""
        keytype = rsaKey
        name = 0x0
        newName = 0x0
        email = 0x0
        keysource = 0x0
        hashAlgTag = SEC_OID_UNKNOWN
        keysize = 2048
        publicExponent = 65537
        certVersion = 2
        serialNumber = 0
        warpmonths = 0
        validityMonths = 3
        commandsEntered = 1
        commandToRun = <optimized out>
        pwdata = {source = PW_NONE, data = 0x0}
        pwdata2 = {source = PW_NONE, data = 0x0}
        readOnly = <optimized out>
        initialized = 1
        keyOpFlagsOn = 0
        keyOpFlagsOff = 0
        keyAttrFlags = 69
        privkey = 0x0
        pubkey = 0x0
        i = <optimized out>
        rv = SECSuccess
#4  0x000055555555bf1f in main (argc=<optimized out>, argv=<optimized out>) at certutil.c:3880
        rv = <optimized out>

[ueno]

Yes, the crash is known and I will include the patch in the next package update:
https://bugzilla.mozilla.org/show_bug.cgi?id=1296263

[fdelapena]

I confirm the nss-3.38.0-1.0.fc28 update candidate, which backports the nss-load-policy-file.patch from 3.39 fixes the issue here with the new approach. Thank you!

[joakim-tjernlund]

I just hit this problem on Gentoo, upgrading from p11-kit-0.23.9 to 0.23.12
FF works if I comment the "disable-in: p11-kit-proxy" line, but after that I am lost.
Gentoo does not have the /etc/crypto-policies/local.d/ dir or the update-crypto-policies cmd
I do have a /etc/pkcs11/ dir but it did n't work for me.
What's next?

[fdelapena]

@joakim-tjernlund I'd suggest checking what update-crypto-policies does in Fedora 29. It probably updates the sql:/etc/pki/nssdb by adding the p11-kit-proxy module via modutil. In some distros, Firefox and other don't use the system wide NSS database by default and may require setting an environment variable to enable it. Note downstream distro still applies patches for features not available upstream for better crypto experience.

[joakim-tjernlund]

@fdelapena I need someone to tell me what Fedora does. I have do not have access to a Fedora install and trying to figure out this by installing Fedora on some host and dig through the install is just too much.