nss: recent non-keg-only pull request crashes all recent LibreOffice releases

[megahall]

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

The problem happens if you are using basically any recent version of libreoffice (casked or non-casked, still or current release), after nss is installed in a non-keg-only mode (such as it is by default, after #43959 was committed).

What happened (include command output)

Libreoffice crashes, with a SIGSEGV and a very obscure backtrace deep in some SSL related code (note that, in order to capture this, it needs to be run using /Applications/LibreOffice.app/Contents/MacOS/soffice --norestore --backtrace and this is not really documented anywhere official or in the --help output).

Command output

$ /Applications/LibreOffice.app/Contents/MacOS/soffice --norestore --backtrace

(pkix_CacheCert_Add: PKIX_PL_HashTable_Add for Certs skipped: entry existed
Fatal exception: Signal 11

Stack:

/Applications/LibreOffice.app/Contents/Frameworks/libuno_sal.dylib.3+0x341e2(_ZN12_GLOBAL__N_110printStackEi+0x32)[0x108e771e2]

/Applications/LibreOffice.app/Contents/Frameworks/libuno_sal.dylib.3+0x340bc(_ZN12_GLOBAL__N_121signalHandlerFunctionEiP9__siginfoPv+0x2ec)[0x108e770bc]

/usr/lib/system/libsystem_platform.dylib+0x1f5a(_sigtramp+0x1a)[0x7fff56ec8f5a]

/usr/local/Cellar/nss/3.49/lib/libnssutil3.dylib+0x1f6d0(ucs2Utf8ConvertFunc+0xea8)[0x1afc7e6d0]

/usr/local/Cellar/nss/3.49/lib/libnssutil3.dylib+0xaeb8(SECOID_FindOID_Util+0x19)[0x1afc69eb8]

/usr/local/lib/libsmime3.dylib+0x11ebc(CERT_DecodeCertPackage+0x17c)[0x1afb30ebc]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0x14adcc(pkix_pl_HttpCertStore_DecodeCertPackage+0x1ec)[0x10c9e8dcc]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0x14b18d(pkix_pl_HttpCertStore_ProcessCertResponse+0x2ad)[0x10c9e918d]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0x144c53(pkix_pl_AIAMgr_GetHTTPCerts+0x783)[0x10c9e2c53]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0x145fb1(PKIX_PL_AIAMgr_GetAIACerts+0x431)[0x10c9e3fb1]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0xe71e8(pkix_BuildForwardDepthFirstSearch+0x9c8)[0x10c9851e8]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0xe4759(pkix_Build_InitiateBuildChain+0x1b89)[0x10c982759]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0xe2860(PKIX_BuildChain+0x180)[0x10c980860]

/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib+0x1c236(CERT_PKIXVerifyCert+0x2a6)[0x10c8ba236]

/Applications/LibreOffice.app/Contents/Frameworks/libxsec_xmlsec.dylib+0x1b4d4(_ZN27SecurityEnvironment_NssImpl17verifyCertificateERKN3com3sun4star3uno9ReferenceINS2_8security12XCertificateEEERKNS3_8SequenceIS7_EE+0x404)[0x1b4d4f4d4]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x16675(_ZN10webdav_ucp11NeonSession19CertificationNotifyEPK20ne_ssl_certificate_s+0x3d5)[0x1b3ca1675]

/Applications/LibreOffice.app/Contents/Frameworks/libneon.dylib+0x12a7a(ne__negotiate_ssl+0x29a)[0x1b3d25a7a]

/Applications/LibreOffice.app/Contents/Frameworks/libneon.dylib+0xa5f6(send_request+0x496)[0x1b3d1d5f6]

/Applications/LibreOffice.app/Contents/Frameworks/libneon.dylib+0x9ac4(ne_begin_request+0xe4)[0x1b3d1cac4]

/Applications/LibreOffice.app/Contents/Frameworks/libneon.dylib+0xae18(ne_request_dispatch+0x28)[0x1b3d1de18]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x19670(_ZN10webdav_ucp11NeonSession7OPTIONSERKN3rtl8OUStringERNS_10DAVOptionsERKNS_21DAVRequestEnvironmentE+0x100)[0x1b3ca4670]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x77ba(_ZN10webdav_ucp17DAVResourceAccess7OPTIONSERNS_10DAVOptionsERKN3com3sun4star3uno9ReferenceINS5_3ucb19XCommandEnvironmentEEE+0x19a)[0x1b3c927ba]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x42353(_ZN10webdav_ucp7Content18getResourceOptionsERKN3com3sun4star3uno9ReferenceINS3_3ucb19XCommandEnvironmentEEERNS_10DAVOptionsERKNSt3__110unique_ptrINS_17DAVResourceAccessENSD_14default_deleteISF_EEEEPb+0xb3)[0x1b3ccd353]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x389a4(_ZN10webdav_ucp7Content4openERKN3com3sun4star3ucb20OpenCommandArgument3ERKNS3_3uno9ReferenceINS4_19XCommandEnvironmentEEE+0x734)[0x1b3cc39a4]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x2f49c(_ZN10webdav_ucp7Content7executeERKN3com3sun4star3ucb7CommandEiRKNS3_3uno9ReferenceINS4_19XCommandEnvironmentEEE+0xb0c)[0x1b3cba49c]

/Applications/LibreOffice.app/Contents/Frameworks/libucpdav1.dylib+0x3e5d2(_ZThn64_N10webdav_ucp7Content7executeERKN3com3sun4star3ucb7CommandEiRKNS3_3uno9ReferenceINS4_19XCommandEnvironmentEEE+0x12)[0x1b3cc95d2]

/Applications/LibreOffice.app/Contents/Frameworks/libupdatefeedlo.dylib+0x8460(_ZN12_GLOBAL__N_125UpdateInformationProvider4loadERKN3rtl8OUStringE+0x440)[0x1af99d460]

/Applications/LibreOffice.app/Contents/Frameworks/libupdatefeedlo.dylib+0x3d7e(ZN12_GLOBAL__N_125UpdateInformationProvider31getUpdateInformationEnumerationERKN3com3sun4star3uno8SequenceIN3rtl8OUStringEEERKS7+0xce)[0x1af998d7e]

/Applications/LibreOffice.app/Contents/Frameworks/libupdatefeedlo.dylib+0x5102(ZThn40_N12_GLOBAL__N_125UpdateInformationProvider31getUpdateInformationEnumerationERKN3com3sun4star3uno8SequenceIN3rtl8OUStringEEERKS7+0x12)[0x1af99a102]

/Applications/LibreOffice.app/Contents/Frameworks/libupdchklo.dylib+0x2a688(Z15checkForUpdatesR10UpdateInfoRKN3com3sun4star3uno9ReferenceINS4_17XComponentContextEEERKNS5_INS3_4task19XInteractionHandlerEEERKNS5_INS3_10deployment26XUpdateInformationProviderEEERKN3rtl8OUStringESN_RKNS4_8SequenceISL_EESN_SN+0xf8)[0x1b27f0688]

/Applications/LibreOffice.app/Contents/Frameworks/libupdchklo.dylib+0x2a409(_Z15checkForUpdatesR10UpdateInfoRKN3com3sun4star3uno9ReferenceINS4_17XComponentContextEEERKNS5_INS3_4task19XInteractionHandlerEEERKNS5_INS3_10deployment26XUpdateInformationProviderEEE+0x2b9)[0x1b27f0409]

/Applications/LibreOffice.app/Contents/Frameworks/libupdchklo.dylib+0xd6d6(_ZN12_GLOBAL__N_117UpdateCheckThread8runCheckERb+0x116)[0x1b27d36d6]

/Applications/LibreOffice.app/Contents/Frameworks/libupdchklo.dylib+0xd29c(_ZN12_GLOBAL__N_117UpdateCheckThread3runEv+0x19c)[0x1b27d329c]

/Applications/LibreOffice.app/Contents/Frameworks/libupdchklo.dylib+0xb49f(threadFunc+0xf)[0x1b27d149f]

/Applications/LibreOffice.app/Contents/Frameworks/libuno_sal.dylib.3+0x3726e(_ZL21osl_thread_start_ImplPv+0x7e)[0x108e7a26e]

/usr/lib/system/libsystem_pthread.dylib+0x3661(_pthread_body+0x154)[0x7fff56ed2661]

/usr/lib/system/libsystem_pthread.dylib+0x350d(_pthread_body+0x0)[0x7fff56ed250d]

/usr/lib/system/libsystem_pthread.dylib+0x2bf9(thread_start+0xd)[0x7fff56ed1bf9]

Abort trap: 6
  

What you expected to happen

I expected libreoffice to operate properly, without any spooky action-at-a-distance side effects from nss.

Step-by-step reproduction instructions (by running brew install commands)

brew install nss
brew cask install libreoffice
[unknown set of conditions, triggered by opening an XLS spreadsheet for me]
libreoffice crashes due to nss conflict of some kind

I had to work around the problem by uninstalling nss and breaking some other brews I use...

@swrobel @fxcoudert

brew config and brew doctor

$ brew config
HOMEBREW_VERSION: 2.2.3-4-gfc0777b
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: fc0777b29181b6789aebe2435171c7044b84a9c4
Last commit: 4 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: f2463cbba086689c872cddd3a8bc242976a603ff
Core tap last commit: 25 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_BREW_GIT_REMOTE: https://github.com/Homebrew/brew
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_DEV_CMD_RUN: 1
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_NO_ANALYTICS_THIS_RUN: 1
HOMEBREW_NO_INSTALL_CLEANUP: 1
CPU: octa-core 64-bit ivybridge
Homebrew Ruby: 2.6.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3/bin/ruby
Clang: 10.0 build 1000
Git: 2.25.0 => /usr/local/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 1.8.0_232
macOS: 10.13.6-x86_64
CLT: 10.1.0.0.1.1539992718
Xcode: N/A
XQuartz: 2.7.11 => /opt/X11

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: You have the following deprecated, official taps tapped:
  Homebrew/homebrew-science
Untap them with `brew untap`.

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/libnanomsg.0.dylib

Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected header files:
  /usr/local/include/nanomsg/bus.h
  /usr/local/include/nanomsg/inproc.h
  /usr/local/include/nanomsg/ipc.h
  /usr/local/include/nanomsg/nn.h
  /usr/local/include/nanomsg/pair.h
  /usr/local/include/nanomsg/pipeline.h
  /usr/local/include/nanomsg/pubsub.h
  /usr/local/include/nanomsg/reqrep.h
  /usr/local/include/nanomsg/survey.h
  /usr/local/include/nanomsg/tcp.h

Warning: Unbrewed .la files were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .la files:
  /usr/local/lib/libnanomsg.la

Warning: Unbrewed .pc files were found in /usr/local/lib/pkgconfig.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .pc files:
  /usr/local/lib/pkgconfig/libnanomsg.pc

Warning: Unbrewed static libraries were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected static libraries:
  /usr/local/lib/libnanomsg.a

Warning: You have uncommitted modifications to Homebrew.
If this is a surprise to you, then you should stash these modifications.
Stashing returns Homebrew to a pristine state but can be undone
should you later need to do so for some reason.
  cd /usr/local/Homebrew/Library && git stash && git clean -d -f

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install md5sha1sum nss

Run `brew missing` for more details.

[swrobel]

This sounds like a LibreOffice issue. @megahall have you filed an issue with them? If so, can you link to it here?

[swrobel]

Also, I just tested w/ LibreOffice 6.3.4.2 on macOS 10.14.6 w/ keg nss & couldn't reproduce a crash after opening several Excel sheets (both XLS as you mentioned as well as the newer XLSX)

[megahall]

I do believe keg mode would fix it, but non keg mode would trigger it, conditionally, if the specific code paths using NSS get hit. I probably do need to file a bug with them as well. I am early into the diagnostic process and I wanted to file this partly as a warning to others to raise awareness and generate discussion of this problem, because it took a really long time to track this down during this week.

[fxcoudert]

I can confirm the bug here. Something that triggers the LibreOffice crash for me in a systematic manner is “Check for updates”. brew unlink nss is a workaround that will fix the issue without removing nss and breaking other Homebrew apps

This is probably a problem in LibreOffice, should be filed with them so they fix it. There are three possible libraries that LibreOffice includes and that could be an issue:

/Applications/LibreOffice.app/Contents/Frameworks/libnssckbi.dylib
/Applications/LibreOffice.app/Contents/Frameworks/libnssutil3.dylib
/Applications/LibreOffice.app/Contents/Frameworks/libnss3.dylib

but it should be possible (and highly desirable) for LibreOffice to load its libraries over those in /usr/local/lib

[fxcoudert]

Actually, the bug was already reported in September 2019 to LibreOffice, but the developers apparently couldn't reproduce (although several users reported it), and they did not figure out the link with Homebrew. So thanks @megahall you rock! I've added a long comment on the issue: https://bugs.documentfoundation.org/show_bug.cgi?id=127619#c41

[megahall]

Ironically, I did see that bug while hunting through the LibreOffice bugs, but I didn't think it quite applied to me, because the bug title and description were not that close to how the problem behaved on my machine. On my machine I had to be editing a document and that sent me down a different mental path. Luckily I've got enough previous C and infosec experience that I was able to collect the data in a different way that clarified the actual problem some more. Thanks very much for helping me make the connection from point A to point B. I had a hard time figuring out what to do. Matthew Hall

[fxcoudert]

LibreOffice has a fix, we have a workaround documented above, so I'm closing the bug.

Thanks @megahall for the report and for providing a backtrace!