Scep enrollment failed with 3des algorithm and attached HSM

[pki-bot]

This issue was migrated from Pagure Issue #2906. Originally filed by mharmsen (@mharmsen) on 2018-01-17 18:44:15:

• Assigned to nobody

---

FloFromAT opened this Issue on Oct 31, 2017

We’ve been using dogtag version 10.4 together with_jss_ version 4.4 on a up-to-date RHEL7 and Safenet HSMs for some years for handling SCEP requests.
We’re running into an issue again which we also had in the past: when using the HSM (a requirement here), only SCEP requests using DES for the encryption
can be decoded. When DES3 is used, dogtag throws an error with “could not unwrap PKCS10 blob”. With no HSM, both algorithms work.

However, the DES3 requests themselves are OK: we can unpack the inner pkcs7, and decrypt the payload using ‘cmsutil’ (pointed at the nss db of the CA instance)
and read the pkcs10 request within. So the HSM itself has no problem decrypting.

We also encountered this issue in the past with RHEL6 / DogTag 9, and it is still present with RHEL7 / Dogtag 10. At that time, we were able to configure the clients
to use DES to avoid the issue, but we can’t always dictate which algorithm the clients use,
and DES is nevertheless very weak.

It may still be related the old BZ: https://bugzilla.redhat.com/show_bug.cgi?id=825887 and be an issue with the FIPS-2 mode (which we are using)

It appears to be an issue with Dogtag. If someone has a suggestion or idea, we would appreciate hearing it.

Below you can find all needed parameters and config which we used.

CA is an subca - (But dont matter, because the same issue occures also on an root ca)
SCEP enrollment works with DES encryption (HSM attached)
SCEP enrollment with DES3 works when NO hsm is used
SCEP requests (DES + 3DES) can be decoded when using cmsutil direct against the HSM libary. (cmsutil -d /var/lib/pki/pkit-tomcat/alias -D -i inner_pkcs7_request.p7 -o request_des3.der )

CS.cfg

ca.scep.allowedEncryptionAlgorithms=DES3,DES
ca.scep.allowedHashAlgorithms=SHA1,SHA256,SHA512
ca.scep.enable=true
ca.scep.encryptionAlgorithm=DES3
ca.scep.hashAlgorithm=SHA1
ca.scep.nonceSizeLimit=16

SSECP call
./sscep enroll -u http://pki-test.example.com:8080/ca/cgi-bin/pkiclient.exe -c ca.crt -k local.key -r local.csr -l cert.crt -S sha1 -E 3des

debug.log

[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:263:init() CRSEnrollment: init: SCEP support is enabled.
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:264:init() CRSEnrollment: init: SCEP nickname: pkit04:caSigningCert cert-pkit04 CA
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:265:init() CRSEnrollment: init: CA nickname: pkit04:caSigningCert cert-pkit04 CA
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:266:init() CRSEnrollment: init: Token name: pkit04
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:267:init() CRSEnrollment: init: Is SCEP using CA keys: true
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:268:init() CRSEnrollment: init: mNonceSizeLimit: 16
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:269:init() CRSEnrollment: init: mHashAlgorithm: SHA1
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:270:init() CRSEnrollment: init: mHashAlgorithmList: SHA1,SHA256,SHA512
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:273:init() CRSEnrollment: init: mAllowedHashAlgorithm[0]=SHA1
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:273:init() CRSEnrollment: init: mAllowedHashAlgorithm[1]=SHA256
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:273:init() CRSEnrollment: init: mAllowedHashAlgorithm[2]=SHA512
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:275:init() CRSEnrollment: init: mEncryptionAlgorithm: DES3
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:276:init() CRSEnrollment: init: mEncryptionAlgorithmList: DES3,DES
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:279:init() CRSEnrollment: init: mAllowedEncryptionAlgorithm[0]=DES3
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:279:init() CRSEnrollment: init: mAllowedEncryptionAlgorithm[1]=DES
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:285:init() CRSEnrollment: init: mProfileId=caRouterCert
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:349:service() operation=PKIOperation
[03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:351:service() message=MIIKywYJKoZIhvcNAQcCoIIKvDCCCrgCAQExCzAJBgUrDgMCGgUAMIIFnwYJKoZI
-...snip..
t3fqG6FkBAh3L1saONZJ0pfzOnnY5CZ4aJuf5ql3XA==

[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: CRSEnrollment.java:920:handlePKIOperation() Processing PKCSReq
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:324:getConn() In LdapBoundConnFactory::getConn()
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:326:getConn() masterConn is connected: true
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:368:getConn() getConn: conn is connected true
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:398:getConn() getConn: mNumConns now 5
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:444:returnConn() returnConn: mNumConns now 6
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: CRSEnrollment.java:1164:unwrapPKCS10() failed to unwrap PKCS10 org.mozilla.jss.crypto.SymmetricKey$NotExtractableException
[03/Oct/2017:07:35:53][http-bio-8080-exec-1]: CRSEnrollment.java:385:service() ServletException javax.servlet.ServletException: Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: null

localhost_access.log
10.10.10.10 - - [02/Oct/2017:11:09:27 +0200] "GET / ca / cgi-bin / pkiclient . exe ? operation = PKIOperation & message = MIIKzgYJKoZIhvcNAQcCoIIKvz...snip.. HTTP/1.0" 500 3071

localhost.log

SEVERE: Servlet.service() for servlet [caSCEP] in context with path [/ca] threw exception [Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: null] with root cause
javax.servlet.ServletException: Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: null
at com.netscape.cms.servlet.cert.scep.CRSEnrollment.service(CRSEnrollment.java:386)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Comment by a java developer which has debugged the code and found that "issue" below. ( Thanks to MAT ).

In file CRSEnrollment.java in method unwrapPKCS10 the call delegation to the jss library fails gracefully at sk = kw.unwrapSymmetric( with the following excpetion:

[30/Oct/2017:07:49:15][http-bio-8080-exec-21]: failed to unwrap PKCS10 org.mozilla.jss.crypto.TokenException: Failed to unwrap key
[30/Oct/2017:07:49:15][http-bio-8080-exec-21]: ServletException javax.servlet.ServletException: Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: Failed to unwrap key

The error comes clearly from the jss JNI library which is forwarded to the pki stack. I was not able to debug the code path. I tried to reproduce the use case in the jss library as well however it was not possible because the KeyWrapper class that is used here is marked as deprecated and shall be replaced by JCA. No test suite exists that tests the KeyWrapper even in very old relaeses of jss back to 2000 only the new JCA interface API is tested.

I hope you can provide a patch or a solution, because at the moment we are not able to issue certificates for clients which are using 3DES with scep.

Thanks in advanced.

BR
Flo

[static0verdrive]

I am getting this same error in Dogtag 11's debug log, after allowing DES and MD5 (just for testing in a non-Prod environment, in order to confirm whether SCEP was working)... The client was an old Cisco ASA 5512 running IOS 9.9, and we were getting errors about DES and then MD5 not being allowed, but at least it seems like I have the enrollment URL correct. The client will try to put 9.12 on their ASA and I will try with a newer client device as well.