Error: failed to load key manager: error initializing PKCS#11: could not open PKCS#11 only when I running step kms outside of pod in k8s

[umegaya]

hi, we try to use step-ca with Thales Proect Server 3+ External with Protect Tool Kit (PTK) 7.2.0.

with PTK emurator mode (did not use actual hardware), step kms sign can create token inside/outside of kubernetes pod without any problem, by using kubectl exec -ti $pod_name. like following

$ kubectl exec -ti $pod_name bash -- step kms sign --in data.jwt --format jws --kms 'pkcs11:module-path=/opt/safenet/protecttoolkit7/ptk/lib/libcryptoki.so;token=$token?pin-value=$pin' 'pkcs11:id=$id'
Defaulted container "main" out of: main, pkcs-tool

n858qRj(redacted)w3-3nyo_nxg

but with actual hardware (Thales Proect Server 3+ External), we got following error. we believe we did setup hsm correctly, because we could create step-ca's root-ca/intermediate-ca key object into the HSM by using step certificate create.

$ kubectl exec -ti $pod_name bash -- step kms sign --in data.jwt --format jws --kms 'pkcs11:module-path=/opt/safenet/protecttoolkit7/ptk/lib/libcryptoki.so;token=$token?pin-value=$pin' 'pkcs11:id=$id'
Defaulted container "main" out of: main, pkcs-tool

Error: failed to load key manager: error initializing PKCS#11: could not open PKCS#11
exit status 1
command terminated with exit code 1

strange thing, if I run the command after logged into pods, error does not happen.

$ kubectl exec -ti smallstep-66dcc85c64-fdjpm -- bash
Defaulted container "main" out of: main, pkcs-tool

# step kms sign --in data.jwt --format jws --kms 'pkcs11:module-path=/opt/safenet/protecttoolkit7/ptk/lib/libcryptoki.so;token=$token?pin-value=$pin' 'pkcs11:id=$id'
aq8DNrx(redacted)zWdv7-f_Is6j-Wg

not only step kms sign but also other command that access to actual hardware (Thales Proect Server 3+ External) like step kms encrypt got same error.

any idea why these commands fails only when run from outside of container (by using kubectl exec -ti $pod -- $command)?

regards,

[umegaya]

further investigation reveals ptk itself fails when it called via kubectl -ti exec $pod -- $command. so I will ask thales support first and close the issue.

sorry for noise!

• when called via kubectl -ti exec $pod -- $command => fails C_Initialize GetFunctionTable(/opt/safenet/protecttoolkit7/ptk/lib/libcthsm.so) failed

pid(67) tid(140454142105344) time(2023-02-13 05:03:40.059)      > C_GetFunctionList ppFunctionList=0x0x7fbe00000cd8
pid(67) tid(140454142105344) time(2023-02-13 05:03:40.059)      < C_GetFunctionList rv=0x00000000{success} ppFunctionList=0x0x7fbe00000cd8{0x0x7fbe041371e0}
    **ppFunctionList: (552Bytes)
    02 14 00 00 00 00 00 00 - 90 EB 11 04 BE 7F 00 00 - E0 EC 11 04 BE 7F 00 00 - 80 FE 11 04 BE 7F 00 00 
    ...(reducted)
    60 F2 11 04 BE 7F 00 00 - 00 F4 11 04 BE 7F 00 00 - B0 EA 11 04 BE 7F 00 00 - 20 EB 11 04 BE 7F 00 00 
    50 E3 11 04 BE 7F 00 00 


pid(67) tid(140454142105344) time(2023-02-13 05:03:40.060)      > C_Initialize pReserved=73f4dd0p{create=0x(nil) destroy=0x(nil) lock=0x(nil) unlock=0x(nil) flags=0x2}
pid(67) tid(140454142105344) time(2023-02-13 05:03:40.060)      C_Initialize Using PKCS#11 Library /opt/safenet/protecttoolkit7/ptk/lib/libcthsm.so
pid(67) tid(140454142105344) time(2023-02-13 05:03:40.060)      C_Initialize To change use: export ET_PTKC_LOGGER_PKCS11LIB=<otherprovider>
pid(67) tid(140454142105344) time(2023-02-13 05:03:40.060)      C_Initialize GetFunctionTable(/opt/safenet/protecttoolkit7/ptk/lib/libcthsm.so) failed
pid(67) tid(140454142105344) time(2023-02-13 05:03:40.060)      << C_Initialize rv=0x00000005{general error} 

• when called via kubectl -ti exec $pod -- bash and invoke command in pod => success < C_Initialize rv=0x00000000{success}

pid(84) tid(139786740528896) time(2023-02-13 05:04:13.365)      > C_GetFunctionList ppFunctionList=0x0x7f2298000cd8
pid(84) tid(139786740528896) time(2023-02-13 05:04:13.365)      < C_GetFunctionList rv=0x00000000{success} ppFunctionList=0x0x7f2298000cd8{0x0x7f22a04fa1e0}
    **ppFunctionList: (552Bytes)
    02 14 00 00 00 00 00 00 - 90 1B 4E A0 22 7F 00 00 - E0 1C 4E A0 22 7F 00 00 - 80 2E 4E A0 22 7F 00 00 
    ...(reducted)
    60 22 4E A0 22 7F 00 00 - 00 24 4E A0 22 7F 00 00 - B0 1A 4E A0 22 7F 00 00 - 20 1B 4E A0 22 7F 00 00 
    50 13 4E A0 22 7F 00 00 


pid(84) tid(139786740528896) time(2023-02-13 05:04:13.365)      > C_Initialize pReserved=a3036dd0p{create=0x(nil) destroy=0x(nil) lock=0x(nil) unlock=0x(nil) flags=0x2}
pid(84) tid(139786740528896) time(2023-02-13 05:04:13.365)      C_Initialize Using PKCS#11 Library /opt/safenet/protecttoolkit7/ptk/lib/libcthsm.so
pid(84) tid(139786740528896) time(2023-02-13 05:04:13.365)      C_Initialize To change use: export ET_PTKC_LOGGER_PKCS11LIB=<otherprovider>
pid(84) tid(139786740528896) time(2023-02-13 05:04:13.389)      < C_Initialize rv=0x00000000{success} 

[maraino]

@umegaya glad it works on a Thales ProtectServer 3, I don't have access to those.

[umegaya]

@maraino finally I solved the problem. root cause is we need to load /opt/safenet/protecttoolkit7/ptk/setvars.sh to run ptk with real hardware (Protect Server 3+ External). it manipulates $PATH, $LD_LIBRARY_PATH, so it make sense. but unfortunately emulator mode (without real hardware) run perfectly without loading setvars.sh, it takes time to realize it.

now smallstep could run with redundant 2 Protect Server 3+ External and create certificate flawlessly. thanks!