TTY_get() in crypto/ui/ui_openssl.c open_console() can also return errno 6 (ENXIO, Solaris) and 5 (EIO, Linux)

[rainerjung]

I am running multiple iterations of the Apache httpd test suite using shell scripts. When I log out while the scripts are running in the background, invocations of "openssl pkcs12" binary throw errors:

Solaris:

openssl pkcs12 -export -in certs/ca.crt -inkey keys/ca.pem -out export/ca.p12 -passin pass:httpd -passout pass:httpd
User interface error
1:error:2807206C:UI routines:open_console:unknown ttyget errno value:crypto/ui/ui_openssl.c:451:errno=6
1:error:2807106B:UI routines:UI_process:processing error:crypto/ui/ui_lib.c:493:while opening session
unable to load private key
1:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:536:
1:error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:63:
1:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:crypto/pkcs12/p12_decr.c:94:
1:error:0907B00D:PEM routines:PEM_read_bio_PrivateKey:ASN1 lib:crypto/pem/pem_pkey.c:87:

Linux:

openssl pkcs12 -export -in certs/ca.crt -inkey keys/ca.pem -out export/ca.p12 -passin pass:httpd -passout pass:httpd
User interface error
140287533819648:error:2807206C:UI routines:open_console:unknown ttyget errno value:crypto/ui/ui_openssl.c:451:errno=5
140287533819648:error:2807106B:UI routines:UI_process:processing error:crypto/ui/ui_lib.c:493:while opening session
unable to load private key
140287533819648:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:536:
140287533819648:error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:63:
140287533819648:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:crypto/pkcs12/p12_decr.c:94:
140287533819648:error:0907B00D:PEM routines:PEM_read_bio_PrivateKey:ASN1 lib:crypto/pem/pem_pkey.c:87:

IMHO that means, errno 6 (ENXIO) for Solaris and 5 (EIO) for Linux should be handled just like the errnos already handled by the code in crypto/ui/ui_openssl.c open_console() (ENOTTY, EINVAL, ENODEV) by setting "is_a_tty = 0;" and continuing instead of returning due to unknown error.

The problem occurs in 1.1.0 and 1.1.1 (and probably master).

Patch for master:

index 3ccd8a720e..e54bdd2e4e 100644
--- a/crypto/ui/ui_openssl.c
+++ b/crypto/ui/ui_openssl.c
@@ -415,6 +415,24 @@ static int open_console(UI *ui)
             is_a_tty = 0;
         else
 #  endif
+#  ifdef ENXIO
+            /*
+             * Solaris can return ENXIO.
+             * This should be ok
+             */
+        if (errno == ENXIO)
+            is_a_tty = 0;
+        else
+#  endif
+#  ifdef EIO
+            /*
+             * Linux can return ENXIO.
+             * This should be ok
+             */
+        if (errno == EIO)
+            is_a_tty = 0;
+        else
+#  endif
 #  ifdef ENODEV
             /*
              * MacOS X returns ENODEV (Operation not supported by device),

Regards,

Rainer

[levitte]

I suppose this occurs in 1.0.2 as well, btw...