SSL error with a ecc secp521r1 key

[multiSnow]

 $>> cat openssl.cfg 
[req]
default_md=sha512
prompt=no
distinguished_name=req_distinguished_name
[req_distinguished_name]
commonName=name

ecc key generated with secp521r1 not works:

 $>> openssl ecparam -genkey -name secp521r1 -out key.key
 $>> openssl req -x509 -new -key key.key -out crt.crt -days 365 -config openssl.cfg

server side:

 $>> openssl s_server -accept 3443 -cert crt.crt -key key.key
Using auto DH parameters
Using default temp ECDH parameters
ACCEPT
ERROR
140466033342104:error:140270C1:SSL routines:ACCEPT_SR_CLNT_HELLO_C:no shared cipher:ssl_srvr.c:1024:
shutting down SSL
CONNECTION CLOSED
ACCEPT

client side exit with 1:

 $>> openssl s_client -state -nbio -CAfile crt.crt -connect localhost:3443
CONNECTED(00000003)
turning on non blocking io
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:error in SSLv3 read server hello A
write R BLOCK
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server hello A
140259466426008:error:14004410:SSL routines:CONNECT_CR_SRVR_HELLO:sslv3 alert handshake failure:ssl_pkt.c:1205:SSL alert number 40
140259466426008:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:ssl_pkt.c:956:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1500551725
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

but ecc key generated with secp384r1 works well:

 $>> openssl ecparam -genkey -name secp384r1 -out key.key
 $>> openssl req -x509 -new -key key.key -out crt.crt -days 365 -config openssl.cfg

server side:

 $>> openssl s_server -accept 3443 -cert crt.crt -key key.key
Using auto DH parameters
Using default temp ECDH parameters
ACCEPT
-----BEGIN SSL SESSION PARAMETERS-----
MFUCAQECAgMDBALALAQABDCsPTVigu6M8evvQ2Xz6D/pf8bkp3ICDB+P8alSva7Z
4VoWirxjfNGF/k6oknW8iJShBgIEWXCbDaIEAgIcIKQGBAQBAAAA
-----END SSL SESSION PARAMETERS-----
Shared ciphers:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305-OLD:ECDHE-RSA-CHACHA20-POLY1305-OLD:DHE-RSA-CHACHA20-POLY1305-OLD:GOST2012256-GOST89-GOST89:DHE-RSA-CAMELLIA256-SHA256:DHE-RSA-CAMELLIA256-SHA:GOST2001-GOST89-GOST89:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA128-SHA256:DHE-RSA-CAMELLIA128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-SHA256:CAMELLIA128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:EDH-RSA-DES-CBC-SHA:DES-CBC-SHA
CIPHER is ECDHE-ECDSA-AES256-GCM-SHA384
Secure Renegotiation IS supported

client side:

 $>> openssl s_client -state -nbio -CAfile crt.crt -connect localhost:3443
CONNECTED(00000003)
turning on non blocking io
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:error in SSLv3 read server hello A
write R BLOCK
SSL_connect:SSLv3 read server hello A
depth=0 CN = name
verify return:1
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:error in SSLv3 read server session ticket A
read R BLOCK
SSL_connect:SSLv3 read server session ticket A
SSL_connect:SSLv3 read finished A
read R BLOCK
---
Certificate chain
 0 s:/CN=name
   i:/CN=name
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBSjCB0QIJAJGUMcK9SyDuMAoGCCqGSM49BAMEMA8xDTALBgNVBAMMBG5hbWUw
HhcNMTcwNzIwMTE1ODM0WhcNMTgwNzIwMTE1ODM0WjAPMQ0wCwYDVQQDDARuYW1l
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE+vYfKTf/r0f4ba7Iu0LNW7v0OZhJbOvn
falDl/b6GqPf/EpCAMIrdVsEFcl+YQbtVlyuJD1UWpoOLSoz5xSiFDxoky0WZ0TL
iNgvWAR/dIK2f/GTfqIXYjQpweKVh+o2MAoGCCqGSM49BAMEA2gAMGUCMCgaMr/T
ftYVk3IZMw++l9hh+r4ng2QCdisULm1WtdukIlwCsL1ddRbC5WY8NWZL3QIxAMNr
/waH/TY8Y8Hpm8/ECoh+mA8irrv9aZhNVMOMG30MyQ+F3yq1CVRRFyex2y3gAQ==
-----END CERTIFICATE-----
subject=/CN=name
issuer=/CN=name
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 833 bytes and written 342 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 384 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: 038331A97E4DE0C12247AF503C06D5AD1E0C338A5739E304D89E07BF9F00E96B
    Session-ID-ctx: 
    Master-Key: AC3D356282EE8CF1EBEF4365F3E83FE97FC6E4A772020C1F8FF1A952BDAED9E15A168ABC637CD185FE4EA89275BC8894
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 51 80 6e 5a 68 52 fc 84-e3 15 66 8a bc 56 c8 e9   Q.nZhR....f..V..
    0010 - 36 47 be 05 7c 4e 75 f3-6a 04 c2 48 78 31 fc d6   6G..|Nu.j..Hx1..
    0020 - db 18 67 61 3f 6c b6 92-21 ea f4 75 51 d0 9e 4b   ..ga?l..!..uQ..K
    0030 - f9 0c 61 f4 e7 5e 99 ca-fe 6c 0d ae 6a 9a 57 12   ..a..^...l..j.W.
    0040 - 3e c1 6b 4c 44 87 20 88-a8 5f 85 44 43 98 e9 11   >.kLD. .._.DC...
    0050 - 4e 4a 3d ad d5 55 55 08-35 82 5f 10 c6 39 59 15   NJ=..UU.5._..9Y.
    0060 - eb 7c 87 05 89 d1 10 53-5a c7 b1 0c 9b eb c1 3b   .|.....SZ......;
    0070 - 6d c6 c9 4d 00 e4 c4 b5-13 5f 73 c7 e0 79 bc f8   m..M....._s..y..
    0080 - e0 e5 09 ec 3d 8c b5 13-6c d5 9b 4a c7 b9 5c 01   ....=...l..J..\.
    0090 - fd 25 56 71 a0 c2 aa fa-19 d5 f4 1d bb f3 c7 64   .%Vq...........d

    Start Time: 1500551949
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

libressl-portable git version, configure option:

--enable-shared
--enable-static
--enable-nc
--enable-asm
--with-pic

[4a6f656c]

This is due to the fact that secp521r1 is not one of the default signature algorithms or SIGALGS, which causes the TLS session negotiation to fail (the current defaults are X25519, secp256r1 and secp384r1). Using the -named_curve option on s_server and the -groups option on s_client will allow this key to work:

$ openssl s_server -accept 3443 -cert crt.crt -key key.key -named_curve secp521r1
openssl s_client -state -nbio -CAfile crt.crt -connect localhost:3443 -groups secp521r1

(thanks for the detailed report)

[multiSnow]

Yes the options does work with openssl s_server and s_client. So what about other programs linked with libssl.so that also failed with secp521r1 ecc key such as curl, python, ruby, etc...?
And, since secp521r1 is written out of default, is there some way to enable it optionally? or I should simply drop it and treat it as insecure and never trust it anymore?

[4a6f656c]

EC curves can be enabled/disabled via the SSL_set1_groups_list(3) (or SSL_set1_curves_list(3)) API - how this is exposed to the user depends on the application or programming language in question.

I'm not aware of any specific issues with secp521r1, however it is unlikely to provide significant advantages over secp384r1 and secp256r1. In many cases X25519 is going to be preferable (see https://safecurves.cr.yp.to/ for some further reading). Additionally, BoringSSL does not enable secp521r1 by default and Chrome does not support it...