sec_error_bad_signature randomly starts after a client mucks up ssl #575

Closed
Borkason opened this Issue Mar 24, 2013 · 78 comments

1 participant

@Borkason
Cherokee Project member

Original author: kallist...@gmail.com (October 06, 2009 23:41:40)

What steps will reproduce the problem?
1. run a ssl domain though cherokee
2. access ssl domain with chromium on linux
3. chrome will core dump eventually with ssl connections (related to cert?
this may be a problem on chrome's side)
4. Cherokee will give sec_error_bad_signature to any browser which accesses
SSL domain until cherokee is restarted. (tested on various platforms and
different locations)

What is the expected output? What do you see instead?
A 3rd party should never be able to bring down a ssl site remotely.

What version of the product are you using? On what operating system?
Cherokee 0.99.24, ubuntu 9.04 32-bit

Please provide any additional information below.
I have not yet reset the web server, feel free to access the ssl page here
and see the error:
https://ssl.unixzen.com

Original issue: http://code.google.com/p/cherokee/issues/detail?id=594

@Borkason
Cherokee Project member

From ste...@konink.de on October 07, 2009 10:21:14
I believe there is a problem if you say so. But how can I crash Chromium? I'm running
4.0.22.5 (27967)

If you want to help out it might be a good thing to have a gdb session available in
case we are unable to crash Chromium in the first place. If you say on the other
hand... I will be able to crash any Cherokee server I can launch a Cherokee+SSL
myself and let you crash it.

@Borkason
Cherokee Project member

From ste...@konink.de on October 07, 2009 10:21:55
The only errors I get from Chromium on the commandline are:

[30813:30836:104525399282:ERROR:/b/slave/chromium-rel-linux-64/build/src/net/base/x509_certificate_nss.cc(530)]
CERT_PKIXVerifyCert for ssl.unixzen.com failed err=-8179

@Borkason
Cherokee Project member

From kallist...@gmail.com on October 07, 2009 12:51:35
here is what I get:

"
[6064:6064:1035505425810:ERROR:/build/buildd/chromium-browser-4.0.221.7~svn20091006r28103/build-tree/src/chrome/browser/first_run_gtk.cc(21)]
Not implemented reached in static bool FirstRun::ProcessMasterPreferences(const
FilePath&, const FilePath&, std::vector std::char_traits, std::allocator >,
std::allocator,
std::allocator > > >, int, bool*)
[6064:6093:1035521152172:ERROR:/build/buildd/chromium-browser-4.0.221.7~svn20091006r28103/build-tree/src/net/base/x509_certificate_nss.cc(530)]
CERT_PKIXVerifyCert for ssl.unixzen.com failed err=-8179
[6064:6091:1035523103478:ERROR:/build/buildd/chromium-browser-4.0.221.7~svn20091006r28103/build-tree/src/net/base/x509_certificate_nss.cc(530)]
CERT_PKIXVerifyCert for ssl.unixzen.com failed err=-8179
[6064:6067:1035523591028:ERROR:/build/buildd/chromium-browser-4.0.221.7~svn20091006r28103/build-tree/src/net/socket/ssl_client_socket_nss.cc(728)]
handshake failed; NSS error code -8182, net_error -207
Segmentation fault
"

This issue seems hard to reproduce, I know it has happened to me several times in the
past... and I see another post refering to the same issue:
http://forums.digitalpoint.com/showthread.php?t=1436685

I think there's a situation which breaks cherokee SSL until it's restarted. I am
working on getting more info on how to reproduce it. I unfortunately did not have
logging enabled last time this happened. Logging for my ssl subdomain has been enabled.

@Borkason
Cherokee Project member

From ste...@konink.de on October 07, 2009 13:02:01
If you have a way to kill my server I can do something for you ;) But now I cannot
crash Chromium nor I can kill Cherokee. I know we have fixed another issue in
Cherokee related to SSL but I think that is unrelated, and actually returned a
segmentationfault.

@Borkason
Cherokee Project member

From ste...@konink.de on December 19, 2009 12:41:36
We need a way to reproduce it... anyone that can. Please step up.

@Borkason
Cherokee Project member

From MellowCe...@gmail.com on December 30, 2009 06:40:40
I've had this problem. I thought it was a Chromium problem so I commented on this bug
in Chromium. http://code.google.com/p/chromium/issues/detail?id=24064#c8. I still
think Chromium has its issues with this, but I can see there's something up with
Cherokee too.

Here's the output from Chromium when it crashed loading
https://cellofellow.homelinux.net/citadel/.

$ chromium-browser
[4069:4098:126982189095:ERROR:net/base/x509_certificate_nss.cc(562)]
CERT_PKIXVerifyCert for cellofellow.homelinux.net failed err=-8172
[4069:4079:126987795621:ERROR:net/socket/ssl_client_socket_nss.cc(1037)] handshake
failed; NSS error code -8182, net_error -207
Segmentation fault

It showed this after showing the obligatory warning about a self-signed SSL certificate.

So, I dunno if this counts as "reproducing" the bug but here it is.

@Borkason
Cherokee Project member

From ste...@konink.de on December 30, 2009 07:01:34
I see a Chromium crash on the url you provide!

But when I take my own self signed stuff and i browse to my 127.0.0.1 I don't see
anything. The only error I get:

[1033:1070:54733470252:ERROR:net/base/x509_certificate_nss.cc(558)]
CERT_PKIXVerifyCert for localhost failed err=-8172
[1033:1071:54737181168:ERROR:net/base/x509_certificate_nss.cc(558)]
CERT_PKIXVerifyCert for localhost failed err=-8172
[1033:1054:54737182788:ERROR:net/base/x509_certificate_nss.cc(558)]
CERT_PKIXVerifyCert for localhost failed err=-8172

Hostname seems not to be the issue.

Maybe it is your cert, maybe something else, from here I cannot ask anymore than
'more info' please and/or your certs.

I have tested this with .23 and .38. Webserver works on my side as a charm.

@Borkason
Cherokee Project member

From MellowCe...@gmail.com on December 30, 2009 19:04:31
I'm using Cherokee .38 on Karmic from the PPA. I've attached my .crt file, hope it
helps.

@Borkason
Cherokee Project member

From ste...@konink.de on December 30, 2009 19:44:46
Again works for me... no chrome crash.

@Borkason
Cherokee Project member

From kallist...@gmail.com on January 23, 2010 15:32:08
hmm, it happened again for me on cherokee 0.99.35. I attached strace to the cherokee
worker thread hit refresh generating "sec_error_bad_signature" error and pulled the
following trace.

strace -o cherokee-strace -vvFfp 6423

You can view the ssl error on the page here:
https://ssl.unixzen.com/mail

I will leave it up and not restart it for a few days.

@Borkason
Cherokee Project member

From kallist...@gmail.com on January 23, 2010 15:34:56
fyi... i am 99.54.163.51.

Also, the crt has been working for months aok, the error causes chrome (linux) to
crash and causes firefox to throw "sec_error_bad_signature"

@Borkason
Cherokee Project member

From kallist...@gmail.com on January 23, 2010 15:37:07
root@discord:~# openssl s_client -connect ssl.unixzen.com:443
CONNECTED(00000003)
depth=1 /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class
1 Primary Intermediate Server CA
verify error:num=20:unable to get local issuer certificate

verify return:0

Certificate chain
0 s:/description=118473-cafEGt2WRrXv3kk0/C=US/O=Persona Not Validated/OU=StartCom
Free Certificate Member/CN=ssl.unixzen.com/emailAddress=postmaster@unixzen.com
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1
Primary Intermediate Server CA
1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1
Primary Intermediate Server CA
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom

Certification Authority

Server certificate
-----BEGIN CERTIFICATE-----
MIIHwTCCBqmgAwIBAgIDAPdbMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYDVQQGEwJJ
TDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0
YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3Mg
MSBQcmltYXJ5IEludGVybWVkaWF0ZSBTZXJ2ZXIgQ0EwHhcNMDkxMjIzMTc1ODMw
WhcNMTAxMjI0MDcxOTQzWjCBuzEgMB4GA1UEDRMXMTE4NDczLWNhZkVHdDJXUnJY
djNrazAxCzAJBgNVBAYTAlVTMR4wHAYDVQQKExVQZXJzb25hIE5vdCBWYWxpZGF0
ZWQxKTAnBgNVBAsTIFN0YXJ0Q29tIEZyZWUgQ2VydGlmaWNhdGUgTWVtYmVyMRgw
FgYDVQQDEw9zc2wudW5peHplbi5jb20xJTAjBgkqhkiG9w0BCQEWFnBvc3RtYXN0
ZXJAdW5peHplbi5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCi
fmlKFrJd2vEw8WLqsQKKS8K8VIuMe0xfDH9AWBY9GnW/xtF2F+Cmo7SGZM/Gyj4o
ZoIdsj1l6MCSw9aE9jkbUOH6WrmJ2XDV+rT2Gho8EXdEsJ9BtcVW7VBUZrSLvqmk
vVWouPVUxAnvSSJcYmIyCHRifRvXWF+rxAqpiA28le45g9NkelFq9bD/hPTGUNVL
2Uj2jovyK/TMq+kspFGWas1RXh9aIWXD8/AzdMeDgUBVYKGcdS19dCeMMv/ER6b6
BBHT0yQpzmKvRsoE7aU53D4hgocAX8CVQkW42+8ZXkTs8PzlmpAXHLpxKfxmJIqt
DqikY9TVI67+DrmPO+PPKtqyaSHnGxWyE6KudGKmwgUrBrNQyKNyeGqUZWeAIuj1
mCfo2ZnXgUNGaiAqIRI02SH6UxuydGxi2Gf2B96/L90PjAHTaczSoWvbsvKq3wno
rglLGx+cB/65C+hg/zl0rjy+U9w6GYdQpb9iuK3Rz0ALJKfLSyW3w+2/sXSpBWBr
7CAUXtZ1rpeZFaYM6H/RvyomxHEwfJkLaEYFk4Xa1Xdb8I8lOjoLkJG+ApywVuvk
8zWZCrnn82qXOHHgiI+RdHBbhBzthfomJk3Jp6HVUwNhF7icDTmufyPWWmo65zYv
gEwPAkRffH00jjnCkhu68QISVyLSaRxnR08hqze4jwIDAQABo4IC+TCCAvUwCQYD
VR0TBAIwADALBgNVHQ8EBAMCA6gwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0O
BBYEFIv1gyq+A1e059TW/pErPtwrXPSAMB8GA1UdIwQYMBaAFOtCNNCYsKuf9Btr
CPfMZC7vDixFMCcGA1UdEQQgMB6CD3NzbC51bml4emVuLmNvbYILdW5peHplbi5j
b20wggFCBgNVHSAEggE5MIIBNTCCATEGCysGAQQBgbU3AQIBMIIBIDAuBggrBgEF
BQcCARYiaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjA0BggrBgEF
BQcCARYoaHR0cDovL3d3dy5zdGFydHNzbC5jb20vaW50ZXJtZWRpYXRlLnBkZjCB
twYIKwYBBQUHAgIwgaowFBYNU3RhcnRDb20gTHRkLjADAgEBGoGRTGltaXRlZCBM
aWFiaWxpdHksIHNlZSBzZWN0aW9uICpMZWdhbCBMaW1pdGF0aW9ucyogb2YgdGhl
IFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFBvbGljeSBhdmFpbGFi
bGUgYXQgaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjBhBgNVHR8E
WjBYMCqgKKAmhiRodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9jcnQxLWNybC5jcmww
KqAooCaGJGh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydDEtY3JsLmNybDCBjgYI
KwYBBQUHAQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wu
Y29tL3N1Yi9jbGFzczEvc2VydmVyL2NhMEIGCCsGAQUFBzAChjZodHRwOi8vd3d3
LnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIuY2xhc3MxLnNlcnZlci5jYS5jcnQwIwYD
VR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMA0GCSqGSIb3DQEBBQUA
A4IBAQCo5JZyIfjJ0gyMkLkZAYAhqdlbhsDqBz3p0iPQsDCjRXGldlM8SMQ5w8l0
1vI8nc5zn8yi+adyUb87+IkJlh411Rxk+cRyv6W9aqCYxxr3iwX2ydVnj19FXb9d
yvNoA1AamesSohxdYP/+CEZ2DodepMtwPhOdEG6lrKowqg0OLJssDSG9nOPzj8/2
XiMrklY1bnkpL3UvumiBo8CtJokdyqWu1tdcChxHBwE3fThB9pBLR8/jeTpY+k9o
SIljxXNsWAhtx0AJ8pTMKrPyZJOc48KFSGO5Fzfgln84rBRutb+Ldbp8g7R4ybSN
8lbp3VPK2UO4zMZguPfgBBY9jIoe
-----END CERTIFICATE-----
subject=/description=118473-cafEGt2WRrXv3kk0/C=US/O=Persona Not Validated/OU=StartCom
Free Certificate Member/CN=ssl.unixzen.com/emailAddress=postmaster@unixzen.com
issuer=/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class

1 Primary Intermediate Server CA

No client certificate CA names sent

SSL handshake has read 4958 bytes and written 316 bytes

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 20FC2785682F696A26BFA3DD1A68C7314C3FE48C421529764D7EEA2C55E9A9E1
Session-ID-ctx:
Master-Key:
C5FEF6698870F39EA61F71EE07D654FA75C32F358FD6A3BF2470AB934E01546C4825923258A42C2E614A28C46C8FA982
Key-Arg : None
Start Time: 1264260957
Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)

read:errno=0

@Borkason
Cherokee Project member

From kallist...@gmail.com on January 24, 2010 18:02:53
eh.. i finally had to restart ssl.unixzen.com, i had work to do.

After a restart of cherokee the error above goes away..

here is the openssl output once it was working again..

#​ openssl s_client -connect ssl.unixzen.com:443
CONNECTED(00000003)
depth=1 /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class
1 Primary Intermediate Server CA
verify error:num=20:unable to get local issuer certificate

verify return:0

Certificate chain
0 s:/description=118473-cafEGt2WRrXv3kk0/C=US/O=Persona Not Validated/OU=StartCom
Free Certificate Member/CN=ssl.unixzen.com/emailAddress=postmaster@unixzen.com
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1
Primary Intermediate Server CA
1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1
Primary Intermediate Server CA
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom

Certification Authority

Server certificate
-----BEGIN CERTIFICATE-----
MIIHwTCCBqmgAwIBAgIDAPdbMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYDVQQGEwJJ
TDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0
YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3Mg
MSBQcmltYXJ5IEludGVybWVkaWF0ZSBTZXJ2ZXIgQ0EwHhcNMDkxMjIzMTc1ODMw
WhcNMTAxMjI0MDcxOTQzWjCBuzEgMB4GA1UEDRMXMTE4NDczLWNhZkVHdDJXUnJY
djNrazAxCzAJBgNVBAYTAlVTMR4wHAYDVQQKExVQZXJzb25hIE5vdCBWYWxpZGF0
ZWQxKTAnBgNVBAsTIFN0YXJ0Q29tIEZyZWUgQ2VydGlmaWNhdGUgTWVtYmVyMRgw
FgYDVQQDEw9zc2wudW5peHplbi5jb20xJTAjBgkqhkiG9w0BCQEWFnBvc3RtYXN0
ZXJAdW5peHplbi5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCi
fmlKFrJd2vEw8WLqsQKKS8K8VIuMe0xfDH9AWBY9GnW/xtF2F+Cmo7SGZM/Gyj4o
ZoIdsj1l6MCSw9aE9jkbUOH6WrmJ2XDV+rT2Gho8EXdEsJ9BtcVW7VBUZrSLvqmk
vVWouPVUxAnvSSJcYmIyCHRifRvXWF+rxAqpiA28le45g9NkelFq9bD/hPTGUNVL
2Uj2jovyK/TMq+kspFGWas1RXh9aIWXD8/AzdMeDgUBVYKGcdS19dCeMMv/ER6b6
BBHT0yQpzmKvRsoE7aU53D4hgocAX8CVQkW42+8ZXkTs8PzlmpAXHLpxKfxmJIqt
DqikY9TVI67+DrmPO+PPKtqyaSHnGxWyE6KudGKmwgUrBrNQyKNyeGqUZWeAIuj1
mCfo2ZnXgUNGaiAqIRI02SH6UxuydGxi2Gf2B96/L90PjAHTaczSoWvbsvKq3wno
rglLGx+cB/65C+hg/zl0rjy+U9w6GYdQpb9iuK3Rz0ALJKfLSyW3w+2/sXSpBWBr
7CAUXtZ1rpeZFaYM6H/RvyomxHEwfJkLaEYFk4Xa1Xdb8I8lOjoLkJG+ApywVuvk
8zWZCrnn82qXOHHgiI+RdHBbhBzthfomJk3Jp6HVUwNhF7icDTmufyPWWmo65zYv
gEwPAkRffH00jjnCkhu68QISVyLSaRxnR08hqze4jwIDAQABo4IC+TCCAvUwCQYD
VR0TBAIwADALBgNVHQ8EBAMCA6gwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0O
BBYEFIv1gyq+A1e059TW/pErPtwrXPSAMB8GA1UdIwQYMBaAFOtCNNCYsKuf9Btr
CPfMZC7vDixFMCcGA1UdEQQgMB6CD3NzbC51bml4emVuLmNvbYILdW5peHplbi5j
b20wggFCBgNVHSAEggE5MIIBNTCCATEGCysGAQQBgbU3AQIBMIIBIDAuBggrBgEF
BQcCARYiaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjA0BggrBgEF
BQcCARYoaHR0cDovL3d3dy5zdGFydHNzbC5jb20vaW50ZXJtZWRpYXRlLnBkZjCB
twYIKwYBBQUHAgIwgaowFBYNU3RhcnRDb20gTHRkLjADAgEBGoGRTGltaXRlZCBM
aWFiaWxpdHksIHNlZSBzZWN0aW9uICpMZWdhbCBMaW1pdGF0aW9ucyogb2YgdGhl
IFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFBvbGljeSBhdmFpbGFi
bGUgYXQgaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjBhBgNVHR8E
WjBYMCqgKKAmhiRodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9jcnQxLWNybC5jcmww
KqAooCaGJGh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydDEtY3JsLmNybDCBjgYI
KwYBBQUHAQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wu
Y29tL3N1Yi9jbGFzczEvc2VydmVyL2NhMEIGCCsGAQUFBzAChjZodHRwOi8vd3d3
LnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIuY2xhc3MxLnNlcnZlci5jYS5jcnQwIwYD
VR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMA0GCSqGSIb3DQEBBQUA
A4IBAQCo5JZyIfjJ0gyMkLkZAYAhqdlbhsDqBz3p0iPQsDCjRXGldlM8SMQ5w8l0
1vI8nc5zn8yi+adyUb87+IkJlh411Rxk+cRyv6W9aqCYxxr3iwX2ydVnj19FXb9d
yvNoA1AamesSohxdYP/+CEZ2DodepMtwPhOdEG6lrKowqg0OLJssDSG9nOPzj8/2
XiMrklY1bnkpL3UvumiBo8CtJokdyqWu1tdcChxHBwE3fThB9pBLR8/jeTpY+k9o
SIljxXNsWAhtx0AJ8pTMKrPyZJOc48KFSGO5Fzfgln84rBRutb+Ldbp8g7R4ybSN
8lbp3VPK2UO4zMZguPfgBBY9jIoe
-----END CERTIFICATE-----
subject=/description=118473-cafEGt2WRrXv3kk0/C=US/O=Persona Not Validated/OU=StartCom
Free Certificate Member/CN=ssl.unixzen.com/emailAddress=postmaster@unixzen.com
issuer=/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class

1 Primary Intermediate Server CA

No client certificate CA names sent

SSL handshake has read 4958 bytes and written 316 bytes

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: D0C18E18BD63721B5139E6AE0D1F01B43B248C28431EFDC9B144544E24692054
Session-ID-ctx:
Master-Key:
7A75880D581F1E7D5488C244D1E4023C6DA96F025DA249A42E6960CA1DD8E7FAD2AAECCA8F15CA4F68A5DFD634036321
Key-Arg : None
Start Time: 1264356127
Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)

@Borkason
Cherokee Project member

From kallist...@gmail.com on February 05, 2010 21:18:05
upgraded to 99.42 yesterday and started receiving this again this afternoon. Really
annoying it is having to restart the web server every few days.

@Borkason
Cherokee Project member

From ste...@konink.de on February 05, 2010 21:20:05
What is the version of your openssl?

@Borkason
Cherokee Project member

From kallist...@gmail.com on February 05, 2010 21:20:13
here is the openssl command run from a remote server to the broken ssl webpage...

$ openssl s_client -connect ssl.unixzen.com:443
CONNECTED(00000003)
depth=1 /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class
1 Primary Intermediate Server CA
verify error:num=20:unable to get local issuer certificate

verify return:0

Certificate chain
0 s:/description=118473-cafEGt2WRrXv3kk0/C=US/O=Persona Not Validated/OU=StartCom
Free Certificate Member/CN=ssl.unixzen.com/emailAddress=postmaster@unixzen.com
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1
Primary Intermediate Server CA
1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1
Primary Intermediate Server CA
i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom

Certification Authority

Server certificate
-----BEGIN CERTIFICATE-----
MIIHwTCCBqmgAwIBAgIDAPdbMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYDVQQGEwJJ
TDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0
YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3Mg
MSBQcmltYXJ5IEludGVybWVkaWF0ZSBTZXJ2ZXIgQ0EwHhcNMDkxMjIzMTc1ODMw
WhcNMTAxMjI0MDcxOTQzWjCBuzEgMB4GA1UEDRMXMTE4NDczLWNhZkVHdDJXUnJY
djNrazAxCzAJBgNVBAYTAlVTMR4wHAYDVQQKExVQZXJzb25hIE5vdCBWYWxpZGF0
ZWQxKTAnBgNVBAsTIFN0YXJ0Q29tIEZyZWUgQ2VydGlmaWNhdGUgTWVtYmVyMRgw
FgYDVQQDEw9zc2wudW5peHplbi5jb20xJTAjBgkqhkiG9w0BCQEWFnBvc3RtYXN0
ZXJAdW5peHplbi5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCi
fmlKFrJd2vEw8WLqsQKKS8K8VIuMe0xfDH9AWBY9GnW/xtF2F+Cmo7SGZM/Gyj4o
ZoIdsj1l6MCSw9aE9jkbUOH6WrmJ2XDV+rT2Gho8EXdEsJ9BtcVW7VBUZrSLvqmk
vVWouPVUxAnvSSJcYmIyCHRifRvXWF+rxAqpiA28le45g9NkelFq9bD/hPTGUNVL
2Uj2jovyK/TMq+kspFGWas1RXh9aIWXD8/AzdMeDgUBVYKGcdS19dCeMMv/ER6b6
BBHT0yQpzmKvRsoE7aU53D4hgocAX8CVQkW42+8ZXkTs8PzlmpAXHLpxKfxmJIqt
DqikY9TVI67+DrmPO+PPKtqyaSHnGxWyE6KudGKmwgUrBrNQyKNyeGqUZWeAIuj1
mCfo2ZnXgUNGaiAqIRI02SH6UxuydGxi2Gf2B96/L90PjAHTaczSoWvbsvKq3wno
rglLGx+cB/65C+hg/zl0rjy+U9w6GYdQpb9iuK3Rz0ALJKfLSyW3w+2/sXSpBWBr
7CAUXtZ1rpeZFaYM6H/RvyomxHEwfJkLaEYFk4Xa1Xdb8I8lOjoLkJG+ApywVuvk
8zWZCrnn82qXOHHgiI+RdHBbhBzthfomJk3Jp6HVUwNhF7icDTmufyPWWmo65zYv
gEwPAkRffH00jjnCkhu68QISVyLSaRxnR08hqze4jwIDAQABo4IC+TCCAvUwCQYD
VR0TBAIwADALBgNVHQ8EBAMCA6gwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHQYDVR0O
BBYEFIv1gyq+A1e059TW/pErPtwrXPSAMB8GA1UdIwQYMBaAFOtCNNCYsKuf9Btr
CPfMZC7vDixFMCcGA1UdEQQgMB6CD3NzbC51bml4emVuLmNvbYILdW5peHplbi5j
b20wggFCBgNVHSAEggE5MIIBNTCCATEGCysGAQQBgbU3AQIBMIIBIDAuBggrBgEF
BQcCARYiaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjA0BggrBgEF
BQcCARYoaHR0cDovL3d3dy5zdGFydHNzbC5jb20vaW50ZXJtZWRpYXRlLnBkZjCB
twYIKwYBBQUHAgIwgaowFBYNU3RhcnRDb20gTHRkLjADAgEBGoGRTGltaXRlZCBM
aWFiaWxpdHksIHNlZSBzZWN0aW9uICpMZWdhbCBMaW1pdGF0aW9ucyogb2YgdGhl
IFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFBvbGljeSBhdmFpbGFi
bGUgYXQgaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjBhBgNVHR8E
WjBYMCqgKKAmhiRodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9jcnQxLWNybC5jcmww
KqAooCaGJGh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydDEtY3JsLmNybDCBjgYI
KwYBBQUHAQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wu
Y29tL3N1Yi9jbGFzczEvc2VydmVyL2NhMEIGCCsGAQUFBzAChjZodHRwOi8vd3d3
LnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIuY2xhc3MxLnNlcnZlci5jYS5jcnQwIwYD
VR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMA0GCSqGSIb3DQEBBQUA
A4IBAQCo5JZyIfjJ0gyMkLkZAYAhqdlbhsDqBz3p0iPQsDCjRXGldlM8SMQ5w8l0
1vI8nc5zn8yi+adyUb87+IkJlh411Rxk+cRyv6W9aqCYxxr3iwX2ydVnj19FXb9d
yvNoA1AamesSohxdYP/+CEZ2DodepMtwPhOdEG6lrKowqg0OLJssDSG9nOPzj8/2
XiMrklY1bnkpL3UvumiBo8CtJokdyqWu1tdcChxHBwE3fThB9pBLR8/jeTpY+k9o
SIljxXNsWAhtx0AJ8pTMKrPyZJOc48KFSGO5Fzfgln84rBRutb+Ldbp8g7R4ybSN
8lbp3VPK2UO4zMZguPfgBBY9jIoe
-----END CERTIFICATE-----
subject=/description=118473-cafEGt2WRrXv3kk0/C=US/O=Persona Not Validated/OU=StartCom
Free Certificate Member/CN=ssl.unixzen.com/emailAddress=postmaster@unixzen.com
issuer=/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class

1 Primary Intermediate Server CA

No client certificate CA names sent

SSL handshake has read 4958 bytes and written 316 bytes

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 8022C39DF19CE1729173C1A748A347EADEFCCD9B5AE04B6F5E8D1357950197BA
Session-ID-ctx:
Master-Key:
E7FFC9923057E1F7F8196E06A8C8C12488349F0419EBFA6088A2888BF130445C8FE9BB39FAE2F26F8DCF563CB0A4D07F
Key-Arg : None
Start Time: 1265404753
Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)

closed

@Borkason
Cherokee Project member

From skar...@gmail.com on February 06, 2010 09:34:52
@kallisti05: Just to clarify... Does it work for a few days and then crash? With all
versions you tried or are there any version that works fine?

@Borkason
Cherokee Project member

From kallist...@gmail.com on February 09, 2010 22:15:29
seems like i've had the problem consistently with all different versions of cherokee.
The timeframe to issue start is always different and seems to not be dependent on
certificate (i've seen it with godaddy certs and self-signed certs.

here is my ssl version:

#​ openssl version
OpenSSL 0.9.8g 19 Oct 2007

@Borkason
Cherokee Project member

From ste...@konink.de on February 09, 2010 22:19:13
Update that openssl version...

@Borkason
Cherokee Project member

From alobbs on February 10, 2010 06:47:57
Stefan, are you certain the OpenSSL version is the source of the problem? :-?

@Borkason
Cherokee Project member

From ste...@konink.de on February 10, 2010 11:11:42
Actually we have had some oddness with another old version...

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on February 15, 2010 15:47:06
I've got this same problem. Restarting the webserver corrects it temporarily.
Ubuntu 9.10 with all updates. Running cherokee, updated from launchpad PPA
(Currently 0.99.42). Openssl 0.9.8g. Openssl 0.9.8g is the ubuntu karmic supported
ver. Are we sure it's the problem? I'd hate to step outside the "current supported"
box on such an important package.

Why would the web server work for a period (2-3 days) before requiring a restart?

I have this issue on 3 servers (physical servers)

@Borkason
Cherokee Project member

From ste...@konink.de on February 15, 2010 18:23:21
Obviously if nobody test it with the latest OpenSSL, nobody will get an answer.

@Borkason
Cherokee Project member

From alobbs on February 15, 2010 18:37:47
Since there are other servers that do not suffer this issue, I'd suppose the problem is in
Cherokee, actually.

@Borkason
Cherokee Project member

From ste...@konink.de on February 15, 2010 18:42:38
This bug was here before... and it was solved with an openssl update. 0.9.8g is more
than 2 years old...

@Borkason
Cherokee Project member

From ste...@konink.de on February 15, 2010 18:44:58
And please check the changelog: http://www.openssl.org/news/changelog.html

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on February 15, 2010 21:19:16
That changelog is huge... Is that your point?

Why does restarting cherokee solve the problem temporarily?

I wonder if- even if it is an openssl issue, there could be some sort of Cherokee
workaround patch so that I don't need to manually keep openssl updated. That's the
point of my apt and distro's repos. Ubuntu is pretty famous. People running into
these issues may shy away from Cherokee when the problem doesn't exist in other web
servers- such as Apache.

-David

@Borkason
Cherokee Project member

From ste...@konink.de on February 15, 2010 21:42:05
The point is that there is quite a triggering entry one version after the one you are
running. And I find it not really acceptable your distribution is shipping a two year
old library for core crypto activities.

So it is trivial to prove me wrong, make it fail with a recent openssl...

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on February 16, 2010 00:12:45
I don't need to prove you wrong. Cherokee is incompatible with Ubuntu Server 9.10's
(The most recent release, not even a LTS)'s maintained packages.

That's too bad. If you don't care, I guess I won't either.

@Borkason
Cherokee Project member

From skar...@gmail.com on February 16, 2010 07:40:43
I also think that the problem is in Cherokee.

@Borkason
Cherokee Project member

From thebigsl...@gmail.com on March 22, 2010 21:13:43
Hello,

I've linked cherokee against openssl 0.9.8M and I'm having the exact same problem.
So It's not openssl.

@Borkason
Cherokee Project member

From ste...@konink.de on March 22, 2010 21:20:46
In that case please get us a binary distro + -O0 -g + configuration + certs. And tell
us how to get it crashed.

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on March 27, 2010 18:22:29
I am now running Arch Linux with Cherokee Web Server 0.99.43, OpenSSL 0.9.8m 25 Feb
2010. I generated new certs since my last Ubuntu issues, and It took a longer this
time for the error to creep back, but there it is... (Error code:
sec_error_bad_signature). Rebooting again, I'll post back with developments.

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on March 27, 2010 18:43:23
I just updated to OpenSSL 0.9.8n and Cherokee0.9.44 and rebooted. I'm going to
generate new certificates... I'll post back if it dies again.

This is a very low traffic server at my house- I'm the only one who hits it.

@Borkason
Cherokee Project member

From oberroc...@gmail.com on April 29, 2010 10:41:53
Any news about OpenSSL 0.9.8n? Is the error gone?

I'm facing this issue since I upgraded to Cherokee 0.99.47 - can't remember any
incident before. My system is Ubuntu hardy 8.04, OpenSSL 0.9.8g.

My site is only internal, too, but when it is used, the issue comes up every 15 minutes.

@Borkason
Cherokee Project member

From lnu...@gmail.com on April 29, 2010 17:24:12
Testing for more than an hour a reload every 10 seconds to an https hardy server with
cherokee 0.99.48 with a 5 seconds modified page and no problems with Firefox 3.6.3

testing with chrome 5.0.342.9 beta on ubuntu be back with results in an hour or so

@Borkason
Cherokee Project member

From lnu...@gmail.com on April 29, 2010 19:07:28
Tested for 1 hour the same reload every 10 seconds on chrome and no problems found

@Borkason
Cherokee Project member

From oberroc...@gmail.com on April 29, 2010 20:11:11
Thanks for your testing!

The issue though still seems pretty common...
http://lists.octality.com/pipermail/cherokee/2010-April/thread.html

My setup is rather complicated, maybe this is related. I have 3 virtual servers
configured: one for a Trac instance (SCGI), one for PHP via FCGI, and another (the
one that causes the problem) is running a Django app via SCGI - maybe this is
related?

All settings are defaults as configured by the Wizards (keep alive is set to allow).
I did a fresh configuration (only cherokee-admin) after installing 0.99.47. The Trac
virtual server is used much more than the Django one but never seemed to trigger the
problem...

I just installed 0.99.48, we'll do more testing tomorrow.

@Borkason
Cherokee Project member

From ste...@konink.de on April 29, 2010 20:24:50
Anyway if anyone wants to get this solved;

  • use a debug version
  • have gdb available

If it happens try to make a GDB dump of all threads.
thread apply all bt

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on April 30, 2010 05:31:57
This was a huge problem for me, but I haven't had any issues since I updated to
OpenSSL 0.9.8n and re-generated/signed my certificates.

@Borkason
Cherokee Project member

From larat...@googlemail.com on April 30, 2010 09:57:52
@davisd: thanks for the info. DId you compile/install OpenSSL 0.9.8n manually, or is
there a deb package somewhere I missed?

I compiled 0.99.48 with debug info (at least I thought so), but GDB failed to find
the symbols. I am not a GDB pro, so this is probably my fault.

Anyway, if I'm running the debug 0.99.48, the error occurs every 5 to 10 requests,
very easy to reproduce. The GDB dump does not contain symbols. I attached it to this
message.

If you can give me a hint about the symbols I'll repeat it.

@Borkason
Cherokee Project member

From oberroc...@gmail.com on April 30, 2010 10:01:20
[Sorry, I added the last post with my wife's account, but it's still me :-)]

Another observation: I cannot reproduce the error with cherokee tracing enabled like
this:

CHEROKEE_TRACE="all" ./cherokee > trace.txt 2>&1

So with all this tracing it seems to run stable...

@Borkason
Cherokee Project member

From davisd.davisd@gmail.com on April 30, 2010 13:45:48
@larathie I'm actually running arch linux now- I use pacman to update my packages
from arch's repos... They're more up-to-date than Debian / Ubuntu.

The thing is, when I originally loaded Arch, I took my old self-signed SSL
certificates with me (Which were generated with openssl 0.9.8g). Even though I was
running a newer version of openssl (0.9.8m) with cherokee, the certificate was in
fact generated with the older openssl (I was mistaken in a previous post, I thought
the certificate was generated under 0.9.8m, but it was not).

BTW: I'm running openssl 1.0.0 now and all is well... Everything also worked well
under 0.9.8m and 0.9.8n

Try generating the certificates with a newer version of openssl. Make sure you're
linked against the right version of openssl:

openssl version

-David

@Borkason
Cherokee Project member

From oberroc...@gmail.com on April 30, 2010 13:51:45
@David: thanks! I don't think it's the certificate, at least not mine: it is not
self-created, it is commercial. However, it uses a certification chain, maybe this
makes cherokee hick up?

From what you posted, there seems no evidence that the problem is caused by Cherokee
running with an older OpenSSL; it might just be the certificate, right?

Thanks again
Markus

@Borkason
Cherokee Project member

From kallist...@gmail.com on April 30, 2010 19:15:01
Bah.

updated to ubuntu 10.04 on my server:
#​ openssl version
OpenSSL 0.9.8k 25 Mar 2009

Created new CA.
Regenerated all ssl keys and certificates and self signed them with my new CA.

Reinstalled desktop machine with Ubuntu 10.04. (thus clearing all cached ssl keys)

Still getting sec_error_bad_signature errors and half loaded pages randomly... SIGH.

@Borkason
Cherokee Project member

From zsd...@gmail.com on May 27, 2010 14:48:55
Hi,

Ubuntu 10.04
Cherokee 0.99.39-4.1
Openssl 0.9.8k-7ubuntu8

greenbar class4 certificate from starssl.com.

Exactly the same error! Does anybody has any news according to this bug?
It's working for 10 minutes and i have to restart the server because I getting
sec_error_bad_signature errors from firefox and half loaded pages randomly!

Thanks!
Zsolt

@Borkason
Cherokee Project member

From zsd...@gmail.com on May 27, 2010 14:48:59
Hi,

Ubuntu 10.04
Cherokee 0.99.39-4.1
Openssl 0.9.8k-7ubuntu8

greenbar class4 certificate from starssl.com.

Exactly the same error! Does anybody has any news according to this bug?
It's working for 10 minutes and i have to restart the server because I getting
sec_error_bad_signature errors from firefox and half loaded pages randomly!

Thanks!
Zsolt

@Borkason
Cherokee Project member

From ste...@konink.de on May 27, 2010 14:58:55
OpenSSL 1 is out.
Cherokee 1 is out.

Can you reproduce it with both?

@Borkason
Cherokee Project member

From zsd...@gmail.com on May 27, 2010 16:04:59
I have compiled the latest cherokee version (1.0.1) from source. Still getting
half loaded pages randomly if I try to use https.

I don't like to recompile openssl a lot of other package is depending on it. I think
so the problem is in cherokee not in openssl... cherokee making general protection
fault and restart when I try to download a page via https.

@Borkason
Cherokee Project member

From ste...@konink.de on May 27, 2010 16:37:28
If you have a cherokee version with debug enabled (standard configuratio). And you can
provide a backtrace what goes wrong. We can help you obviously.

But since I personally (reading the logs of openssl) am still convinced this is not
Cherokee specific, someone has to prove me wrong.

@Borkason
Cherokee Project member

From zsd...@gmail.com on May 27, 2010 18:06:19
Interesting. When I just clicking on the links of the webpage in the browser it's
working, but when i press ctrl+r in the browser, then it's start to half load the
pages and gives sec_error_bad_signature error and cherokee segfaulting.

@Borkason
Cherokee Project member

From ste...@konink.de on May 27, 2010 18:10:12
If it is /that/ easy to reproduce.

gdb cherokee-worker

once it crashes:

bt

followed by:

thread apply all bt

Please paste the output.

@Borkason
Cherokee Project member

From alobbs on May 28, 2010 08:07:49
@zsdemi: What browser, version and platform are you working with?

@Borkason
Cherokee Project member

From peter.do...@gmail.com on May 28, 2010 08:47:22
i have similar problem, ubuntu karmic, i use simple roundcube webmail only with https.

@Borkason
Cherokee Project member

From zsd...@gmail.com on May 28, 2010 09:11:00
I have tired from ubuntu 10.04 desktop and from windows 7 too. I'm using firefox
3.6.3 on both platform. IE8 is do the same on windows7, but konqueror is working on
ubuntu and opera is working on both platform (but konqueror and opera don't accept my
certificate as a trusted one - i think so this is the reason).

I have noticed the problem is starting, when cherokee response 304 for requests and
send 0 byte data. Since yesterday cherokee not segfaulting, but firefox and ie still
load the half of the pages and throw sec_error_bad_signature after some page reload.

Here you can check my certificate
openssl s_client -connect madtatu.com:443

@Borkason
Cherokee Project member

From magyar.g...@gmail.com on May 28, 2010 17:11:01
I have installed apache2 on the server with exactly the same certificate and it's
working correctly. If you like to reproduce the error, simply install a new ubuntu
10.04 and install cherokee from source and make a signed certificate. You can sign it
for free at startssl . com . Restart the server and make an apache benchmark test on
the server ( ab -n 1000 -c 10 https://server ) You will get the error!

@Borkason
Cherokee Project member

From ste...@konink.de on May 28, 2010 20:06:50
Installing ubuntu, I want this bug killed by tonight. Like we killed the queenbee
today, in order to allow her daughters to live happily ever after. (I'm getting
offtopic and sentimental).

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 02:00:59
Lets fix this issue bit by bit.

First; kill the memoryleak:

Index: socket.c

--- socket.c (revision 5163)
+++ socket.c (working copy)
@@ -147,7 +147,6 @@

    if (socket->cryptor != NULL) {
            cherokee_cryptor_socket_clean (socket->cryptor);
  •           socket->cryptor = NULL;
    }
    
    return ret_ok;
    

Now what I have still remaining;

ab complains about:
SSL handshake failed (1).
14793:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not
01:rsa_pk1.c💯
14793:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check
failed:rsa_eay.c:697:
14793:error:1408D07B:SSL routines:SSL3_GET_KEY_EXCHANGE:bad signature:s3_clnt.c:1448:

And Chromium crashes hard on:
[14866:14878:3158390104:ERROR:net/socket/ssl_client_socket_nss.cc(1022)] handshake
failed; NSS error code -8182, net_error -207
Segmentation fault

@Borkason
Cherokee Project member

From alobbs on May 29, 2010 07:07:04
Goot catch Stefan!! :-)

IMO the patch should be something in this line, though:

diff --git a/cherokee/socket.c b/cherokee/socket.c
index 82c7241..b893355 100644
--- a/cherokee/socket.c
+++ b/cherokee/socket.c
@@ -146,7 +146,7 @@ cherokee_socket_clean (cherokee_socket_t *socket)
memset (&socket->client_addr, 0, sizeof(cherokee_sockaddr_t));

    if (socket->cryptor != NULL) {
  • cherokee_cryptor_socket_clean (socket->cryptor);
  • cherokee_cryptor_socket_free (socket->cryptor); socket->cryptor = NULL; }

If I'm not wrong, we actually want the cryptor object to be freed.
Could you please give it a try?

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 08:05:09
The free will happen anyway. Are you sure you want this? Since _clean != _mrproper?

If you want to debug yourself recompile openssl with -DPURIFY, I'm currently going for
the SSL_SHUTDOWN stuff since that comes up next.

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 08:08:04
==19204== Thread 8:
==19204== Invalid read of size 4
==19204== at 0x480C91D: SSL_shutdown (in /usr/lib/libssl.so.0.9.8)
==19204== by 0x40A144F: _socket_close (cryptor_libssl.c:540)
==19204== by 0x40525D4: cherokee_cryptor_socket_close (cryptor.c:202)
==19204== by 0x405446D: cherokee_socket_close (socket.c:193)
==19204== by 0x40844B7: cherokee_connection_clean_close (connection.c:383)
==19204== by 0x408C162: purge_connection (thread.c:344)
==19204== by 0x408D198: process_active_connections (thread.c:1308)
==19204== by 0x408E0DF: cherokee_thread_step_MULTI_THREAD (thread.c:1825)
==19204== by 0x408E847: thread_routine (thread.c:97)
==19204== by 0x40C28FE: start_thread (in /lib/libpthread-2.11.so)
==19204== by 0x433338D: clone (in /lib/libc-2.11.so)
==19204== Address 0x20 is not stack'd, malloc'd or (recently) free'd

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 08:30:08

Index: cryptor_libssl.c

--- cryptor_libssl.c (revision 5163)
+++ cryptor_libssl.c (working copy)
@@ -537,7 +537,10 @@
static ret_t
_socket_close (cherokee_cryptor_socket_libssl_t *cryp)
{

  • SSL_shutdown (cryp->session);
  • if (cryp->session != NULL) {
  • SSL_shutdown (cryp->session);
  • } + return ret_ok; }

I'm suggesting this. But only because I don't get why session can become NULL before
the socket close.

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 09:13:41
I wonder the following is 'ok'.

==23985== Possible data race during write of size 4 at 0x42531c4 by thread #​10 (Github: #89)
==23985== at 0x411703B: CRYPTO_malloc (in /usr/lib/libcrypto.so.0.9.8)
==23985== by 0x419CEFC: EVP_MD_CTX_copy_ex (in /usr/lib/libcrypto.so.0.9.8)
==23985== by 0x4803471: tls1_final_finish_mac (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x47FC909: ssl3_do_change_cipher_spec (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x47FC49C: ssl3_read_bytes (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x47FD676: ssl3_get_message (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x47F0C8E: ssl3_get_cert_verify (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x47ED8FA: ssl3_accept (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x480F6D0: SSL_accept (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x40A448F: _socket_init_tls (cryptor_libssl.c:518)
==23985== by 0x4055612: cherokee_cryptor_socket_init_tls (cryptor.c:193)
==23985== by 0x405752C: cherokee_socket_init_tls (socket.c:169)
==23985== This conflicts with a previous write of size 4 by thread #​1 (Github: #81)
==23985== at 0x411703B: CRYPTO_malloc (in /usr/lib/libcrypto.so.0.9.8)
==23985== by 0x480E165: SSL_new (in /usr/lib/libssl.so.0.9.8)
==23985== by 0x40A455B: _socket_init_tls (cryptor_libssl.c:466)
==23985== by 0x4055612: cherokee_cryptor_socket_init_tls (cryptor.c:193)
==23985== by 0x405752C: cherokee_socket_init_tls (socket.c:169)
==23985== by 0x409099B: process_active_connections (thread.c:721)
==23985== by 0x40910DF: cherokee_thread_step_MULTI_THREAD (thread.c:1825)
==23985== by 0x408AE83: cherokee_server_step (server.c:1132)

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 09:27:00
/lib/libc.so.6(+0x6b311)[0xb7ccc311]
/lib/libc.so.6(+0x6cb88)[0xb7ccdb88]
/lib/libc.so.6(cfree+0x6d)[0xb7cd0c6d]
/usr/lib/libcrypto.so.0.9.8(CRYPTO_free+0x40)[0xb7de8361]
/usr/lib/libcrypto.so.0.9.8(EVP_PKEY_free+0x8b)[0xb7e74656]
/usr/lib/libcrypto.so.0.9.8(X509_get_pubkey_parameters+0x188)[0xb7ea4316]
/usr/lib/libcrypto.so.0.9.8(X509_verify_cert+0x733)[0xb7ea29bf]
/usr/lib/libssl.so.0.9.8(ssl3_output_cert_chain+0x159)[0xb7c063d5]
/usr/lib/libssl.so.0.9.8(ssl3_send_server_certificate+0x99)[0xb7bfa8a4]
/usr/lib/libssl.so.0.9.8(ssl3_accept+0x5ed)[0xb7bf64cc]
/usr/lib/libssl.so.0.9.8(SSL_accept+0x38)[0xb7c186d1]
/usr/lib/libssl.so.0.9.8(ssl23_get_client_hello+0xab5)[0xb7c07b79]
/usr/lib/libssl.so.0.9.8(ssl23_accept+0x1f9)[0xb7c06ff1]
/usr/lib/libssl.so.0.9.8(SSL_accept+0x38)[0xb7c186d1]
/opt/cherokee-ctk/lib/cherokee/libplugin_libssl.so(+0x2490)[0xb7f69490]
/opt/cherokee-ctk/lib/libcherokee-
base.so.0(cherokee_cryptor_socket_init_tls+0x23)[0xb7fbb613]
/opt/cherokee-ctk/lib/libcherokee-
base.so.0(cherokee_socket_init_tls+0x3d)[0xb7fbd52d]
/opt/cherokee-ctk/lib/libcherokee-server.so.0(+0x1699c)[0xb7f8399c]
/opt/cherokee-ctk/lib/libcherokee-
server.so.0(cherokee_thread_step_MULTI_THREAD+0x2b0)[0xb7f840e0]
/opt/cherokee-ctk/lib/libcherokee-server.so.0(+0x17848)[0xb7f84848]
/lib/libpthread.so.0(+0x58ff)[0xb7f3c8ff]
/lib/libc.so.6(clone+0x5e)[0xb7d2e38e]

...pssst

I think this is all related to a socket close (client) while the server doesn't
expect it.

@Borkason
Cherokee Project member

From ste...@konink.de on May 29, 2010 22:45:50
Your suggested change _clean => _free results in:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb3bd8b70 (LWP 4630)]
0xb7e6840b in sk_pop_free () from /usr/lib/libcrypto.so.0.9.8
(gdb) bt
#​0 0xb7e6840b in sk_pop_free () from /usr/lib/libcrypto.so.0.9.8
#​1 0xb7e7464b in EVP_PKEY_free () from /usr/lib/libcrypto.so.0.9.8
#​2 0xb7ea4316 in X509_get_pubkey_parameters ()
from /usr/lib/libcrypto.so.0.9.8
#​3 0xb7ea29bf in X509_verify_cert () from /usr/lib/libcrypto.so.0.9.8
#​4 0xb7c063d5 in ssl3_output_cert_chain () from /usr/lib/libssl.so.0.9.8
#​5 0xb7bfa8a4 in ssl3_send_server_certificate () from /usr/lib/libssl.so.0.9.8
#​6 0xb7bf64cc in ssl3_accept () from /usr/lib/libssl.so.0.9.8
#​7 0xb7c186d1 in SSL_accept () from /usr/lib/libssl.so.0.9.8
#​8 0xb7c07b79 in ssl23_get_client_hello () from /usr/lib/libssl.so.0.9.8
#​9 0xb7c06ff1 in ssl23_accept () from /usr/lib/libssl.so.0.9.8
#​10 0xb7c186d1 in SSL_accept () from /usr/lib/libssl.so.0.9.8
#​11 0xb7f69490 in _socket_init_tls (cryp=0x81a4d18, sock=0x81150e8,
vsrv=0x807b8c8) at cryptor_libssl.c:518
#​12 0xb7fbb5f3 in cherokee_cryptor_socket_init_tls (cryp=0xb7e83f49,
sock=0x81150e8, vsrv=0x807b8c8) at cryptor.c:193
#​13 0xb7fbd50d in cherokee_socket_init_tls (socket=0x81150e8,
vserver=0x807b8c8) at socket.c:170
#​14 0xb7f8399c in process_active_connections (thd=0x805ed38) at thread.c:721
#​15 0xb7f840e0 in cherokee_thread_step_MULTI_THREAD (thd=0x805ed38,
dont_block=false) at thread.c:1825
#​16 0xb7f84848 in thread_routine (data=0x805ed38) at thread.c:97
#​17 0xb7f3c8ff in start_thread () from /lib/libpthread.so.0
---Type to continue, or q to quit---
#​18 0xb7d2e38e in clone () from /lib/libc.so.6

@Borkason
Cherokee Project member

From ste...@konink.de on May 30, 2010 19:59:30
This basically fixes the memoryleak caused by the DH_new in the cryptor_libssl_dh_*.c
files.

Index: cryptor_libssl.c

--- cryptor_libssl.c (revision 5163)
+++ cryptor_libssl.c (working copy)
@@ -69,6 +73,18 @@
*/
EVP_cleanup();

+#if HAVE_OPENSSL_ENGINE_H

  • while (e != NULL) {
  • ENGINE_finish(e);
  • ENGINE_free(e);
  • } +#endif +
  • if (dh_param_512 != NULL) DH_free(dh_param_512);
  • if (dh_param_1024 != NULL) DH_free(dh_param_1024);
  • if (dh_param_2048 != NULL) DH_free(dh_param_2048);
  •   if (dh_param_4096 != NULL) DH_free(dh_param_4096);
    

    +
    cherokee_cryptor_free_base (CRYPTOR(cryp));
    return ret_ok;
    }
    @@ -250,17 +266,17 @@

    switch (keylen) {
    case 512:
    
  • if (dh_param_512) return dh_param_512;
  • return get_dh512();
  • if (dh_param_512 == NULL) dh_param_512 = get_dh512();
  • return dh_param_512; case 1024:
  • if (dh_param_1024) return dh_param_1024;
  • return get_dh1024();
  • if (dh_param_1024 == NULL) dh_param_1024 = get_dh1024();
  • return dh_param_1024; case 2048:
  • if (dh_param_2048) return dh_param_2048;
  • return get_dh2048();
  • if (dh_param_2048 == NULL) dh_param_2048 = get_dh2048();
  • return dh_param_2048; case 4096:
  • if (dh_param_4096) return dh_param_4096;
  • return get_dh4096();
  • if (dh_param_4096 == NULL) dh_param_4096 = get_dh4096();
  • return dh_param_4096; } return NULL; }

What currently remains are ENGINE functions by the plugin init. I have tried several
approaches even by keeping the 'e' global. But that still doesn't free it with
ENGINE_free.

==5129== 257 (104 direct, 153 indirect) bytes in 1 blocks are definitely lost in loss
record 77 of 80
==5129== at 0x402586E: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-
linux.so)
==5129== by 0x4113A47: default_malloc_ex (mem.c:79)
==5129== by 0x41140A0: CRYPTO_malloc (mem.c:328)
==5129== by 0x418200F: ENGINE_new (eng_lib.c:68)
==5129== by 0x4183049: ENGINE_by_id (eng_list.c:370)
==5129== by 0x4183103: ENGINE_by_id (eng_list.c:405)
==5129== by 0x40A09A2: ???
==5129== by 0x408FD1A: load_common (plugin_loader.c:250)
==5129== by 0x408FEEB: cherokee_plugin_loader_get (plugin_loader.c:498)
==5129== by 0x4087C01: configure_server_property (server.c:1475)
==5129== by 0x405E8E2: cherokee_config_node_while (config_node.c:226)
==5129== by 0x4086991: configure_server (server.c:1562)

@Borkason
Cherokee Project member

From ste...@konink.de on May 30, 2010 20:00:45
Ignore the first while loop, that doesn't work;

@Borkason
Cherokee Project member

From ste...@konink.de on May 31, 2010 13:30:28
Incorporates the above fixes plus the mutices.

@Borkason
Cherokee Project member

From alobbs on May 31, 2010 14:09:30
http://svn.cherokee-project.com/changeset/5171

@Borkason
Cherokee Project member

From ste...@konink.de on June 01, 2010 01:36:14
Patch filename says enough.

The other calls to free might be redundant. The actual issue is that there seams to be
application exchange until a proper shutdown. The ret_eagain makes sure every thing is
sent, and the _free is convenient to differentiate between the first round and the
second one.

@Borkason
Cherokee Project member

From alobbs on June 01, 2010 07:30:20
Very interesting findings, congrats!

Although, I don't get the rationale behind the patch. Even if that return might have some positive side effect,
it's certainly not the solution. Actually, cherokee_socket_close() and cherokee_socket_clean() are called in line
from connection.c, so all the libssl are executed in the same order.

I'll try to figure out what the problem is, and how to work out the code.
Cheers!

@Borkason
Cherokee Project member

From ste...@konink.de on June 01, 2010 08:17:03
The possitive effect comes from any delay that allows the last part of the data, to be
sent. Maybe there should be a 'flush' higher in the chain. It is very easy to spot
when you compare one request, where you break in gdb on the socket_close operation.
And one without.

The RST caused by fd_close prevents the remaining data to be sent.

@Borkason
Cherokee Project member

From alobbs on June 01, 2010 08:34:31
Right now, there are two possibilities:

¹ Try to use cherokee_socket_flush() during the TLS shutdown process. Pros: Trivial to implement. Cons: It is
not completely safe.

² Turn the socket into synchronous mode before the TLS shutdown. Pros: Easy to implement. Cons: Would
have a big performance penalty. It'd open the door to DoS attacks.

Then, the third option would be to create a new "phase_close" connection phase, where the sockets could be
closed safely. It'd require much more work, but AFAIK it's the only safe way to address the problem.

@Borkason
Cherokee Project member

From thebigsl...@gmail.com on June 01, 2010 11:32:29
Just a quick thought.

Option 1 does allow a race, however, the negative effect of satisfying that race
results in the same effect as we see here. The timing window is much narrower,
though, so tuning as a short term solution is still of significant benefit to active
users of the webserver.

@Borkason
Cherokee Project member

From cjm...@gmail.com on July 08, 2010 18:20:04
Stefan suggested I add this screenshot to this bug, as it seems this is related.

Also, as a side note, I tend to experience this more with Firefox, IE never seems to trigger the bug!

@Borkason
Cherokee Project member

From kallist...@gmail.com on September 16, 2010 15:24:44
It seems the ssl crashes have gotten a lot better since r5171 in chrome . I have been poking at ssl.unixzen.com for a while now in chrome without any catastrophic failures.

I do keep getting blank pages in firefox however as of 1.0.8

@Borkason
Cherokee Project member

From ste...@konink.de on April 23, 2011 19:38:50
Latest SVN test applies here too.

@Borkason
Cherokee Project member

From ste...@konink.de on October 13, 2011 22:53:29
I think this SSL error is gone. A few are remaining for Chrome.

@Borkason Borkason closed this Mar 24, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment