Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meaningless error message: bad header found ... #429

Closed
minfrin opened this issue Dec 18, 2017 · 9 comments

Comments

Projects
None yet
3 participants
@minfrin
Copy link

commented Dec 18, 2017

  • Search Guard and Elasticsearch version

elasticsearch v6.0.0
search-guard-6 v6.0.0+17~beta1

  • JVM version and operating system version

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

  • Number of nodes in your cluster

2

  • Description of the bug

When running two elasticsearch nodes for the very first time after installation, the following error is thrown:

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This typically means that
one node tried to connect to another with a non-node certificate (it had no OID or 
the searchguard.nodes_dn setting was incorrectly configured) or that someoneis spoofing requests.
See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md
        at com.floragunn.searchguard.ssl.util.ExceptionUtils.createBadHeaderException(ExceptionUtils.java:56) ~[?:?]
        at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:183) ~[?:?]

This message is thrown when SSLRequestHelper.containsBadHeader() returns true, but contains nothing useful that would pin down the cause of the message.

The message contains the weasel words "typically".

The message makes reference to a "non node certificate" without explicitly defining what a "non node certificate" means.

The message says that the certificate might have no OID without the OID being specified in the error message.

The message says the searchguard.nodes_dn setting might be incorrectly configured, but does not say what kind of misconfiguration has occurred. If "misconfiguration" actually meant "mismatch", the error did not print the DN of the certificate presented (as opposed to the DN of the certificate the admin thinks they presented).

The message links to the full TLS documentation URL https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md, which is too vague to be of any use. Obviously this part of the manual has been followed, or this error message would never appear.

To fix this, the following code needs to be fixed to include proper error handling:

https://github.com/floragunncom/search-guard-ssl/blob/e6c9a16d6b00ccc7d04b3e0b2fedd13a4903a3ac/src/main/java/com/floragunn/searchguard/ssl/http/netty/ValidatingDispatcher.java#L72

This specific code needs to be fixed so that the "key" and the "prefix" are properly described in the error message, so that the admin can take some kind of meaningful action:

https://github.com/floragunncom/search-guard-ssl/blob/15d781fc70c3ef71289abc28df94f8c42a71229a/src/main/java/com/floragunn/searchguard/ssl/util/SSLRequestHelper.java#L171

As it stands, you need blind luck to configure TLS - put one foot wrong and the entire server comes crashing down with no explanation.

@floragunncom

This comment has been minimized.

Copy link
Owner

commented Dec 18, 2017

We are happy to accept a PR

@minfrin

This comment has been minimized.

Copy link
Author

commented Dec 18, 2017

Further testing shows more meaningless error messages in the log on server B when elasticsearch is restarted on server A.

None of these logged messages tell us what the error is:

[2017-12-18T22:43:10,562][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:43:11,157][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:43:12,098][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:43:54,063][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:43:54,742][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:43:55,714][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:44:24,427][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:44:25,073][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:44:26,019][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:45:03,674][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:45:04,361][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T22:45:05,341][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T23:20:25,548][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T23:20:26,223][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-12-18T23:20:27,197][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers

@minfrin

This comment has been minimized.

Copy link
Author

commented Dec 18, 2017

ssldump picks up the following, which seems to show a successfully negotiated TLS connection.

Starting program: /usr/sbin/ssldump port 9300
New TCP connection #1: 172.29.230.121(38460) <-> 172.29.231.208(9300)
1 1 0.1060 (0.1060) C>S Handshake
ClientHello
Version 3.3
cipher suites
Unknown value 0xc024
Unknown value 0xc028
Unknown value 0x6b
Unknown value 0xc00a
Unknown value 0xc014
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Unknown value 0xc023
Unknown value 0xc027
TLS_DHE_DSS_WITH_NULL_SHA
Unknown value 0x40
Unknown value 0xc009
Unknown value 0xc013
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
Unknown value 0xc02c
Unknown value 0xc02b
Unknown value 0xc030
Unknown value 0x9f
Unknown value 0xa3
Unknown value 0xc02f
Unknown value 0x9e
Unknown value 0xa2
compression methods
NULL
1 2 0.1170 (0.0109) S>C Handshake
ServerHello
Version 3.3
session_id[32]=
5a 38 4d 39 ef f0 79 b1 d7 b8 e6 f2 53 5a d6 8b
2c 1f 13 bc 61 e3 2c 3d 60 f2 2f 50 62 71 cb 57
cipherSuite Unknown value 0xc028
compressionMethod NULL
Certificate
ServerKeyExchange
CertificateRequest
certificate_types rsa_sign
certificate_types dss_sign
certificate_types unknown value
Not enough data. Found 159 bytes (expecting 32767)
ServerHelloDone
1 3 0.2562 (0.1391) C>S Handshake
Certificate
ClientKeyExchange
1 4 0.2562 (0.0000) C>S Handshake
CertificateVerify
Not enough data. Found 258 bytes (expecting 16384)
1 5 0.2562 (0.0000) C>S ChangeCipherSpec
1 6 0.2562 (0.0000) C>S Handshake
1 7 0.2701 (0.0139) S>C ChangeCipherSpec
1 8 0.2701 (0.0000) S>C Handshake
1 9 0.2725 (0.0023) C>S application_data
1 10 0.2731 (0.0006) S>C application_data
1 11 0.2861 (0.0129) C>S application_data

@minfrin

This comment has been minimized.

Copy link
Author

commented Dec 18, 2017

We are happy to accept a PR

I would happily submit a PR, but to do so I need to understand more about what this code is trying to do.

What is a "header" in this context, and what makes it "bad"?

What is the significance of the prefix "sg_ssl", and why would this error be triggered if this prefix is somehow present?

@minfrin

This comment has been minimized.

Copy link
Author

commented Dec 19, 2017

Looking at the history of this code, it looks like an attempt was made to explain the bad header message here:

floragunncom/search-guard-ssl@ca7af70#diff-a07c36f84aa8d6675e5eeb74ac48bbba

Unfortunately the explanation makes no sense to me, as an admin I need concrete steps that I need to take to make this work, but I am mystified as to what those steps are.

Are you able to explain the significance of what makes a header bad? It seems you've received a header that starts with sg_ssl*, but there is nothing to explain why this would be a bad thing.

Would it be possible to confirm?

@minfrin

This comment has been minimized.

Copy link
Author

commented Dec 19, 2017

Following the advice in the following thread:

https://groups.google.com/forum/#!topic/search-guard/lZ3rbemeQE4

Setting as follows avoids the error:

searchguard.nodes_dn:
  - '*'

The original value was:

searchguard.nodes_dn:
  - DC=foo,DC=foo,O=Foo WW,OU=Machine Servers,CN=foo:bar:unstable:*:baz:*:elastic01

To start with, the error message needs to include the subject of the certificate, so it is clear what is being matched. At the moment it's a shot in the dark.

The next problem is wildcard handling - are multiple wildcards handled in a string, or does the wildcard handling only support wildcards under certain conditions?

@floragunncom floragunncom changed the title Meaningless error message: "bad header found. This typically means that one node tried to connect to another with a non-node certificate (it had no OID or the searchguard.nodes_dn setting was incorrectly configured) or that someoneis spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certifi cates.md" Meaningless error message: bad header found ... Dec 20, 2017

@floragunncom

This comment has been minimized.

Copy link
Owner

commented Dec 20, 2017

We will make the error messages more detailed and improve the docs. Would be great if you could contribute on this and help us here.

@servergeeks

This comment has been minimized.

Copy link

commented Jan 2, 2018

+1 on this issue... I'm having the same issue and I'm not sure how should I specify the nodes...

floragunncom added a commit to floragunncom/search-guard-ssl that referenced this issue Feb 2, 2018

floragunncom added a commit to floragunncom/search-guard-ssl that referenced this issue Feb 2, 2018

Merge branch 'master' into es-6.1.0
* master:
  Better error message for "Bad headers" floragunncom/search-guard#429
  update checksum plugin version

floragunncom added a commit that referenced this issue Feb 2, 2018

floragunncom added a commit to floragunncom/search-guard-ssl that referenced this issue Feb 7, 2018

floragunncom added a commit to floragunncom/search-guard-ssl that referenced this issue Feb 7, 2018

Merge branch '6.1.0' into es-6.1.2
* 6.1.0:
  Better error message for "Bad headers" floragunncom/search-guard#429
  update checksum plugin version

floragunncom added a commit to floragunncom/search-guard-ssl that referenced this issue Feb 7, 2018

Merge branch '6.1.0' into es-6.1.1
* 6.1.0:
  Better error message for "Bad headers" floragunncom/search-guard#429
  update checksum plugin version

floragunncom added a commit that referenced this issue Feb 7, 2018

floragunncom added a commit that referenced this issue Feb 7, 2018

Merge branch '6.1.0' into es-6.1.2
* 6.1.0:
  Implement custom attributes for internal authentication backend
  adjust kibana_user role, removed ro flag for some roles
  Print out installed Search Guard version
  Fix null keys
  ask for passwords, check cluster sanity
  fix #429
  Issue a warning when admin certificate is also a node certificate
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  fix possible NPE

floragunncom added a commit that referenced this issue Feb 7, 2018

Merge branch '6.1.0' into es-6.1.0
* 6.1.0:
  Implement custom attributes for internal authentication backend
  adjust kibana_user role, removed ro flag for some roles
  Print out installed Search Guard version
  Fix null keys
  ask for passwords, check cluster sanity
  fix #429
  Issue a warning when admin certificate is also a node certificate
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  fix possible NPE

floragunncom added a commit that referenced this issue Feb 7, 2018

Merge branch '6.1.0' into es-6.1.1
* 6.1.0:
  Implement custom attributes for internal authentication backend
  adjust kibana_user role, removed ro flag for some roles
  Print out installed Search Guard version
  Fix null keys
  ask for passwords, check cluster sanity
  fix #429
  Issue a warning when admin certificate is also a node certificate
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  fix possible NPE
@floragunncom

This comment has been minimized.

Copy link
Owner

commented Feb 7, 2018

Fixed for
SG SSL 5.6.7-23
SG 5.6.7-19
SG 6.1.x-21.0

floragunncom added a commit that referenced this issue Feb 13, 2018

Merge branch 'master' into es-6.x-api
* master:
  Bump to 6.2.1
  add smoketest to cci
  Update third party info
  Bump to ES 6.2.0
  remove circle ci 1.0 file
  adjust kibana_user role, removed ro flag for some roles
  update to 6.1.3
  Print out installed Search Guard version
  Fix null keys
  ask for passwords, check cluster sanity
  Implement custom attributes for internal authentication backend
  fix #429
  Issue a warning when admin certificate is also a node certificate
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  fix unittests, fix enterprise modules naming
  [TEST] fix truststore location to be in the same directory like the keystore

# Conflicts:
#	plugin-descriptor.properties
#	pom.xml

floragunncom added a commit that referenced this issue Jun 15, 2018

Merge branch '5.6.0' into es-5.6.5
* 5.6.0:
  update guava dependency to version 25.1
  Merge pull request #503 from floragunncom/feature/sgadmin_explicit_replicas
  Turn off query node cache for fls requests
  Add searchguard.dynamic.multi_rolespan_enabled to support evaluation permissions across different sg roles
  update demo certificates
  exclude deps to avoid jar hell
  Print out installed Search Guard version
  Fix null keys
  fix #429
  ask for passwords, check cluster sanity
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  Fix scroll check for internal requests

floragunncom added a commit that referenced this issue Jun 15, 2018

Merge branch '5.6.0' into es-5.6.0
* 5.6.0:
  update guava dependency to version 25.1
  Merge pull request #503 from floragunncom/feature/sgadmin_explicit_replicas
  Turn off query node cache for fls requests
  Add searchguard.dynamic.multi_rolespan_enabled to support evaluation permissions across different sg roles
  update demo certificates
  exclude deps to avoid jar hell
  Print out installed Search Guard version
  Fix null keys
  fix #429
  ask for passwords, check cluster sanity
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  Fix scroll check for internal requests

# Conflicts:
#	src/main/java/com/floragunn/searchguard/user/AuthCredentials.java

floragunncom added a commit that referenced this issue Jun 15, 2018

Merge branch '5.6.0' into es-5.6.4
* 5.6.0:
  update guava dependency to version 25.1
  Merge pull request #503 from floragunncom/feature/sgadmin_explicit_replicas
  Turn off query node cache for fls requests
  Add searchguard.dynamic.multi_rolespan_enabled to support evaluation permissions across different sg roles
  update demo certificates
  exclude deps to avoid jar hell
  Print out installed Search Guard version
  Fix null keys
  fix #429
  ask for passwords, check cluster sanity
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  Fix scroll check for internal requests

floragunncom added a commit that referenced this issue Jun 15, 2018

Merge branch '5.6.0' into es-5.6.6
* 5.6.0:
  update guava dependency to version 25.1
  Merge pull request #503 from floragunncom/feature/sgadmin_explicit_replicas
  Turn off query node cache for fls requests
  Add searchguard.dynamic.multi_rolespan_enabled to support evaluation permissions across different sg roles
  update demo certificates
  exclude deps to avoid jar hell
  Print out installed Search Guard version
  Fix null keys
  fix #429
  ask for passwords, check cluster sanity
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.

floragunncom added a commit that referenced this issue Jun 15, 2018

Merge branch '5.6.0' into es-5.6.2
* 5.6.0:
  update guava dependency to version 25.1
  Merge pull request #503 from floragunncom/feature/sgadmin_explicit_replicas
  Turn off query node cache for fls requests
  Add searchguard.dynamic.multi_rolespan_enabled to support evaluation permissions across different sg roles
  update demo certificates
  exclude deps to avoid jar hell
  Print out installed Search Guard version
  Fix null keys
  fix #429
  ask for passwords, check cluster sanity
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  Fix scroll check for internal requests

floragunncom added a commit that referenced this issue Jun 15, 2018

Merge branch '5.6.0' into es-5.6.3
* 5.6.0:
  update guava dependency to version 25.1
  Merge pull request #503 from floragunncom/feature/sgadmin_explicit_replicas
  Turn off query node cache for fls requests
  Add searchguard.dynamic.multi_rolespan_enabled to support evaluation permissions across different sg roles
  update demo certificates
  exclude deps to avoid jar hell
  Print out installed Search Guard version
  Fix null keys
  fix #429
  ask for passwords, check cluster sanity
  handle null hashes correctly
  Fix "Password dependent timing side channel in AuthCredentials" #439 by replacing Arrays.equals() with MessageDigest.isEqual() which is time constant.
  Fix scroll check for internal requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.