Logstash fails to connect the second elasticsearch node #179

lyr3051 · 2022-02-23T03:43:14Z

lyr3051
Feb 23, 2022

Hi

I'm Sunny.

There are two nodes in my Elasticsearch Cluster, and they are run in different VM.

When I start to run my "logstash", I found it fails to connect to the second node (elasticsearch2) in the elasticsearch cluster,
but my "elasticsearch2" and "logstash" is still running, please find the attached "logstash-log.txt" and "kibana-monitoring.png".

I try to add the extra_hosts in "logstash" container, but it also fail to connect "elasticsearch2" and the logstash couldn't be run, because there are no the certificate of "elasticsearch2", please find the attached "logstash-log-withextrahost.txt".

Do I have to add the certificate of "elasticsearch2" in this solution?

logstash-connect.zip

cwiechmann · 2022-02-23T07:20:34Z

cwiechmann
Feb 23, 2022

As you can see in the log, the hosts you configured cannot be resolved:
[2022-02-23T03:30:18,778][ERROR][logstash.outputs.elasticsearch][DomainAudit] Unable to retrieve Elasticsearch cluster uuid {:message=>"Elasticsearch Unreachable: [https://elasticsearch2:9201/][Manticore::ResolutionFailure] elasticsearch2", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError

I see you configured the following for ELASTICSEARCH_HOSTS: https://elasticsearch1:9200, https://elasticsearch2:9201

Please note:

the host-names must obviously match to your VM-Machines you created running Elasticsearch
- the hostnames must be resolveable from within your Docker-Container that runs Logstash
also the port must match to your configuration
- very likely the second is not 9201 and instead also 9200
And for HA & the possibility to do maintenance (like upgrades) you need to have 3 Elasticsearch-Instances running
- always unequal - 3, 5, 7, ... but not 2
- You can read more here: https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design.html

1 reply

lyr3051 Feb 23, 2022
Author

If I use the hostname of my VM-Machines to set as my ELASTICSEARCH_HOSTS, should I change the certificate?

Why I ask this question is because when I tried to set my ELASTICSEARCH_HOSTS to match my VM-Machines,
the second node add into cluster successfully,
however when I run the Kibana, it return the below logs:

kibana | {"type":"log","@timestamp":"2022-02-23T08:19:00+00:00","tags":["error","elasticsearch-service"],"pid":11,"message":"Unable to retrieve version information from Elasticsearch nodes. Hostname/IP does not match certificate's altnames: Host: elk. is not in the cert's altnames: DNS:localhost, DNS:api-env, DNS:elasticsearch1, DNS:.ec2.internal, DNS:.compute-1.amazonaws.com"}

cwiechmann · 2022-02-23T08:57:35Z

cwiechmann
Feb 23, 2022

Yes, in that case the default certificates doesn't match to the hostname anymore and normally you would have to configure a new set of certificates.
However, if you are using version >=4.1.0 you can disable the certificate validation if you want, which is okay for a test/dev system.

See here:

apigateway-openlogging-elk/env-sample

Lines 607 to 637 in 413e2e5

    
           # Controls the Kibana to Elasticsearch certificate validation mode  
        
           # Possible values: (full|certificate|none) 
        
           # Defaults to: full 
        
           # For more information please read  
        
           # https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html 
        
           # KIBANA_ELASTICSEARCH_SSL_VERIFICATIONMODE=none 
        
           APM_SERVER_KEY=config/certificates/apm-server.key 
        
           APM_SERVER_CRT=config/certificates/apm-server.crt 
        
           APM_SERVER_KEY_PASSPHRASE= 
        
           # Controls the APM-Server to Elasticsearch certificate validation mode 
        
           # Possible values: (full|strict|certificate|none) 
        
           # Defaults to: full 
        
           # For more information please read  
        
           # https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html 
        
           # APM_ELASTICSEARCH_SSL_VERIFICATIONMODE=none 
        
           # Controls the Logstash to Elasticsearch certificate validation mode for pipelines and monitoring 
        
           # Value certificate is translated into true for pipelines certificate validation. none into false 
        
           # Possible values: (certificate|none) 
        
           # Defaults to: certificate 
        
           # For more information please read  
        
           # https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_certificate_verification 
        
           # LOGSTASH_ELASTICSEARCH_SSL_VERIFICATIONMODE=none 
        
           # Controls the Filebeat to Elasticsearch certificate validation mode (for monitoring only) 
        
           # Possible values: (full|strict|certificate|none) 
        
           # Defaults to: full 
        
           # For more information please read  
        
           # https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html 
        
           # FILEBEAT_ELASTICSEARCH_SSL_VERIFICATIONMODE=none

1 reply

lyr3051 Feb 23, 2022
Author

I already generate a new certificate for ELASTICSEARCH_KEY and ELASTICSEARCH_CRT, and the they also be configured in the .env file.
It is fine to run the elasticsearch in single node, however when I try to run Kibana, it return the below logs:

kibana | {"type":"log","@timestamp":"2022-02-23T10:22:03+00:00","tags":["error","elasticsearch-service"],"pid":10,"message":"Unable to retrieve version information from Elasticsearch nodes. self signed certificate"}

The below steps is how I generate certificate:

create ssl.conf
please find the attached.
openssl req -x509 -new -nodes -sha256 -utf8 -days 3650 -newkey rsa:2048 -keyout server.key -out server.crt -config ssl.conf
ssl.conf.zip

cwiechmann · 2022-02-23T10:51:43Z

cwiechmann
Feb 23, 2022

You cannot use a Self-Signed-Certificate for your Elasticsearch as you ultimately need to register the signing CA for Logstash, Filebeat & APIBuilder4Easltic with the following parameter:

################################################################################################
#                         Certificates and Certificate authorities
################################################################################################
# This CA-File is mounted into all applications and used to validate certificates for each component.
# For instance, when Filebeat is sending Metrics to Elasticsearch or the ANM communicates with
# the API-Builder. 
# You can concatenet multiple certificates into one certificate file if you also use intermediate 
# certificates. Learn more: https://bit.ly/3mxWDSu
# Example command to created the chained cert file:
# openssl pkcs7 -inform DER -in certificate.p7b -print_certs > certificate_bundle.cer
# Used-By: APIBuilder4Elastic, Elasticsearch, Kibana, Logstash, Filebeat, Admin-Node-Manager
ELASTICSEARCH_CA=config/certificates/ca.crt

Check out the cert-tool from Elastic in order to generate the certificate. Some notes are here which might help:
https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/blob/develop/DEVELOPMENT.MD#create-the-shipped-certificates

6 replies

cwiechmann Feb 25, 2022

Logstash cannot validate the certificate of Elasticsearch:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://apim:9201/

As you are running Elasticsearch externally, the parameters
ELASTICSEARCH_KEY, ELASTICSEARCH_KEY_PASSPHRASE and ELASTICSEARCH_CRT
are not relevant to you. These parameters are used by the solution, when setting up an Elasticsearch-Cluster for you.

The relevant parameter you need to set is:

apigateway-openlogging-elk/env-sample

Lines 562 to 573 in 413e2e5

    
           ################################################################################################ 
        
           #                         Certificates and Certificate authorities 
        
           ################################################################################################ 
        
           # This CA-File is mounted into all applications and used to validate certificates for each component. 
        
           # For instance, when Filebeat is sending Metrics to Elasticsearch or the ANM communicates with 
        
           # the API-Builder.  
        
           # You can concatenet multiple certificates into one certificate file if you also use intermediate  
        
           # certificates. Learn more: https://bit.ly/3mxWDSu 
        
           # Example command to created the chained cert file: 
        
           # openssl pkcs7 -inform DER -in certificate.p7b -print_certs > certificate_bundle.cer 
        
           # Used-By: APIBuilder4Elastic, Elasticsearch, Kibana, Logstash, Filebeat, Admin-Node-Manager 
        
           ELASTICSEARCH_CA=config/certificates/ca.crt

This parameter must point to the CA which should be used by Logstash, API-Builder and Filebeat to validate the Elasticsearch-Server-Certificate.
It must point to the same CA, which has been generated by cert-util as it signed the certificates now used by Elasticsearch.

Concerning you instance.yaml. The name (e.g. - name: "elasticsearch1") becomes you certificate common name. As your Elasticsearch-Nodes are running on VM's: elk and apim it must be:

instances:
  - name: "elk"
  - name: "apim"

The dns: are subject alternative names, that are used as well to validate the server-certificate. You don't them, as long you use the same hostnames later, when you setup the connection.
For example: Your certificate CN becomes: elk. And only if you establish the connection with https://elk:9999 it will work. It will fail, if you try to connect for instance with: https://elk.domain.com:9999

cwiechmann Mar 3, 2022

Did you finally succeed @lyr3051?

lyr3051 Mar 3, 2022
Author

Did you finally succeed @lyr3051?

Thanks for your concern.

I'm still keep trying.

lyr3051 Mar 7, 2022
Author

Hi @cwiechmann

If all my Elastic Stack are installed by the release package 4.0.3, but I want to install two elasticsearch nodes to run on two VMs respectively.
Do I also need to do the configuration as you said?

Logstash cannot validate the certificate of Elasticsearch:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://apim:9201/
As you are running Elasticsearch externally, the parameters ELASTICSEARCH_KEY, ELASTICSEARCH_KEY_PASSPHRASE and ELASTICSEARCH_CRT are not relevant to you. These parameters are used by the solution, when setting up an Elasticsearch-Cluster for you.

The relevant parameter you need to set is:

apigateway-openlogging-elk/env-sample

Lines 562 to 573 in 413e2e5

################################################################################################

# Certificates and Certificate authorities

################################################################################################

# This CA-File is mounted into all applications and used to validate certificates for each component.

# For instance, when Filebeat is sending Metrics to Elasticsearch or the ANM communicates with

# the API-Builder.

# You can concatenet multiple certificates into one certificate file if you also use intermediate

# certificates. Learn more: https://bit.ly/3mxWDSu

# Example command to created the chained cert file:

# openssl pkcs7 -inform DER -in certificate.p7b -print_certs > certificate_bundle.cer

# Used-By: APIBuilder4Elastic, Elasticsearch, Kibana, Logstash, Filebeat, Admin-Node-Manager

ELASTICSEARCH_CA=config/certificates/ca.crt

This parameter must point to the CA which should be used by Logstash, API-Builder and Filebeat to validate the Elasticsearch-Server-Certificate. It must point to the same CA, which has been generated by cert-util as it signed the certificates now used by Elasticsearch.

Concerning you instance.yaml. The name (e.g. - name: "elasticsearch1") becomes you certificate common name. As your Elasticsearch-Nodes are running on VM's: elk and apim it must be:
instances:
  - name: "elk"
  - name: "apim"
The dns: are subject alternative names, that are used as well to validate the server-certificate. You don't them, as long you use the same hostnames later, when you setup the connection. For example: Your certificate CN becomes: elk. And only if you establish the connection with https://elk:9999 it will work. It will fail, if you try to connect for instance with: https://elk.domain.com:9999

cwiechmann Mar 7, 2022

Yes, of course. The most important parameter is then ELASTICSEARCH_HOSTS that will contain all of your Elasticsearch-Nodes and used by clients such a Logstash, APIBuilder4Elastic and Filebeat for Load-Balancing.

Please note: For a production environment you should have a 3 Node Elasticsearch-Cluster: https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design.html

lyr3051 · 2022-03-23T10:57:44Z

lyr3051
Mar 23, 2022
Author

Hi @cwiechmann
Thanks for your notice.

I will have 3 Node in my Elasticsearch-Cluster, but I want to make sure the to add the second node first to let logstash can connect to it successfully.

The issue is solved when I create the below instances.yml to generate the certificates:

The parameters ELASTICSEARCH_KEY and ELASTICSEARCH_CRT of two nodes on VM elk and apim are elk.crt and elk.key , and their ELASTICSEARCH_CA is pointed to the same CA.

The value of my ELASTICSEARCH_HOSTS is "https://elk:9200,https://apim:9201".

According to my configuration, the logstash connect to three nodes successfully,
however when I reference the below link to create the instances.yml like below setting, the ELASTICSEARCH_KEY and ELASTICSEARCH_CRT are elk.crt and elk.key on elk but apim.crt and apim.key on apim, the logstash still fail to connect the apim:9201.
https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash#step-2-4

Is my successful configuration available for this solution?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logstash fails to connect the second elasticsearch node #179

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Logstash fails to connect the second elasticsearch node #179

lyr3051 Feb 23, 2022

Replies: 4 comments · 8 replies

cwiechmann Feb 23, 2022

lyr3051 Feb 23, 2022 Author

cwiechmann Feb 23, 2022

lyr3051 Feb 23, 2022 Author

cwiechmann Feb 23, 2022

cwiechmann Feb 25, 2022

cwiechmann Mar 3, 2022

lyr3051 Mar 3, 2022 Author

lyr3051 Mar 7, 2022 Author

cwiechmann Mar 7, 2022

lyr3051 Mar 23, 2022 Author

lyr3051
Feb 23, 2022

Replies: 4 comments 8 replies

cwiechmann
Feb 23, 2022

lyr3051 Feb 23, 2022
Author

cwiechmann
Feb 23, 2022

lyr3051 Feb 23, 2022
Author

cwiechmann
Feb 23, 2022

lyr3051 Mar 3, 2022
Author

lyr3051 Mar 7, 2022
Author

lyr3051
Mar 23, 2022
Author