Server error: ConnectTimeout: HTTPConnectionPool

[kakaNo1]

16:34:46.561Z ERROR elastalert-server:
ProcessController: ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1173, in run_all_rules
num_matches = self.run_rule(rule, endtime, self.starttime)
File "/opt/elastalert/elastalert/elastalert.py", line 870, in run_rule
if not self.run_query(rule, rule['starttime'], endtime):
File "/opt/elastalert/elastalert/elastalert.py", line 602, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/opt/elastalert/elastalert/elastalert.py", line 331, in get_hits
if self.current_es.is_atleastsixsix():
File "elastalert/init.py", line 69, in is_atleastsixsix
major, minor = map(int, self.es_version.split(".")[:2])
File "elastalert/init.py", line 43, in es_version
self._es_version = self.info()['version']['number']
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/init.py", line 259, in info
return self.transport.perform_request("GET", "/", params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/connection/http_requests.py", line 138, in perform_request
raise ConnectionTimeout("TIMEOUT", str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7efc1794ced0>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=20)')))

ERROR:root:Uncaught exception running rule poolqq: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7efc1794ced0>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=20)')))

If this error occurs on my server, the alarm will not be issued again, and only by restarting the server can the alarm be triggered again
However, 9200 is normal. May I ask why?

[nsano-rururu]

It is a sample

Elasticsearch 7.8.0
Kibana 7.8.0
bitsensor/elastalert:3.0.0-beta.0

/home/user/docker-wk
|--docker-compose.yml
|--Dockerfiles
|  |--Dockerfile-elastalert
|
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|
|--kibana
|  |--config
|  |  |--kibana.yml
|
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--config.json
|  |  |--elastalert-test.yaml
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.8.0
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-elastalert
    image: elastalert:3.0.0-beta.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
      - kibana
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
      - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

Dockerfiles/Dockerfile-elastalert

FROM bitsensor/elastalert:3.0.0-beta.0

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh & \
    chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

elastalert/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

elastalert/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

elastalert/config/config.json

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "writeback_index": "elastalert_status"
}

elastalert/config/elastalert-test.yml

# NOTE: This config is used when testing a rule

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

elastalert/config/elastalert.yml

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

chmod 777 es/data
chmod 777 elastalert/rules
chmod 777 elastalert/rule_templates
docker-compose up -d

$ docker logs -f elastalert

Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@3.0.0-beta.0 start /opt/elastalert-server
> sh ./scripts/start.sh

13:00:34.886Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
13:00:34.897Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
13:00:34.922Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
13:00:35.013Z  INFO elastalert-server: Router:  Listening for GET request on /.
13:00:35.014Z  INFO elastalert-server: Router:  Listening for GET request on /status.
13:00:35.015Z  INFO elastalert-server: Router:  Listening for GET request on /status/control/:action.
13:00:35.015Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
13:00:35.016Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
13:00:35.018Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id.
13:00:35.027Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id.
13:00:35.027Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id.
13:00:35.031Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
13:00:35.032Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id.
13:00:35.035Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id.
13:00:35.035Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id.
13:00:35.038Z  INFO elastalert-server: Router:  Listening for POST request on /test.
13:00:35.041Z  INFO elastalert-server: Router:  Listening for GET request on /config.
13:00:35.041Z  INFO elastalert-server: Router:  Listening for POST request on /config.
13:00:35.042Z  INFO elastalert-server: Router:  Listening for POST request on /download.
13:00:35.043Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/:type.
13:00:35.043Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
13:00:35.044Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
13:00:35.048Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
13:00:35.048Z  INFO elastalert-server: ProcessController:  Creating index
13:00:40.962Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.8.0
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
    
13:00:40.962Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
13:00:40.971Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
13:00:41.006Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 229)
13:00:41.022Z  INFO elastalert-server: Server:  Server listening on port 3030
13:00:41.041Z  INFO elastalert-server: Server:  Websocket listening on port 3333
13:00:41.042Z  INFO elastalert-server: Server:  Server started
13:00:42.949Z  INFO elastalert-server:
    ProcessController:  0 rules loaded
    
13:00:53.024Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

$ docker ps

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED              STATUS                        PORTS                                            NAMES
e3d3620a31b9        elastalert:3.0.0-beta.0                               "/usr/local/bin/elas…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp   elastalert
d6ab98d03836        docker.elastic.co/kibana/kibana:7.8.0                 "/usr/local/bin/dumb…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:5601->5601/tcp                           kibana
14f1e131c0b9        docker.elastic.co/elasticsearch/elasticsearch:7.8.0   "/tini -- /usr/local…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch

[kakaNo1]

Ok, thank you for your timely reply. I deployed the official Docker, and there was no problem at the beginning (the alarm was normal). Within a week or a few days, the above error would appear, and then no alarm would be sent

[xuanyuanaosheng]

@nsano-rururu The elastalert now is only support the python 3.

[nsano-rururu]

Ok, thank you for your timely reply. I deployed the official Docker, and there was no problem at the beginning (the alarm was normal). Within a week or a few days, the above error would appear, and then no alarm would be sent

・Which Docker image are you using?

Docker image name	tag	ElastAlert	Remarks
bitsensor/elastalert	2.0.1	0.1.39	Problem with Elastcserach 7.x
bitsensor/elastalert	lastet	0.1.39	Problem with Elastcserach 7.x
bitsensor/elastalert	3.0.0-beta.0	0.2.0b2
bitsensor/elastalert	3.0.0-beta.1	0.2.0b2
servercentral/elastalert	latest	0.2.1
daichi703n/elastalert	0.2.1-dev2	0.2.1
johnsusek/elastalert-server	1592081541	0.2.4	Library update Babel 6 → 7 bug fix

・Cloud (AWS1, Azure, etc.), physical server?

・What are the server specifications? (CPU, number of CPU cores, memory, etc.)

・Are you using Kibana without problems when an error occurs?

・Is Elasticsearch running on docker ps?

・Did you check docker logs to see if there are any errors in the Elasticsearch container?

・How much memory is available by executing the "free -h" command when an error occurs?

[nsano-rururu]

@nsano-rururu The elastalert now is only support the python 3.

Yeah, right,

[kakaNo1]

好的，谢谢您的及时答复。我部署了官方的Docker，一开始没有问题（警报是正常的）。在一周或几天内，将出现上述错误，然后将不发送警报

・您使用哪个Docker映像？

Docker映像名称 标签 ElastAlert 备注
位传感器/弹性 2.0.1 0.1.39 Elastcserach 7.x问题
位传感器/弹性 et 0.1.39 Elastcserach 7.x问题
位传感器/弹性 3.0.0-beta.0 0.2.0b2
位传感器/弹性 3.0.0-beta.1 0.2.0b2
服务器中心/弹性 最新 0.2.1
daichi703n / elastalert 0.2.1-dev2 0.2.1
约翰苏塞克/弹性服务器 1592081541 0.2.4 库更新
Babel 6→7
错误修复
・云（AWS1，Azure等），物理服务器？

・服务器规格是多少？（CPU，CPU内核数，内存等）

・发生错误时，您是否在使用Kibana时没有问题？

・ Elasticsearch是否在docker ps上运行？

・您是否检查了docker日志以查看Elasticsearch容器中是否有任何错误？

・发生错误时，通过执行“ free -h”命令可以使用多少存储空间？

The mirror is bitsensor/elastalert: 3.0.0 - beta. 1
Elasticsearch version 7.2.0
Elastalert Server is available in the virtual machine
Elasticsearch Cluster in Aliyun (2 8C16G)
There's nothing wrong with Kibana when an error occurs
There's enough storage space
Python3 is used to run

[nsano-rururu]

I've never run ElastAlert on an Elasticsearch Cluster, so my current knowledge may not tell...

[nsano-rururu]

I'm aware that Elasticsearch, Kibana and ElastAlert are running on the same server.
It seems to be connectivity issue. Can you connect to ES with curl -v http://localhost:9200 from docker host?

[nsano-rururu]

Is that server up and running with anything besides Elasticsearch, Kibana and ElatAlert installed?
Could you tell me the information of docker-compose.yml and the configuration file when docker container is started?

[nsano-rururu]

Since the error message is connection timeout, the size of the index to be searched may be very large, but I can not clearly say that it is only the information provided so far.

[nsano-rururu]

I found a setting in the ElastAlert documentation to change the timeout period.
Add es_conn_timeout to elastalert.yml of ElastAlert and set the setting value to a large number (default is 20)
Try restarting the Docker container for ElastAlert.

es_conn_timeout: Optional; sets timeout for connecting to and reading from es_host; defaults to 20.
https://elastalert.readthedocs.io/en/latest/elastalert.html#configuration

[kakaNo1]

我在ElastAlert文档中找到一个设置来更改超时时间。
将es_conn_timeout添加到ElastAlert的elastalert.yml，并将设置值设置为一个较大的数字（默认值为20），
尝试重新启动ElastAlert的Docker容器。

es_conn_timeout：可选；设置连接和读取es_host的超时；默认为20。https
://elastalert.readthedocs.io/en/latest/elastalert.html#configuration

Thank you for your timely reply. I will try today. My server is running in K8S, do you need to check YAML

[kakaNo1]

I set it up in config.yaml
Es_conn_timeout: 50
Let's see if we can make the same mistakes tomorrow