Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

search() got an unexpected keyword argument 'doc_type' #2204

Open
wisp888 opened this issue Apr 12, 2019 · 30 comments

Comments

Projects
None yet
@wisp888
Copy link

commented Apr 12, 2019

[root@afadsfasfsad /opt/elastalert]# /usr/local/python27/bin/elastalert
/usr/local/python27/lib/python2.7/site-packages/elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(__file__), 'schema.yaml'))))
Traceback (most recent call last):
  File "/usr/local/python27/bin/elastalert", line 11, in <module>
    load_entry_point('elastalert==0.1.39', 'console_scripts', 'elastalert')()
  File "/usr/local/python27/lib/python2.7/site-packages/elastalert/elastalert.py", line 1925, in main
    client.start()
  File "/usr/local/python27/lib/python2.7/site-packages/elastalert/elastalert.py", line 1106, in start
    self.run_all_rules()
  File "/usr/local/python27/lib/python2.7/site-packages/elastalert/elastalert.py", line 1158, in run_all_rules
    self.send_pending_alerts()
  File "/usr/local/python27/lib/python2.7/site-packages/elastalert/elastalert.py", line 1534, in send_pending_alerts
    pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
  File "/usr/local/python27/lib/python2.7/site-packages/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
    size=1000)
  File "/usr/local/python27/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
@rajujha373

This comment has been minimized.

Copy link

commented Apr 12, 2019

try pip uninstall elasticsearch
and then pip install elasticsearch==5.5.3

@hollowimage

This comment has been minimized.

Copy link

commented Apr 12, 2019

this is because of #2203
es pip package 7.0.0 went live yesterday

@cellango

This comment has been minimized.

Copy link

commented Apr 14, 2019

I am using the 6.7.0 elastic stack. Cannot migrate to 7.0.0 since elastalert kibana plugin is not available. I am seeing elastalert | File "/opt/elastalert/elastalert/elastalert.py", line 1528, in send_pending_alerts
elastalert | pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
elastalert | File "/opt/elastalert/elastalert/elastalert.py", line 1520, in find_recent_pending_alerts
elastalert |
elastalert | 15:20:39.834Z ERROR elastalert-server:
elastalert | ProcessController: size=1000)
elastalert | File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.0-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
elastalert | return func(*args, params=params, **kwargs)
elastalert | TypeError: search() got an unexpected keyword argument 'doc_type'

Cannot go to the 5.5.3.

What are my options?

@caleb15

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2019

For me doing the following fixed it:

pip install elasticsearch==6.3.1
pip uninstall elastalert
pip install elastalert

VeselaHouba pushed a commit to VeselaHouba/log-management that referenced this issue Apr 15, 2019

@remotephone

This comment has been minimized.

Copy link

commented Apr 15, 2019

From the elasticsearch pypi page:

Compatibility
The library is compatible with all Elasticsearch versions since 0.90.x but you have to use a matching major version:

For Elasticsearch 6.0 and later, use the major version 6 (6.x.y) of the library.

For Elasticsearch 5.0 and later, use the major version 5 (5.x.y) of the library.

For Elasticsearch 2.0 and later, use the major version 2 (2.x.y) of the library, and so on.

If you install an incompatible version it won't work, so make sure you pick the right one.

@wellbastos

This comment has been minimized.

Copy link

commented Apr 16, 2019

try pip uninstall elasticsearch
and then pip install elasticsearch==5.5.3
solved my problem

@johnjbateman

This comment has been minimized.

Copy link

commented Apr 30, 2019

Hi, I am getting this as well using a containerized version 0.1.38. I see there is 0.1.39 from 3/19. Is this fixed there too?

19:41:27.410Z ERROR elastalert-server:
    ProcessController:  Traceback (most recent call last):

19:41:27.411Z ERROR elastalert-server:
    ProcessController:    File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main

19:41:27.412Z ERROR elastalert-server: ProcessController:
19:41:27.412Z ERROR elastalert-server:
    ProcessController:  "__main__", fname, loader, pkg_name)

19:41:27.413Z ERROR elastalert-server:
    ProcessController:    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code

19:41:27.414Z ERROR elastalert-server:
    ProcessController:      exec code in run_globals
      File "/opt/elastalert/elastalert/elastalert.py", line 1923, in <module>

19:41:27.415Z ERROR elastalert-server:
    ProcessController:      sys.exit(main(sys.argv[1:]))

19:41:27.415Z ERROR elastalert-server:
    ProcessController:    File "/opt/elastalert/elastalert/elastalert.py", line 1919, in main

19:41:27.416Z ERROR elastalert-server:
    ProcessController:      client.start()
      File "/opt/elastalert/elastalert/elastalert.py", line 1100, in start

19:41:27.416Z ERROR elastalert-server:
    ProcessController:      self.run_all_rules()
      File "/opt/elastalert/elastalert/elastalert.py", line 1152, in run_all_rules

19:41:27.417Z ERROR elastalert-server: ProcessController:
19:41:27.418Z ERROR elastalert-server:
    ProcessController:  self.send_pending_alerts()

19:41:27.418Z ERROR elastalert-server:
    ProcessController:    File "/opt/elastalert/elastalert/elastalert.py", line 1528, in send_pending_alerts

19:41:27.419Z ERROR elastalert-server:
    ProcessController:      pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
      File "/opt/elastalert/elastalert/elastalert.py", line 1520, in find_recent_pending_alerts

19:41:27.419Z ERROR elastalert-server:
    ProcessController:      size=1000)
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.0-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
    TypeError: search() got an unexpected keyword argument 'doc_type'

19:41:27.455Z ERROR elastalert-server: ProcessController:  ElastAlert exited with code 1
@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 7, 2019

Hello,
i am getting the following error after running test rule. It is able to load the test rule successfully but get an TypeError '_source_includes'.

Thanks in advance
Max

cloud@elk-stack-ubuntu-16:~/elastalert$ elastalert-test-rule example_rules/example_frequency.yaml
/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(__file__), 'schema.yaml'))))
/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/test_rule.py:343: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  conf = yaml.load(fh)
Successfully loaded Example frequency rule

Traceback (most recent call last):
  File "/home/cloud/.local/bin/elastalert-test-rule", line 11, in <module>
    load_entry_point('elastalert==0.1.39', 'console_scripts', 'elastalert-test-rule')()
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/test_rule.py", line 463, in main
    test_instance.run_rule_test()
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/test_rule.py", line 455, in run_rule_test
    self.run_elastalert(rule_yaml, conf, args)
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/test_rule.py", line 311, in run_elastalert
    client.run_rule(rule, endtime, starttime)
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 847, in run_rule
    if not self.run_query(rule, rule['starttime'], tmp_endtime):
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 598, in run_query
    data = self.get_hits(rule, start, end, index, scroll)
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 351, in get_hits
    **extra_args
  File "/home/cloud/.local/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument '_source_includes'
@johnjbateman

This comment has been minimized.

Copy link

commented May 7, 2019

I was able to fix my issue for v0.1.39. I use this dockerfile:

FROM alpine:latest as py-ea
ARG ELASTALERT_VERSION=v0.1.39
ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
# URL from which to download Elastalert.
ARG ELASTALERT_URL=https://github.com/Yelp/elastalert/archive/$ELASTALERT_VERSION.zip
ENV ELASTALERT_URL=${ELASTALERT_URL}
# Elastalert home directory full path.
ENV ELASTALERT_HOME /opt/elastalert

RUN apk update && \
    apk add ca-certificates

WORKDIR /opt

RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \
# Download and unpack Elastalert.
    wget -O elastalert.zip "${ELASTALERT_URL}" && \
    unzip elastalert.zip && \
    rm elastalert.zip && \
    mv e* "${ELASTALERT_HOME}"

WORKDIR "${ELASTALERT_HOME}"

# Copy requirements.txt with specific elasticsearch version
RUN rm requirements.txt
COPY requirements.txt requirements.txt

# Install Elastalert.
# see: https://github.com/Yelp/elastalert/issues/1654
RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && \
    python setup.py install && \
    pip install -r requirements.txt

FROM node:alpine
LABEL maintainer="BitSensor <dev@bitsensor.io>"
# Set timezone for this container
ENV TZ Etc/UTC

RUN apk add --update --no-cache curl tzdata python2 make libmagic

COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages
COPY --from=py-ea /opt/elastalert /opt/elastalert
COPY --from=py-ea /usr/bin/elastalert* /usr/bin/

WORKDIR /opt/elastalert-server
COPY . /opt/elastalert-server

RUN npm install --production --quiet
COPY config/elastalert.yaml /opt/elastalert/config.yaml
COPY config/elastalert-test.yaml /opt/elastalert/config-test.yaml
COPY config/smtp_auth.yaml /opt/elastalert/smtp_auth.yaml
COPY config/config.json config/config.json
COPY rule_templates/ /opt/elastalert/rule_templates
COPY elastalert_modules/ /opt/elastalert/elastalert_modules

EXPOSE 3030
ENTRYPOINT ["npm", "start"]

As you can see above, after downloading the release I desire, I remove the packaged requirements.txt that specified the latest version of elasticsearch and committed a version of my own to my repo that specified the exact version I wanted. In my case:

elasticsearch==6.3.1

@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 8, 2019

Hi John,

thanks for reply. So i am currently using the following versions:

elastalert 0.1.39
elasticsearch-py 6.3.1
elasticsearch on ELK 6.6.2

What I did to install elastalert was the following:

  1. Clone git repo
  2. Changed the setup.py from elasticsearch>=7.0.0 to elasticsearch==6.3.1
  3. Ran the setup.py --> was successful
  4. Create the indices --> was successful
    5.1 test the example rule --> TypeError: search() got an unexpected keyword argument '_source_includes'
    5.2 test another rule --> same error appears (TypeError: search() got an unexpected keyword argument '_source_includes')
  5. Ran elastalert --> it works, but the rule did never match, probably because of the above error

Unfortunately I dont know anymore what to do else now....

@matsgoran

This comment has been minimized.

Copy link
Contributor

commented May 8, 2019

@MaximilianKaltner Have you tried using 0.2.0b2?
Ref. #2202.

@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 8, 2019

Hi @matsgoran,

thanks for your advice.

Is it necessary to upgrade elasticsearch to 7.0.0 for elastalert 0.2.0b2?

@matsgoran

This comment has been minimized.

Copy link
Contributor

commented May 8, 2019

@MaximilianKaltner No it should not. 0.2.0b2 has some specific fixes related to API changes starting with Elasticsearch 6.6. (#2194)

@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 8, 2019

Perfect, i will check this out and give feedback to you!

Thanks a lot

@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 8, 2019

Hi @matsgoran,

installation and creating indices was no problem with elastalert 0.2.0b2.

Unfortunately I am still getting errors during command elastalert-test-rule. It looks like that he is expecting 'elastalert==0.1.39' distribution, which was not found:

cloud@elk-stack-ubuntu-16:~$ sudo elastalert-test-rule --config /etc/elastalert/config.yaml --rule /etc/elastalert/rules/test.yaml
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/cloud/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/home/cloud/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/home/cloud/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/cloud/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/cloud/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/cloud/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'elastalert==0.1.39' distribution was not found and is required by the application

Running elastalert with this rule look like that it is working fine, found some hits and matches, but gives another error message, that it is not able to connect or sent alert to ms_teams via webhook:

Ccloud@elk-stack-ubuntu-16:~sudo python -m elastalert.elastalert --verbose --config /etc/elastalert/config.yaml  --rule /etc/elastalert/rules/test.yaml
/home/cloud/.local/lib/python2.7/site-packages/elastalert/loaders.py:93: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  yaml.load(open(os.path.join(os.path.dirname(__file__), 'schema.yaml'))))
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Sleeping for 59.999966 seconds
INFO:elastalert:Queried rule Test rule from 2019-05-08 10:47 CEST to 2019-05-08 10:47 CEST: 0 / 0 hits
INFO:elastalert:Ran Test rule from 2019-05-08 10:47 CEST to 2019-05-08 10:47 CEST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Test rule from 2019-05-08 10:47 CEST to 2019-05-08 10:48 CEST: 0 / 0 hits
INFO:elastalert:Ran Test rule from 2019-05-08 10:47 CEST to 2019-05-08 10:48 CEST: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Background configuration change check run at 2019-05-08 10:48 CEST
ERROR:root:Error while running alert ms_teams: Error posting to ms teams: HTTPSConnectionPool(host='outlook.office.com', port=443): Max retries exceeded with url: /webhook/66ddb43b-b497-487a-8e99-87de72eeaa58@ce849bab-cc1c-465b-b62e-18f07c9ac198/IncomingWebhook/0639a735c2e1428a82524b0d51a80b55/49bccbaf-7164-4535-8434-e3cf4f091e7d (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6e72c08d90>: Failed to establish a new connection: [Errno -2] Name or service not known',))
INFO:elastalert:Sleeping for 59.99998 seconds
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2019-05-08 10:50 CEST
INFO:elastalert:Queried rule Test rule from 2019-05-08 10:47 CEST to 2019-05-08 10:50 CEST: 5 / 5 hits
ERROR:root:Error while running alert ms_teams: Error posting to ms teams: HTTPSConnectionPool(host='outlook.office.com', port=443): Max retries exceeded with url: /webhook/66ddb43b-b497-487a-8e99-87de72eeaa58@ce849bab-cc1c-465b-b62e-18f07c9ac198/IncomingWebhook/0639a735c2e1428a82524b0d51a80b55/49bccbaf-7164-4535-8434-e3cf4f091e7d (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6e72c16a50>: Failed to establish a new connection: [Errno -2] Name or service not known',))
INFO:elastalert:Ran Test rule from 2019-05-08 10:47 CEST to 2019-05-08 10:50 CEST: 5 query hits (0 already seen), 1 matches, 0 alerts sent
INFO:elastalert:Background configuration change check run at 2019-05-08 10:50 CEST
ERROR:root:Error while running alert ms_teams: Error posting to ms teams: HTTPSConnectionPool(host='outlook.office.com', port=443): Max retries exceeded with url: /webhook/66ddb43b-b497-487a-8e99-87de72eeaa58@ce849bab-cc1c-465b-b62e-18f07c9ac198/IncomingWebhook/0639a735c2e1428a82524b0d51a80b55/49bccbaf-7164-4535-8434-e3cf4f091e7d (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6e72c4f4d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Thanks for your support again!

@matsgoran

This comment has been minimized.

Copy link
Contributor

commented May 8, 2019

@MaximilianKaltner Did you uninstall elastalert 0.1.39 using pip uninstall? To me it seems like you are executing a dangling 0.1.39 console script.

You could always try to execute the elastalert-test rule in a similar fashion as you did with elastalert thus circumventing the script entirely:
python -m elastalert.test-rule --config /etc/elastalert/config.yaml /etc/elastalert/rules/test.yaml

FYI: The test-rule in 0.2.0b2 contains some bugs that seems fixed in the master branch. Especially an assertion error caused by buggy argument parsing.

Traceback (most recent call last):
  File "/home/m/Documents/elastalert/venv/bin/elastalert-test-rule", line 11, in <module>
    load_entry_point('elastalert==0.2.0b2', 'console_scripts', 'elastalert-test-rule')()
  File "/home/m/Documents/elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/test_rule.py", line 441, in main
    test_instance.run_rule_test()
  File "/home/m/Documents/elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/test_rule.py", line 408, in run_rule_test
    conf = load_conf(args, defaults, overwrites)
  File "/home/m/Documents/elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/config.py", line 47, in load_conf
    configure_logging(args, conf)
  File "/home/m/Documents/elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/config.py", line 106, in configure_logging
    if args.verbose and args.debug:
AttributeError: 'Namespace' object has no attribute 'verbose'
@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 9, 2019

Hi @matsgoran
thanks for your reply.

Correct, I did uninstall using pip. Trying to execute the testrule-script directly with python is not working, i dont understand why :(

cloud@elk-stack-ubuntu-16:/usr/local/bin$ ls
chardetect        elastalert                   elastalert-test-rule  jsonschema   rst2html4.py  rst2latex.py           rst2odt.py        rst2xetex.py    stomp
easy_install      elastalert-create-index      jirashell             pbr          rst2html5.py  rst2man.py             rst2pseudoxml.py  rst2xml.py
easy_install-3.5  elastalert-rule-from-kibana  jp.py                 __pycache__  rst2html.py   rst2odt_prepstyles.py  rst2s5.py         rstpep2html.py
cloud@elk-stack-ubuntu-16:/usr/local/bin$ python -m elastalert-test-rule  --config /etc/elastalert/config.yaml --rule /etc/elastalert/rules/test.yaml
/usr/bin/python: No module named elastalert-test-rule
cloud@elk-stack-ubuntu-16:/usr/local/bin$

elasticsearch-py 7.0.0 is correct to use with the latest elastalert version and also compatible to elasticsearch 6.6.2 on ELK?

@matsgoran

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

@MaximilianKaltner Seems like you have two typos in your command (elastalert-test-rule and --rule). The command should be python -m elastalert.test_rule --config /etc/elastalert/config.yaml /etc/elastalert/rules/test.yaml

@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 9, 2019

@matsgoran Correct, sorry - now I get the mentioned issue. How can I get the master branch for fixed version?

cloud@elk-stack-ubuntu-16:/usr/local/bin$ ls
chardetect        elastalert                   elastalert-test-rule  jsonschema   rst2html4.py  rst2latex.py           rst2odt.py        rst2xetex.py    stomp
easy_install      elastalert-create-index      jirashell             pbr          rst2html5.py  rst2man.py             rst2pseudoxml.py  rst2xml.py
easy_install-3.5  elastalert-rule-from-kibana  jp.py                 __pycache__  rst2html.py   rst2odt_prepstyles.py  rst2s5.py         rstpep2html.py
cloud@elk-stack-ubuntu-16:/usr/local/bin$ python -m elastalert.test_rule --config /etc/elastalert/config.yaml /etc/elastalert/rules/test.yaml
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert/test_rule.py", line 445, in <module>
    main()
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert/test_rule.py", line 441, in main
    test_instance.run_rule_test()
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert/test_rule.py", line 408, in run_rule_test
    conf = load_conf(args, defaults, overwrites)
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert/config.py", line 47, in load_conf
    configure_logging(args, conf)
  File "/home/cloud/.local/lib/python2.7/site-packages/elastalert/config.py", line 106, in configure_logging
    if args.verbose and args.debug:
AttributeError: 'Namespace' object has no attribute 'verbose'
@matsgoran

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

@MaximilianKaltner you could simply do a checkout of the git repository. Run the code directly from your local copy or build the python package via setup-tools and install it using pip

@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 9, 2019

@matsgoran
Sorry, I am not really familiar with that. Where can i clone the repo with version 0.2.0b2. I just know the repo https://github.com/Yelp/elastalert.git with version 0.1.39 to clone.

@matsgoran

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

@MaximilianKaltner The changes in 0.2.0b2 have been merged into the master branch.

git clone https://github.com/Yelp/elastalert.git;
cd elastalert;
#You should already be on the master branch, but just in case
git checkout master; 
# Now you can run your 'python -m commands'
# This is how you could build and install the package
#If you would like to roll your own version change the version in setup.py
python setup.py install (--user);
#If you later would like to uninstall it
pip uninstall elastalert;
@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 9, 2019

Cool - thank you so much @matsgoran ! Testing the rule is working fine now :)

cloud@elk-stack-ubuntu-16:~/elastalert$ python -m elastalert.test_rule --config /etc/elastalert/config.yaml /etc/elastalert/rules/test.yaml
elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(__file__), 'schema.yaml'))))
/home/cloud/elastalert/elastalert/test_rule.py:339: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  conf = yaml.load(fh)
Successfully loaded Test rule

INFO:elastalert:Queried rule Test rule from 2019-05-09 09:08 CEST to 2019-05-09 09:11 CEST: 5 / 5 hits
INFO:elastalert:Alert for Test rule at 2019-05-09T07:10:00.993Z:
INFO:elastalert:Test rule

At least 5 events occurred between 2019-05-09 09:07 CEST and 2019-05-09 09:10 CEST

@timestamp: 2019-05-09T07:10:00.993Z
@version: 1
_id: settm2oBQGI8AYHdbYuH
_index: futuremap_db_size-2019.05
_type: doc
environment: PROD
extendible_gb: 0
free_gb: 10.5
num_hits: 5
num_matches: 1
pct_gb_used: 78.43
recycle_gb: 0
tablespace_name: ORAUSER
total_gb: 48.8
used_gb: 38.3


Would have written the following documents to writeback index (default is elastalert_status):

silence - {'rule_name': 'Test rule', '@timestamp': datetime.datetime(2019, 5, 9, 7, 11, 46, 710966, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2019, 5, 9, 7, 12, 46, 710957, tzinfo=tzutc())}

elastalert_status - {'hits': 5, 'matches': 1, '@timestamp': datetime.datetime(2019, 5, 9, 7, 11, 46, 712790, tzinfo=tzutc()), 'rule_name': 'Test rule', 'starttime': datetime.datetime(2019, 5, 9, 7, 8, 44, 892793, tzinfo=tzutc()), 'endtime': datetime.datetime(2019, 5, 9, 7, 11, 46, 692793, tzinfo=tzutc()), 'time_taken': 0.013704061508178711}

Do you also have an idea for my (hopefully) last problem? It looks like that i am not able to sent alerts to ms_teams

cloud@elk-stack-ubuntu-16:~/elastalert$ sudo python -m elastalert.elastalert --verbose --config /etc/elastalert/config.yaml  --rule /etc/elastalert/rules/test.yaml
elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(__file__), 'schema.yaml'))))
INFO:elastalert:Starting up
INFO:elastalert:Queried rule Test rule1 from 2019-05-09 09:07 CEST to 2019-05-09 09:22 CEST: 15 / 15 hits
ERROR:root:Error while running alert ms_teams: Error posting to ms teams: HTTPSConnectionPool(host='outlook.office.com', port=443): Max retries exceeded with url: /webhook/66ddb43b-b497-487a-8e99-87de72eeaa58@ce849bab-cc1c-465b-b62e-18f07c9ac198/IncomingWebhook/0639a735c2e1428a82524b0d51a80b55/49bccbaf-7164-4535-8434-e3cf4f091e7d (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb4276fb350>: Failed to establish a new connection: [Errno -2] Name or service not known',))
INFO:elastalert:Ignoring match for silenced rule Test rule1
INFO:elastalert:Ignoring match for silenced rule Test rule1
INFO:elastalert:Ran Test rule1 from 2019-05-09 09:07 CEST to 2019-05-09 09:22 CEST: 15 query hits (0 already seen), 3 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.47874 seconds
ERROR:root:Error while running alert ms_teams: Error posting to ms teams: HTTPSConnectionPool(host='outlook.office.com', port=443): Max retries exceeded with url: /webhook/66ddb43b-b497-487a-8e99-87de72eeaa58@ce849bab-cc1c-465b-b62e-18f07c9ac198/IncomingWebhook/0639a735c2e1428a82524b0d51a80b55/49bccbaf-7164-4535-8434-e3cf4f091e7d (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb4275ec450>: Failed to establish a new connection: [Errno -2] Name or service not known',))
INFO:elastalert:Queried rule Test rule1 from 2019-05-09 09:08 CEST to 2019-05-09 09:23 CEST: 15 / 15 hits
INFO:elastalert:Ran Test rule1 from 2019-05-09 09:08 CEST to 2019-05-09 09:23 CEST: 15 query hits (15 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.80434 seconds

My rule looks like:

es_host: 10.135.128.15
es_port: 9200
name: Test rule1
type: frequency
index: futuremap_db_size*
num_events: 5
timeframe:
    minutes: 3
alert:
- "ms_teams"
ms_teams_webhook_url:
-  "https://outlook.office.com/webhook/66ddb43b-b497-487a-8e99-87de72eeaa58@ce849bab-cc1c-465b-b62e-18f07c9ac198/IncomingWebhook/0639a735c2e1428a82524b0d51a80b55/49bccbaf-7164-4535-8434-e3cf4f091e7d"
ms_teams_alert_summary: "Test"
@MaximilianKaltner

This comment has been minimized.

Copy link

commented May 9, 2019

@matsgoran I fixed the problem with ms-teams. I missed the proxy configuration in my rule!

ms_teams_proxy: By default ElastAlert will not use a network proxy to send notifications to MS Teams. Set this option using hostname:port if you need to use a proxy.

Now everything works - thanks again!

cloud@elk-stack-ubuntu-16:~/elastalert$ sudo python -m elastalert.elastalert --verbose --config /etc/elastalert/config.yaml  --rule /etc/elastalert/rules/test.yaml
elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(__file__), 'schema.yaml'))))
INFO:elastalert:Starting up
INFO:elastalert:Alert sent to MS Teams
INFO:elastalert:Alert sent to MS Teams
INFO:elastalert:Alert sent to MS Teams
INFO:elastalert:Alert sent to MS Teams
INFO:elastalert:Queried rule Test rule1 from 2019-05-09 09:49 CEST to 2019-05-09 09:50 CEST: 5 / 5 hits
INFO:elastalert:Alert sent to MS Teams
INFO:elastalert:Ran Test rule1 from 2019-05-09 09:49 CEST to 2019-05-09 09:50 CEST: 5 query hits (0 already seen), 1 matches, 5 alerts sent
INFO:elastalert:Sleeping for 56.70884 seconds
INFO:elastalert:Queried rule Test rule1 from 2019-05-09 09:49 CEST to 2019-05-09 09:51 CEST: 5 / 5 hits
INFO:elastalert:Ran Test rule1 from 2019-05-09 09:49 CEST to 2019-05-09 09:51 CEST: 5 query hits (5 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.950538 seconds
@ryanpersaud

This comment has been minimized.

Copy link

commented May 20, 2019

Thanks to @johnjbateman for mentioning a version of elasticsearch that worked with 0.1.38. I have a docker image that pulls down the elastalert source and builds it, so I used sed to replace elasticsearch in setup.py with a specific version (the same could be done in requirements.txt):
sed -i 's/elasticsearch/elasticsearch==6.3.1/' /opt/elastalert/setup.py

@johnjbateman

This comment has been minimized.

Copy link

commented May 21, 2019

@ryanpersaud Nice! I am changing my dockerfile to use sed instead of what I did with committing the requirements.txt. Thanks for the suggestion!

@pietrolatorre

This comment has been minimized.

Copy link

commented May 24, 2019

Hi!
I edited the dockerfile from here
and put:

RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py &&
sed -i 's/elasticsearch>=7.0.0/elasticsearch==6.3.1/g' setup.py

RUN pip install urllib3==1.24.3

It's working now!

@johnjbateman

This comment has been minimized.

Copy link

commented May 24, 2019

@pietrolatorre Nice call out there on urllib3. I also struggled through that.

Couple notes:

  • They fixed the jira issue at some point, so I removed that from my dockerfile.
  • Instead of running pip you can also do this to solve the urllib3 issue:
# Install Elastalert.
RUN sed -i 's/elasticsearch/elasticsearch==6.3.1/' /opt/elastalert/setup.py && \
    sed -i 's/aws-requests-auth>=0.3.0/aws-requests-auth==0.3.0/' /opt/elastalert/setup.py && \
    sed -i 's/boto3>=1.4.4/boto3==1.4.4/' /opt/elastalert/setup.py && \
    python setup.py install && \
    pip install -r requirements.txt

I did this because I don't use aws, but might not work for others.

@ryanpersaud

This comment has been minimized.

Copy link

commented May 24, 2019

@pietrolatorre I had to do the same thing as well (manually install urllib3). I was able to use 1.22.

@invizus

This comment has been minimized.

Copy link
Contributor

commented May 29, 2019

Had same issue, fixed:

pip uninstall elasticsearch
pip install "elasticsearch<7"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.