exotel dependencies break install and running module

[fabianlee]

The exotel library used in an alerter (introduced into this project Mar 2 2017) has pinned modules dependencies on 'requests' and 'requests-oauthlib'. These dependencies are incompatible with ElastAlert's version of these libraries and is causing errors when trying to run install as well as run the binary.

These errors are not seen if the import and code referring to exotel is commented out from elastalert/alerts.py

• ElastAlert's requirements.txt wants exotel=0.1.1, requests==2.5.1, and requests-oauthlib==0.4.2
• Exotel 0.1.1 wants requests=2.2.1, requests-oauthlib==0.5.0

There are two ways to see this issue:

1. After install, then running 'pip install -r requirements.txt', run '/usr/local/bin/elasticsearch'

Trying to run /usr/local/bin/elastalert results in a stack trace with this as the ending line:
pkg_resources.DistributionNotFound: requests-oauthlib==0.5.0

Even if you upgrade requests-oauthlib to 0.5.0, you still can't satisfy the 'requests' package and get this error when running the binary:
pkg_resources.VersionConflict: (requests 2.5.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('requests==2.2.1'))

2. After install, then running 'pip install -r requirements.txt', try running the installer again 'python setup.py install'

error: Installed distribution requests 2.5.1 conflicts with requirement requests==2.2.1

I've contacted the author of the exotel module about allowing more general dependency checks (>=) in their module, but they have responded that they want ElastAlert to change it's requirements.txt
sarathsp06/exotel-py#3

My very shallow testing has shown mixed success with moving up to the later version of the exotel module and the versions of the dependent modules below. When using pip to install the newer versions in requirements.txt I get gcc compilation errors about yaml.h not being found when trying to build check_libyaml.c. Despite that, /usr/local/bin/elastalert is then able to run successfully, but I'm not sure what other functionality might be affected.
exotel==0.1.3
requests=2.13.0
requests-oauthlib==0.5.0

Again, these errors are not seen if the import and code referring to exotel is commented out from elastalert/alerts.py.

[ardentisys-rajshekhar]

@fabianlee pls try pip install "requests==2.2.1" and similar.
I was facing similar issue. but this solved it check if this one works for you link

[fabianlee]

I do need to install "setuptools>=11.3" before install in order for it to work.

But the mismtach in the "requests" module between this project and exotel continue to be an issue.

[Qmando]

Maybe this behavior is depending on version of pip/site packages/pypi cache or something else, but installing directly from requirements.txt works for me.

$ virtualenv --python=python2.7 new_venv
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in new_venv/bin/python2.7
Also creating executable in new_venv/bin/python
Installing setuptools, pip, wheel...done.
$ source new_venv/bin/activate
$ pip install -r requirements.txt
Collecting argparse==1.3.0 (from -r requirements.txt (line 1))
  Using cached argparse-1.3.0-py2.py3-none-any.whl
Collecting aws-requests-auth==0.2.5 (from -r requirements.txt (line 2))
Collecting blist==1.3.6 (from -r requirements.txt (line 3))
Collecting boto3 (from -r requirements.txt (line 4))
  Using cached boto3-1.4.4-py2.py3-none-any.whl
Collecting configparser>=3.3.0r2 (from -r requirements.txt (line 5))
Collecting croniter==0.3.8 (from -r requirements.txt (line 6))
Collecting cryptography==1.4 (from -r requirements.txt (line 7))
Collecting elasticsearch (from -r requirements.txt (line 8))
  Using cached elasticsearch-5.3.0-py2.py3-none-any.whl
Collecting exotel==0.1.1 (from -r requirements.txt (line 9))
Collecting jira==0.32 (from -r requirements.txt (line 10))
Collecting jsonschema==2.2.0 (from -r requirements.txt (line 11))
Collecting mock==1.0.0 (from -r requirements.txt (line 12))
Collecting oauthlib==0.7.2 (from -r requirements.txt (line 13))
Collecting PyStaticConfiguration==0.9.0 (from -r requirements.txt (line 14))
Collecting python-dateutil==2.4.0 (from -r requirements.txt (line 15))
  Using cached python_dateutil-2.4.0-py2.py3-none-any.whl
Collecting PyYAML==3.11 (from -r requirements.txt (line 16))
Collecting requests==2.5.1 (from -r requirements.txt (line 17))
  Using cached requests-2.5.1-py2.py3-none-any.whl
Collecting requests-oauthlib==0.4.2 (from -r requirements.txt (line 18))
  Using cached requests_oauthlib-0.4.2-py2.py3-none-any.whl
Collecting simplejson==3.3.0 (from -r requirements.txt (line 19))
Collecting six==1.10.0 (from -r requirements.txt (line 20))
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting stomp.py==4.1.15 (from -r requirements.txt (line 21))
  Using cached stomp.py-4.1.15-py2.py3-none-any.whl
Collecting supervisor==3.1.2 (from -r requirements.txt (line 22))
Collecting texttable==0.8.4 (from -r requirements.txt (line 23))
Collecting tlslite==0.4.8 (from -r requirements.txt (line 24))
Collecting twilio==6.0.0 (from -r requirements.txt (line 25))
Collecting unittest2==0.8.0 (from -r requirements.txt (line 26))
  Using cached unittest2-0.8.0-py2.py3-none-any.whl
Collecting urllib3==1.8.2 (from -r requirements.txt (line 27))
Requirement already satisfied (use --upgrade to upgrade): wsgiref==0.1.2 in /usr/lib/python2.7 (from -r requirements.txt (line 28))
Collecting botocore<1.6.0,>=1.5.0 (from boto3->-r requirements.txt (line 4))
  Using cached botocore-1.5.39-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->-r requirements.txt (line 4))
  Using cached jmespath-0.9.2-py2.py3-none-any.whl
Collecting s3transfer<0.2.0,>=0.1.10 (from boto3->-r requirements.txt (line 4))
  Using cached s3transfer-0.1.10-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./new_venv/lib/python2.7/site-packages (from croniter==0.3.8->-r requirements.txt (line 6))
Collecting enum34 (from cryptography==1.4->-r requirements.txt (line 7))
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting ipaddress (from cryptography==1.4->-r requirements.txt (line 7))
  Using cached ipaddress-1.0.18-py2-none-any.whl
Collecting pyasn1>=0.1.8 (from cryptography==1.4->-r requirements.txt (line 7))
  Using cached pyasn1-0.2.3-py2.py3-none-any.whl
Collecting idna>=2.0 (from cryptography==1.4->-r requirements.txt (line 7))
  Using cached idna-2.5-py2.py3-none-any.whl
Collecting cffi>=1.4.1 (from cryptography==1.4->-r requirements.txt (line 7))
Collecting meld3>=0.6.5 (from supervisor==3.1.2->-r requirements.txt (line 22))
  Using cached meld3-1.0.2-py2.py3-none-any.whl
Collecting pytz (from twilio==6.0.0->-r requirements.txt (line 25))
  Using cached pytz-2017.2-py2.py3-none-any.whl
Collecting pyOpenSSL>=0.14 (from twilio==6.0.0->-r requirements.txt (line 25))
  Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl
Collecting PyJWT>=1.4.2 (from twilio==6.0.0->-r requirements.txt (line 25))
  Using cached PyJWT-1.4.2-py2.py3-none-any.whl
Collecting docutils>=0.10 (from botocore<1.6.0,>=1.5.0->boto3->-r requirements.txt (line 4))
  Using cached docutils-0.13.1-py2-none-any.whl
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.2.0,>=0.1.10->boto3->-r requirements.txt (line 4))
  Using cached futures-3.1.1-py2-none-any.whl
Collecting pycparser (from cffi>=1.4.1->cryptography==1.4->-r requirements.txt (line 7))
Installing collected packages: argparse, requests, aws-requests-auth, blist, jmespath, six, python-dateutil, docutils, botocore, futures, s3transfer, boto3, configparser, croniter, enum34, ipaddress, pyasn1, idna, pycparser, cffi, cryptography, urllib3, elasticsearch, oauthlib, requests-oauthlib, exotel, tlslite, jira, jsonschema, mock, PyStaticConfiguration, PyYAML, simplejson, stomp.py, meld3, supervisor, texttable, pytz, pyOpenSSL, PyJWT, twilio, unittest2
Successfully installed PyJWT-1.4.2 PyStaticConfiguration-0.9.0 PyYAML-3.11 argparse-1.3.0 aws-requests-auth-0.2.5 blist-1.3.6 boto3-1.4.4 botocore-1.5.39 cffi-1.10.0 configparser-3.5.0 croniter-0.3.8 cryptography-1.4 docutils-0.13.1 elasticsearch-5.3.0 enum34-1.1.6 exotel-0.1.1 futures-3.1.1 idna-2.5 ipaddress-1.0.18 jira-0.32 jmespath-0.9.2 jsonschema-2.2.0 meld3-1.0.2 mock-1.0.0 oauthlib-0.7.2 pyOpenSSL-16.2.0 pyasn1-0.2.3 pycparser-2.17 python-dateutil-2.4.0 pytz-2017.2 requests-2.5.1 requests-oauthlib-0.4.2 s3transfer-0.1.10 simplejson-3.3.0 six-1.10.0 stomp.py-4.1.15 supervisor-3.1.2 texttable-0.8.4 tlslite-0.4.8 twilio-6.0.0 unittest2-0.8.0 urllib3-1.8.2

If someone knows why this works for me just fine but not for others, I'd love to know.

In either case, I plan on a lot of these alerter libraries optional in the near future to avoid these types of issues for most people.

[fabianlee]

@Qmando - I get the same thing as you up to that point (latest vagrant 'ubuntu/trusty' box), but after that type "$ bin/python setup.py install" and see if you get the following error

error: requests 2.5.1 is installed but requests==2.2.1 is required by set(['exotel'])

[Qmando]

Don't do setup.py install afterwards. Once you run pip, you've installed everything. Setuptools is much stricter when it comes to dependencies. Instead, do pip install -e . to install elastalert itself locally.

[fabianlee]

@Qmando - you are correct that running 'pip install -e' does produce the executable successfully. However, then when trying to then run the application "bin/elastalert" an error is thrown and this is the error:

pkg_resources.DistributionNotFound: The 'requests-oauthlib==0.5.0' distribution was not found and is required by exotel

This happens whether running the sequence of "pip install -r requirements.txt" then "pip install -e ." from a virtualenv (bin/elastalert) or not inside a virtual env (/usr/local/bin/elastalert).

Quick note that when NOT using a virtualenv, I do have to run "pip install -r 'setuptools>=11.3'" before installing requirements.txt, or else I get a compile error about a missing yaml.h.

[sarathsp06]

The exotel version used in elastalert is exotel==0.1.1, while the current version is exotel==0.1.3 . The later version (0.1.3) does not require requests-oauthlib==0.5.0

[nejczupan]

I have same problem with this issue. I have applied your solutions but the problem is still here.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 120: ordinal not in range(128) WARNING:elasticsearch:POST http://192.00.00.00:9200/elastalert/elastalert_status [status:N/A request:0.001s] Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 75, in perform_request timeout=timeout or self.timeout) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 362, in send File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 516, in urlopen File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 308, in _make_request File "/usr/lib/python2.7/httplib.py", line 1039, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 877, in _send_output msg += message_body UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 120: ordinal not in range(128) ERROR:root:Error writing alert info to Elasticsearch: ConnectionError('ascii' codec can't decode byte 0xc5 in position 120: ordinal not in range(128)) caused by: UnicodeDecodeError('ascii' codec can't decode byte 0xc5 in position 120: ordinal not in range(128))

+1

[victorssilva]

It seems to have been changed: https://github.com/sarathsp06/exotel-py/blob/master/requirements.txt