Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Netdata notification integration raises 500 #1180

Closed
decentral1se opened this issue Apr 9, 2020 · 2 comments
Closed

Netdata notification integration raises 500 #1180

decentral1se opened this issue Apr 9, 2020 · 2 comments
Labels
invalid This doesn't seem right

Comments

@decentral1se
Copy link

Issue Summary
When testing from Netdata side, sending an alert to Alerta, I see a 500 on the Alerta logs. The logs I see are the following:

2020-04-09T09:20:04.101163151Z app[web.1]: **************************************** request_id=ee9b35ee-a63c-4a8c-b12b-e98512bd71a2 ip=94.130.105.60
2020-04-09T09:20:04.101166666Z app[web.1]: 
2020-04-09T09:20:04.105140247Z app[web.1]: 2020-04-09 09:20:04,104 DEBG 'uwsgi' stdout output:
2020-04-09T09:20:04.105167165Z app[web.1]: 2020-04-09 09:20:04,104 flask.app[24]: [DEBUG] ****************************************
2020-04-09T09:20:04.105171552Z app[web.1]: 
2020-04-09T09:20:04.105174721Z app[web.1]:             SELECT *
2020-04-09T09:20:04.105178053Z app[web.1]:             FROM blackouts
2020-04-09T09:20:04.105181061Z app[web.1]:             WHERE start_time <= '2020-04-09T09:20:04.104Z' AND end_time > '2020-04-09T09:20:04.104Z'
2020-04-09T09:20:04.105184717Z app[web.1]:               AND environment='foo-internal'
2020-04-09T09:20:04.105187884Z app[web.1]:               AND (
2020-04-09T09:20:04.105199441Z app[web.1]:                  (resource IS NULL AND service='{}' AND event IS NULL AND "group" IS NULL AND tags='{}')
2020-04-09T09:20:04.105202906Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event IS NULL AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105206284Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event IS NULL AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105209649Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event IS NULL AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105212925Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event='test.chart.test_alarm' AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105216190Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event='test.chart.test_alarm' AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105219524Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event='test.chart.test_alarm' AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105222827Z app[web.1]:               OR ( resource IS NULL AND service='{}' AND event='test.chart.test_alarm' AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105226127Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105229463Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105232811Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105236170Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105240416Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105243767Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105247091Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105250481Z app[web.1]:               OR ( resource IS NULL AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105253838Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event IS NULL AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105257049Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event IS NULL AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105260380Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event IS NULL AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105263690Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event IS NULL AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105269682Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event='test.chart.test_alarm' AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105273053Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event='test.chart.test_alarm' AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105276352Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event='test.chart.test_alarm' AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105279625Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service='{}' AND event='test.chart.test_alarm' AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105282934Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105286259Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105289956Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105293319Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event IS NULL AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105296710Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group" IS NULL AND tags='{}' )
2020-04-09T09:20:04.105300046Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group" IS NULL AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105303432Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group"='Performance' AND tags='{}' )
2020-04-09T09:20:04.105306806Z app[web.1]:               OR ( resource='autonomic-dokku:test.family' AND service <@ ARRAY['Netdata'] AND event='test.chart.test_alarm' AND "group"='Performance' AND tags <@ ARRAY['alarm_id:1'] )
2020-04-09T09:20:04.105310213Z app[web.1]:                 )
2020-04-09T09:20:04.105313102Z app[web.1]:         
2020-04-09T09:20:04.105316035Z app[web.1]: **************************************** request_id=ee9b35ee-a63c-4a8c-b12b-e98512bd71a2 ip=94.130.105.60
2020-04-09T09:20:04.105319215Z app[web.1]: 
2020-04-09T09:20:04.111384223Z app[web.1]: 2020-04-09 09:20:04,111 DEBG 'uwsgi' stdout output:
2020-04-09T09:20:04.111422320Z app[web.1]: 2020-04-09 09:20:04,110 flask.app[24]: [ERROR] 'ip' request_id=ee9b35ee-a63c-4a8c-b12b-e98512bd71a2 ip=94.130.105.60
2020-04-09T09:20:04.111427479Z app[web.1]: Traceback (most recent call last):
2020-04-09T09:20:04.111430890Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/utils/api.py", line 40, in process_alert
2020-04-09T09:20:04.111434305Z app[web.1]:     alert = plugin.pre_receive(alert, config=wanted_config)
2020-04-09T09:20:04.111445421Z app[web.1]: TypeError: pre_receive() got an unexpected keyword argument 'config'
2020-04-09T09:20:04.111448766Z app[web.1]: 
2020-04-09T09:20:04.111452231Z app[web.1]: During handling of the above exception, another exception occurred:
2020-04-09T09:20:04.111457596Z app[web.1]: 
2020-04-09T09:20:04.111462084Z app[web.1]: Traceback (most recent call last):
2020-04-09T09:20:04.111465679Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/views/alerts.py", line 51, in receive
2020-04-09T09:20:04.111469105Z app[web.1]:     alert = process_alert(alert)
2020-04-09T09:20:04.111472112Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/utils/api.py", line 42, in process_alert
2020-04-09T09:20:04.111475239Z app[web.1]:     alert = plugin.pre_receive(alert)  # for backward compatibility
2020-04-09T09:20:04.111478248Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta_geoip.py", line 22, in pre_receive
2020-04-09T09:20:04.111481472Z app[web.1]:     ip_addr = alert.attributes['ip'].split(', ')[0]
2020-04-09T09:20:04.111484462Z app[web.1]: KeyError: 'ip'
2020-04-09T09:20:04.111487395Z app[web.1]: 
2020-04-09T09:20:04.111490199Z app[web.1]: During handling of the above exception, another exception occurred:
2020-04-09T09:20:04.111493307Z app[web.1]: 
2020-04-09T09:20:04.111496131Z app[web.1]: Traceback (most recent call last):
2020-04-09T09:20:04.111499054Z app[web.1]:   File "/venv/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
2020-04-09T09:20:04.111502299Z app[web.1]:     rv = self.dispatch_request()
2020-04-09T09:20:04.111506609Z app[web.1]:   File "/venv/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
2020-04-09T09:20:04.111511423Z app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2020-04-09T09:20:04.111516128Z app[web.1]:   File "/venv/lib/python3.7/site-packages/flask_cors/decorator.py", line 128, in wrapped_function
2020-04-09T09:20:04.111520915Z app[web.1]:     resp = make_response(f(*args, **kwargs))
2020-04-09T09:20:04.111524083Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/auth/decorators.py", line 53, in wrapped
2020-04-09T09:20:04.111527316Z app[web.1]:     return f(*args, **kwargs)
2020-04-09T09:20:04.111530266Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/models/metrics.py", line 258, in wrapped
2020-04-09T09:20:04.111533410Z app[web.1]:     response = f(*args, **kwargs)
2020-04-09T09:20:04.111536422Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/utils/response.py", line 19, in decorated
2020-04-09T09:20:04.111539526Z app[web.1]:     return func(*args, **kwargs)
2020-04-09T09:20:04.111542430Z app[web.1]:   File "/venv/lib/python3.7/site-packages/alerta/views/alerts.py", line 65, in receive
2020-04-09T09:20:04.111545562Z app[web.1]:     raise ApiError(str(e), 500)
2020-04-09T09:20:04.111548525Z app[web.1]: alerta.exceptions.ApiError: 'ip'

My netdata config is (in health_alarm_config.conf):

ALERTA_WEBHOOK_URL="https://alerta.foo.com/api"
ALERTA_API_KEY="mykey"
DEFAULT_RECIPIENT_ALERTA="foo-internal"

Unsure if this is a netdata or alerta side issue... please let me know and I will pass the buck onto netdata side (and apologies for noise).

Environment

  • OS: Linux

  • API version: 7.4.5

  • Deployment: self-hosted

  • For self-hosted, WSGI environment: nginx/uwsgi

  • Database: Postgres

  • Server config:
    Auth enabled? Yes
    Auth provider? Keycloak
    Customer views? No

  • web UI version: 7.4.5

  • CLI version: 7.4.0

To Reproduce
Steps to reproduce the behavior:

  1. Install Netdata
  2. Follow Netdata docs to configure alerta notifications integration
  3. Run sudo -su netdata bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test

Expected behavior

  • Success reporting of the error

Screenshots

  • N/A

Additional context

  • netdata version v1.21.0
@decentral1se
Copy link
Author

Disabling geoip plugin worked... feel free to close!

@satterly
Copy link
Member

satterly commented Apr 9, 2020

You didn't include your alertad.conf file so I can only guess that you have enabled "geoip" plugin but do not have the "remote_ip" plugin enabled. GeoIP can't lookup an IP address if it does have an IP.

@satterly satterly closed this as completed Apr 9, 2020
@satterly satterly added the invalid This doesn't seem right label Apr 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
invalid This doesn't seem right
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@satterly @decentral1se