On query failure metric is not removed

[jamiguet]

I have a terms query setup via an alias.
I launch the exporter and all is good, the metric is there and then I get a doc count from it.

I change the index associated with the alias and I see the following error in the logs:

  File "/usr/src/app/prometheus_es_exporter/__init__.py", line 115, in run_query
    response = es_client.search(index=indices, body=query, request_timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 660, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 186, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query:

The terms query count does not disappear I can then not react to it with an is absent the last value remains in place.
I have not look at the code but it looks like the exception is not correctly handled to update the metrics exported.

[braedon]

Hi @jamiguet, can you confirm why the exception is actually being thrown? Is the alias no longer pointing to an index? Or is the query not valid in the new index somehow? Or something else?

Note that this PR is looking at one aspect of how to deal with metrics after query failures: #34 I feel like the solutions might be related.

[braedon]

Hi @jamiguet, I've finally gotten around to working on a solution for this. PR #57 updates the exporter to drop metrics produced by a query if that query throws an exception. It can also be configured to preserve the old values instead (what it used to do), or zero all the metrics for that query.

I hope that fixes your issue here!

[braedon]

The solution has been released in 0.7.0