Committime exporter failing to return any data because of one build.

[etsauer]

I have the committime exporter deployed in a cluster, returning the following error in the logs:

INFO:root:Namespace: kenwilli-basic-spring-boot-build, App: kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d, Build: kenwilli-basic-spring-boot-3
Traceback (most recent call last):
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/opt/app-root/lib/python3.6/site-packages/prometheus_client/exposition.py", line 52, in prometheus_app
    status, header, output = _bake_output(registry, accept_header, params)
  File "/opt/app-root/lib/python3.6/site-packages/prometheus_client/exposition.py", line 40, in _bake_output
    output = encoder(registry)
  File "/opt/app-root/lib/python3.6/site-packages/prometheus_client/openmetrics/exposition.py", line 56, in generate_latest
    floatToGoString(s.value),
  File "/opt/app-root/lib/python3.6/site-packages/prometheus_client/utils.py", line 9, in floatToGoString
    d = float(d)
TypeError: ("float() argument must be a string or a number, not 'NoneType'", Metric(github_commit_timestamp, Commit timestamp, gauge, , [Sample(name='github_commit_timestamp', labels={'namespace': 'basic-nginx-build', 'app': 'basic-nginx-d5959a2450fa575f95e27a77951ee296319ecfae', 'image_sha': 'sha256:b662b493c37a2f0810e6c96268a46bda0364bb5e93f0d7c672ea8fd20966da2e'}, value=1591804346.0, timestamp=None, exemplar=None), Sample(name='github_commit_timestamp', labels={'namespace': 'basic-nginx-build', 'app': 'basic-nginx-d5959a2450fa575f95e27a77951ee296319ecfae', 'image_sha': 'sha256:5818ec7641d40144628c2537914d2874167c6edf64cfefa8bda89f8b525a36b6'}, value=1591804346.0, timestamp=None, exemplar=None), Sample(name='github_commit_timestamp', labels={'namespace': 'basic-nginx-build', 'app': 'basic-nginx-d5959a2450fa575f95e27a77951ee296319ecfae', 'image_sha': 'sha256:2e7efc7c011eef8795f2d73fe37b0ce71198aeb62eca97de49dd2379d8855885'}, value=1591804346.0, timestamp=None, exemplar=None), Sample(name='github_commit_timestamp', labels={'namespace': 'basic-nginx-build', 'app': 'basic-nginx-d5959a2450fa575f95e27a77951ee296319ecfae', 'image_sha': 'sha256:2f934e61518d81b9decd423ec8ea88f06e379281df860d120f564721328b7ae3'}, value=1591804346.0, timestamp=None, exemplar=None), Sample(name='github_commit_timestamp', labels={'namespace': 'basic-nginx-build', 'app': 'basic-nginx-d5959a2450fa575f95e27a77951ee296319ecfae', 'image_sha': 'sha256:fbcb7bd0acf00c0f012b09e7fbab09c445e3ff71ca96e5bf2625970751029777'}, value=1591804346.0, timestamp=None, exemplar=None), Sample(name='github_commit_timestamp', labels={'namespace': 'kenwilli-basic-spring-boot-build', 'app': 'kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d', 'image_sha': 'sha256:46332b417660900361f7185830a4eb6d5ddc7e3002944ba26ed260e83f415197'}, value=None, timestamp=None, exemplar=None)]))

Because it this one error, the exporter won't return ANY data. If you hit the endpoint for the exporter, it simply returns:

$ curl http://committime-exporter-pelorus-etsauer.apps.cluster-eric.blue.osp.opentlc.com/
A server error occurred.  Please contact the administrator.

We need to do the following:

1. Figure out why this build is triggering NoneType error (it shouldn't be)
2. Find a way to ensure this type of error can be handled in the future, and just skip the broken build instead of failing to return data.

[etsauer]

The issue here is that this build is returning a value=None, which shouldn't happen:

Sample(name='github_commit_timestamp', labels={'namespace': 'kenwilli-basic-spring-boot-build', 'app': 'kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d', 'image_sha': 'sha256:46332b417660900361f7185830a4eb6d5ddc7e3002944ba26ed260e83f415197'}, value=None, timestamp=None, exemplar=None)

[etsauer]

Interestingly, this looks to be an issue that ONLY occurs when the exporter receives a request.

When I run the exporter locally, the first time it starts up, and runs through collection, it doesn't hit this issue.

$ python exporters/committime/app.py
...
INFO:root:Namespace: kenwilli-basic-spring-boot-build, App: kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d, Build: kenwilli-basic-spring-boot-3
INFO:root:Namespace: kenwilli-basic-spring-boot-build, App: kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d, Build: kenwilli-basic-spring-boot-4
...

but then when I curl localhost:8080/, I get an error:

$ curl http://localhost:8080
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
        "http://www.w3.org/TR/html4/strict.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
        <title>Error response</title>
    </head>
    <body>
        <h1>Error response</h1>
        <p>Error code: 500</p>
        <p>Message: error generating metric output.</p>
        <p>Error code explanation: 500 - Server got itself in trouble.</p>
    </body>
</html>

And in the logs:

INFO:root:Namespace: basic-nginx-build, App: basic-nginx-c25aa93fb38da038e4e6c1decb6b0a965acd14d2, Build: basic-nginx-1
INFO:root:Namespace: kenwilli-basic-spring-boot-build, App: kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d, Build: kenwilli-basic-spring-boot-3
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 50312)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib64/python3.7/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python3.7/socketserver.py", line 720, in __init__
    self.handle()
  File "/usr/lib64/python3.7/http/server.py", line 426, in handle
    self.handle_one_request()
  File "/usr/lib64/python3.7/http/server.py", line 414, in handle_one_request
    method()
  File "/home/esauer/.local/lib/python3.7/site-packages/prometheus_client/exposition.py", line 152, in do_GET
    output = encoder(registry)
  File "/home/esauer/.local/lib/python3.7/site-packages/prometheus_client/exposition.py", line 121, in generate_latest
    output.append(sample_line(s))
  File "/home/esauer/.local/lib/python3.7/site-packages/prometheus_client/exposition.py", line 87, in sample_line
    line.name, labelstr, floatToGoString(line.value), timestamp)
  File "/home/esauer/.local/lib/python3.7/site-packages/prometheus_client/utils.py", line 8, in floatToGoString
    d = float(d)
TypeError: ("float() argument must be a string or a number, not 'NoneType'", Metric(github_commit_timestamp, Commit timestamp, gauge, , [Sample(name='github_commit_timestamp', labels={'namespace': 'basic-nginx-build', 'app': 'basic-nginx-c25aa93fb38da038e4e6c1decb6b0a965acd14d2', 'image_sha': 'sha256:0b551f831ff7e506aea7453ae7439f21f348e74d63081d0350c9f4ff86af53d1'}, value=1591816927.0, timestamp=None, exemplar=None), Sample(name='github_commit_timestamp', labels={'namespace': 'kenwilli-basic-spring-boot-build', 'app': 'kenwilli-basic-spring-boot-ea67853d8f9b4f43400500f807d72cdfb5b0936d', 'image_sha': 'sha256:46332b417660900361f7185830a4eb6d5ddc7e3002944ba26ed260e83f415197'}, value=None, timestamp=None, exemplar=None)]))
----------------------------------------