expected a valid start token, got "\x1f" ("INVALID") while parsing: "\x1f"

[darkmatter18]

What did you do?

1. Using the python client to get the data
2. Using FastAPI and Gunicorn as per the python client docs
3. The data is shown in the browser as like this

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 2.91281e+06
python_gc_objects_collected_total{generation="1"} 595556.0
python_gc_objects_collected_total{generation="2"} 32846.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 4879.0
python_gc_collections_total{generation="1"} 443.0
python_gc_collections_total{generation="2"} 9.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="4",version="3.11.4"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 6.64231936e+08
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 9.9127296e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.6892464761e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 198.38
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 22.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP scaned_user_total Number of Users scanned by whatsapp
# TYPE scaned_user_total counter
scaned_user_total 0.0
# HELP scaned_user_created Number of Users scanned by whatsapp
# TYPE scaned_user_created gauge
scaned_user_created 1.689246484044349e+09
# HELP onboarded_user_total Number of onboarded Users by whatsapp
# TYPE onboarded_user_total counter
onboarded_user_total 0.0

What did you expect to see?

No response

What did you see instead? Under which circumstances?

In Prometheus targets, it is down and showing this error expected a valid start token, got "\x1f" ("INVALID") while parsing: "\x1f"

System information

Docker image quay.io/prometheus/prometheus:v2.45.0

Prometheus version

`quay.io/prometheus/prometheus:v2.45.0`

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[miltlima]

Hello everyone, I have the same problem with same stack (Python + FastAPI)

Python - 3.10
fastAPI - 1.100
Prometheus - 2.45.0

[miltlima]

@darkmatter18 I solved the problem!

You will need to import this Library.

from prometheus_client import make_asgi_app, Counter, Summary, Histogram

And put this code.

metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

[raveenb]

I also faced this issue during dev. I located the issue with some middlewares I was using. In my case the compression middleware brotli from brotli_asgi.

[SergeyFeskov]

You just disabled this compression middleware for metrics endpoint or you found a way to encode this endpoint? I have the same problem with FastAPI's GZipMiddleware

[raveenb]

had disabled the compression middleware. I know, its not the most elegant way considering the amount of data this metrics data sends out. but that is how I got it to work. It may have changed in the newer versions, its been close to an year now.

[SergeyFeskov]

Understood. I came to such solution also, but found another way to solve it. prometheus_client has disable_compression param in make_asgi_app function. It looks so that prometheus_client compresses by itself metrics by default, and another middleware for compression in app makes it another time and looks like Prometheus server decompresses metrics one time and that's why it fails. And setting disable_compression to True solves this problem.

[raveenb]

wow, great find @SergeyFeskov. I will use this param from now. Thanks for getting back on a problem I had forgot. Much appreciated!