You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Deploying prometheus via official helm chart, with additinal sidecar to find metrics endpoints within a subnet, which will generate a json file with targets
What did you expect to see?
prometheus reading this file_sd file
What did you see instead? Under which circumstances?
...
ts=2023-05-22T08:53:29.253Z caller=file.go:343 level=error component="discovery manager scrape" discovery=file
config=network_discovery msg="Error reading file" path=/discovery/direct-node.json err="unexpected end of JSON input"
...
The issue you're experiencing can happen if Prometheus attempts to read the JSON file while it's in the middle of being written or updated. If the file isn't completely written, this can lead to an "unexpected end of JSON input" error, as you're seeing in your logs.
To avoid this situation, you should ensure that the JSON file is written atomically. This can be achieved by writing the new contents to a temporary file and then renaming the temporary file to the desired file name.
For instance, if you're using a shell script to generate the JSON, you could do something like:
# Assume direct-node.json.tmp is your temporary file and direct-node.json is your desired fileecho"[YOUR_JSON_HERE]"> /discovery/direct-node.json.tmp && mv /discovery/direct-node.json.tmp /discovery/direct-node.json
The mv command, or move, is an atomic operation - this means the file system ensures the file either fully exists at the new location or doesn't exist at all. Thus, the above script will ensure the direct-node.json file will only exist if it's fully written, and Prometheus won't have a chance to read a partial file.
I hope this helps. Please let me know if you have further questions or if the problem persists
What did you do?
Deploying prometheus via official helm chart, with additinal sidecar to find metrics endpoints within a subnet, which will generate a json file with targets
What did you expect to see?
prometheus reading this
file_sd
fileWhat did you see instead? Under which circumstances?
System information
Helm, Kubernetes v1.23 @opentelekomcloud
Prometheus version
Prometheus configuration file
/discovery/direct-node.json
Alertmanager configuration file
not used
Logs
The text was updated successfully, but these errors were encountered: