file_sd_configs - "unexpected end of JSON input"

[sfhl]

What did you do?

Deploying prometheus via official helm chart, with additinal sidecar to find metrics endpoints within a subnet, which will generate a json file with targets

What did you expect to see?

prometheus reading this file_sd file

What did you see instead? Under which circumstances?

...
ts=2023-05-22T08:53:29.253Z caller=file.go:343 level=error component="discovery manager scrape" discovery=file 
config=network_discovery msg="Error reading file" path=/discovery/direct-node.json err="unexpected end of JSON input"
...

System information

Helm, Kubernetes v1.23 @opentelekomcloud

Prometheus version

v2.44.0

Prometheus configuration file

/etc/config/prometheus.yml
...
- job_name: 'network_discovery'
  file_sd_configs:
    - files:
      - /discovery/*.json
...

/discovery/direct-node.json

[
        {
                "targets": [
                        "172.29.42.211:9100"
                ],
                "labels": {
                        "host": "172.29.42.211",
                        "subnet": "172.29.42.192/26"
                }
        },
        {
                "targets": [
                        "172.29.42.210:9100"
                ],
                "labels": {
                        "host": "172.29.42.210",
                        "subnet": "172.29.42.192/26"
                }
        }
]

### Alertmanager version

```text
not used

Alertmanager configuration file

not used

Logs

...
ts=2023-05-22T08:53:29.253Z caller=file.go:343 level=error component="discovery manager scrape" discovery=file 
config=network_discovery msg="Error reading file" path=/discovery/direct-node.json err="unexpected end of JSON input"
...

[roidelapluie]

The issue you're experiencing can happen if Prometheus attempts to read the JSON file while it's in the middle of being written or updated. If the file isn't completely written, this can lead to an "unexpected end of JSON input" error, as you're seeing in your logs.

To avoid this situation, you should ensure that the JSON file is written atomically. This can be achieved by writing the new contents to a temporary file and then renaming the temporary file to the desired file name.

For instance, if you're using a shell script to generate the JSON, you could do something like:

# Assume direct-node.json.tmp is your temporary file and direct-node.json is your desired file
echo "[YOUR_JSON_HERE]" > /discovery/direct-node.json.tmp && mv /discovery/direct-node.json.tmp /discovery/direct-node.json

The mv command, or move, is an atomic operation - this means the file system ensures the file either fully exists at the new location or doesn't exist at all. Thus, the above script will ensure the direct-node.json file will only exist if it's fully written, and Prometheus won't have a chance to read a partial file.

I hope this helps. Please let me know if you have further questions or if the problem persists

[sfhl]

Hi there,
i have rewritten the component creating the target files to create yaml files and the errors are gone.
So i will stick to the yaml format and watch if the proble reoccurs.

Thank you very much!