Rule evaluation warnings prevent clean shutdown

[spyrexd]

A sigterm was sent to the prometheus container to cleanly shutdown the process:

docker kill --signal="SIGTERM" prom

After the shutting down message the following log message is logged for each alert rule that is configured:

level=warning msg="Error while evaluating rule "ALERT CritDiskUsage\n\tIF collectd_df_percent_bytes{type=\"free\"} < 5\n\tWITH {severity=\"critical\"}\n\tSUMMARY \"Warning Disk Volume {{$labels.df}} on {{$labels.exported_instance}} is low on space\"\n\tDESCRIPTION \"Waarning Disk Volume {{$lables.df}} on {{$lables.exported_instance}} has less than 5% disk available. Current Available {{$value}}%\"\n\tRUNBOOK \"\"": query was canceled in query queue" source="manager.go:257"

The logs messages keep repeating and prometheus does not shutdown.

[fabxc]

Which revision are you running? (Shown on the /status page in the web UI.)

[spyrexd]

I am running 0.16.2

[beorn7]

Further up in the logfile, can you see any messages about suspended ingestion? (There are usually a lot of them.) Then it is #1319 .

[spyrexd]

I do not see any logs mentioning suspended ingestion, but when I looked at the status this morning I did notice it had been over 60 hours since the last scrape of my targets. The long time since last scrape is what prompted me to stop Prometheus.

[fabxc]

@spyrexd We need to know the revision rather than the version since the version is only bumped on releases. Please check the status page again for the revision row.

I do believe that this logging was already fixed and similarly the blocked shutdown, which is unrelated.

[spyrexd]

The revision is 287d9b2

[fabxc]

We made some changes that might affect this. Can you please build from HEAD and verify whether the problem is still there.
If it is, please share the full log this time and indicate at which point you shut it down? The fact that the above message is printed, has most likely nothing to do with the actual issue.

[beorn7]

I assume this is fixed. Please re-open if that's not the case.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.