Cortex Alertmanager fails to load template filenames which contain directories

[mattmendick]

Describe the bug
Alertmanager has functionality to receive templates and use them with the provided rules files. However I believe the directory processing logic has a bug in it which causes a failure.

To Reproduce
To see the error message, where 999999 is the tenant ID:

Using the cortextool utility (from https://github.com/grafana/cortex-tools) with this command:

./cortextool alertmanager load --id 999999 --address https://alertmanager.url.here --key $ALERTMANAGER_API_KEY alerting/policies/alerts.yml alerting/policies/thetemplatefile.tmpl

Yields the error:

error validating Alertmanager config: unable to create Alertmanager template file \"/tmp/validate-config197042527/templates/999999/alerting/policies/alerting/policies/thetemplatefile.tmpl\": open /tmp/validate-config197042527/templates/999999/alerting/policies/alerting/policies/thetemplatefile.tmpl: no such file or directory\n" status="400 Bad Request"

Expected behavior
It should be able to create the file at the right directory and use it.

Additional Context
I believe the issue is this:

If you provide a filename like some/path/templatefile.tmpl it will create datadir/some/path/ and then try to write datadir/some/path/some/path/templatefile.tmpl

Creating directory at:

cortex/pkg/alertmanager/multitenant.go

Line 544 in b135907

dir := filepath.Join(dataDir, "templates", userID, filepath.Dir(fn))

But using the entire path at:

cortex/pkg/alertmanager/multitenant.go

Line 550 in b135907

file := filepath.Join(dir, fn)

550 should be something like file := filepath.Join(dir, filepath.Base(fn)) to only add templatefile.tmpl to the dir already created.

[gotjosh]

Hey @mattmendick ! Thanks for reporting this one.

Sadly, we don't support paths (only filenames) as part of the Alertmanager template definitions.

The key difference here is that unlike Prometheus cortex does not have a need to "look" for these files as we store them in object storage (and then write it to disk temporarily to consume it).

An alternative to what you proposed, to have a clearer error message to signal that this is not supported and make it clear in the documentation. I can see the appeal of just making it work for our users but I'm generally I'm the opposite side of modifying users configuration.

[mattmendick]

Thanks @gotjosh, I created grafana/cortex-tools#114 to address the issue on the cortextool side.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[jtlisi]

still valid

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

We recently discussed it and it's currently the intended behaviour.

[Carmezim]

I am having a similar issue when trying to configure the alertmanager

cortextool alertmanager load ./alertmanager_config.yml

yields:

error validating Alertmanager config: mkdir /tmp/validate-config001396953: read-only file system

Similarly through the API directly

[pracucci]

error validating Alertmanager config: mkdir /tmp/validate-config001396953: read-only file system

This is a different issue. Let's discuss it in the dedicated issue you've opened #4089.