Add template testing functionality

[JacobsonMT]

This adds template testing functionality.

Description
The method will attempt to interpolate each of the top-level definition blocks passed in through the template field with the given alerts as template context data. A top-level definition is one that isn't used by any others.

Default templates and any existing templates defined in the current AM will be available for reference by those being tested. The exception to this are the existing templates coming from the same filename as the one passed via the name field. This is so that users can real-time test out replacements to existing templates.

Notes for reviewer
There's some added complexity in this PR because template testing needs to recreate the existing templates from scratch. It needs to do this for a couple reasons:

1. Upstream Template does not provide a lot of way to manipulate its underlying text/template fields.
2. Even if it did, text/template does not provide a way to remove an already defined template. We need this in order to not mistakenly parse an existing template file that matches the name of the one we're testing as the new template might have removed some definitions. We could overwrite the templates with blanks but this might hide execution_errors.

So, in order to allow template testing to recreate the existing template we store the filenames and working directory for re-reading. I chose not to store the template text itself to keep the code as close as possible to existing template parsing which uses templates.FromGlobs instead of parsing from string.

Previously, the already parsed template was passed into grafana_alertmanager during ApplyConfig. This PR makes it so that grafana_alertmanager is the one that creates the template from the passed in filenames instead.

[JacobsonMT]

This is moved from grafana/grafana which seems correct since it's used only for receiver testing.

[yuri-tceretian]

Testing API code needs to be refactored to build by receiver

[yuri-tceretian]

Testing API code needs to be refactored to build by receiver

[grobinson-grafana]

I get a recursive stack depth error for the following:

{
  "alerts": [{
    "annotations": {
      "summary": "Instance instance1 has been down for more than 5 minutes"
    },
    "labels": {
      "instance": "instance1"
    },
    "startsAt": "2023-04-01T00:00:00Z",
    "endsAt": "2023-04-01T00:05:00Z"
  }],
  "template": "{{ define \"test\" }}{{ range .Alerts }}{{ index .Annotations \"summary\" }}\n{{ end }}{{ end }}{{ template \"test\" . }}"
}

I think we should be able to make this work as per the following example:

package main

import (
	"bytes"
	"fmt"
	"text/template"
)

func main() {
	tmplText := `{{ define "hello" }}Hello, world!{{end}}{{ template "hello" . }}`
	tmpl, err := template.New("").Parse(tmplText)
	if err != nil {
		panic(err)
	}
	var buf bytes.Buffer
	tmpl.ExecuteTemplate(&buf, "", nil)
	fmt.Println(buf.String())
}

[grobinson-grafana]

I get a recursive stack depth error for the following:

{
  "alerts": [{
    "annotations": {
      "summary": "Instance instance1 has been down for more than 5 minutes"
    },
    "labels": {
      "instance": "instance1"
    },
    "startsAt": "2023-04-01T00:00:00Z",
    "endsAt": "2023-04-01T00:05:00Z"
  }],
  "template": "{{ define \"test\" }}{{ range .Alerts }}{{ index .Annotations \"summary\" }}\n{{ end }}{{ end }}{{ template \"test\" . }}"
}

I think we should be able to make this work as per the following example:

package main

import (
	"bytes"
	"fmt"
	"text/template"
)

func main() {
	tmplText := `{{ define "hello" }}Hello, world!{{end}}{{ template "hello" . }}`
	tmpl, err := template.New("").Parse(tmplText)
	if err != nil {
		panic(err)
	}
	var buf bytes.Buffer
	tmpl.ExecuteTemplate(&buf, "", nil)
	fmt.Println(buf.String())
}

This patch works for me :)

@@ -106,28 +106,26 @@ func (am *GrafanaAlertmanager) TestTemplate(ctx context.Context, c TestTemplates
 	ctx = notify.WithReceiverName(ctx, DefaultReceiverName)
 	ctx = notify.WithGroupLabels(ctx, model.LabelSet{DefaultGroupLabel: DefaultGroupLabelValue})

-	var tmplErr error
-	templater, _ := templates.TmplText(ctx, newTmpl, alerts, am.logger, &tmplErr)
+	promTmplData := notify.GetTemplateData(ctx, newTmpl, alerts, am.logger)
+	data := templates.ExtendData(promTmplData, am.logger)

 	// Iterate over each definition in the template and evaluate it.
 	var results TestTemplatesResults
 	for _, def := range definitions {
-		s := fmt.Sprintf(`{{ template "%s" . }}`, def)
-		val := templater(s)
-		if tmplErr != nil {
+		var buf bytes.Buffer
+		err := tmpl.ExecuteTemplate(&buf, def, data)
+		if err != nil {
 			results.Errors = append(results.Errors, TestTemplatesErrorResult{
 				Name:  def,
 				Kind:  ExecutionError,
-				Error: tmplErr,
+				Error: err,
+			})
+		} else {
+			results.Results = append(results.Results, TestTemplatesResult{
+				Name: def,
+				Text: buf.String(),
 			})
-			tmplErr = nil
-			continue
 		}
-
-		results.Results = append(results.Results, TestTemplatesResult{
-			Name: def,
-			Text: val,
-		})
 	}

 	return &results, nil

I'm not sure what you want to do with this code though as it's not required in the design doc for what we're going to deliver before Grafana 10.

newTmpl, err := am.TemplateFromPaths(paths, captureTemplate)
if err != nil {
	return nil, err
}

[yuri-tceretian]

I think this can be a slice because information about receiver name is in APIReceiver.

[yuri-tceretian]

By doing this you will be able to remove this confusing method

https://github.com/grafana/grafana/pull/66901/files#diff-4aa6a5bc542bc278b3cddc81fe825af177f1f354c9dbb08d0f01d86b7dbe414eR313-R319

[yuri-tceretian]

LGTM. My last comments are nice to have but not blockers