Pass policy_template name to Kibana for system tests

docs/howto/system_testing.md

@@ -114,7 +114,7 @@ will not be used. An agent will be deployed as a Docker compose service named `d
 which base configuration is provided [here](../../internal/install/_static/docker-custom-agent-base.yml).
 This configuration will be merged with the one provided in the `custom-agent.yml` file.
 This is useful if you need different capabilities than the provided by the
-`elastic-agent` used by the `elastic-package stack` command. 
+`elastic-agent` used by the `elastic-package stack` command.
 
 `custom-agent.yml`
 ```
@@ -260,9 +260,12 @@ elastic-package test system --data-streams pod -v # start system tests for the "
 
 ### Test case definition
 
-Next, we must define at least one configuration for each data stream that we want to system test. There can be multiple test cases defined for the same data stream.
+Next, we must define at least one configuration for each data stream that we
+want to system test. There can be multiple test cases defined for the same data
+stream.
 
-_Hint: if you plan to define only one test case, you can consider the filename `test-default-config.yml`._
+_Hint: if you plan to define only one test case, you can consider the filename
+`test-default-config.yml`._
 
 ```
 <package root>/
@@ -274,25 +277,57 @@ _Hint: if you plan to define only one test case, you can consider the filename `
             test-<test_name>-config.yml
 ```
 
-The `test-<test_name>-config.yml` file allows you to define values for package and data stream-level variables. For example, the `apache/access` data stream's `test-access-log-config.yml` is shown below.
+The `test-<test_name>-config.yml` file allows you to define values for package
+and data stream-level variables. These are the available configuration options
+for system tests.
 
-```
+| Option | Type | Required | Description |
+|---|---|---|---|
+| data_stream.vars | dictionary |  | Data stream level variables to set (i.e. declared in `package_root/data_stream/$data_stream/manifest.yml`). If not specified the defaults from the manifest are used. |
+| input | string | yes | Input type to test (e.g. logfile, httpjson, etc). Defaults to the input used by the first stream in the data stream manifest. |
+| numeric_keyword_fields | []string |  | List of fields to ignore during validation that are mapped as `keyword` in Elasticsearch, but their JSON data type is a number. |
+| policy_template | string |  | Name of policy template associated with the data stream and input. Required when multiple policy templates include the input being tested. |
+| service | string |  | Name of a specific Docker service to setup for the test. |
+| service_notify_signal | string |  | Signal name to send to 'service' when the test policy has been applied to the Agent. This can be used to trigger the service after the Agent is ready to receive data. |
+| skip.link | URL |  | URL linking to an issue about why the test is skipped. |
+| skip.reason | string |  | Reason to skip the test. If specified the test will not execute. |
+| vars | dictionary |  | Package level variables to set (i.e. declared in `$package_root/manifest.yml`). If not specified the defaults from the manifest are used. |
+| wait_for_data_timeout | duration |  | Amount of time to wait for data to be present in Elasticsearch. Defaults to 10m. |
+
+For example, the `apache/access` data stream's `test-access-log-config.yml` is
+shown below.
+
+```yaml
 vars: ~
 input: logfile
 data_stream:
   vars:
     paths:
-      - "{{SERVICE_LOGS_DIR}}/access.log*"
+      - "{{{SERVICE_LOGS_DIR}}}/access.log*"
 ```
 
-The top-level `vars` field corresponds to package-level variables defined in the `apache` package's `manifest.yml` file. In the above example we don't override any of these package-level variables, so their default values, as specified in the `apache` package's `manifest.yml` file are used.
+The top-level `vars` field corresponds to package-level variables defined in the
+`apache` package's `manifest.yml` file. In the above example we don't override
+any of these package-level variables, so their default values, as specified in
+the `apache` package's `manifest.yml` file are used.
 
-The `data_stream.vars` field corresponds to data stream-level variables for the current data stream (`apache/access` in the above example). In the above example we override the `paths` variable. All other variables are populated with their default values, as specified in the `apache/access` data stream's `manifest.yml` file.
+The `data_stream.vars` field corresponds to data stream-level variables for the
+current data stream (`apache/access` in the above example). In the above example
+we override the `paths` variable. All other variables are populated with their
+default values, as specified in the `apache/access` data stream's `manifest.yml`
+file.
 
-Notice the use of the `{{SERVICE_LOGS_DIR}}` placeholder. This corresponds to the `${SERVICE_LOGS_DIR}` variable we saw in the `docker-compose.yml` file earlier. In the above example, the net effect is as if the `/usr/local/apache2/logs/access.log*` files located inside the Apache integration service container become available at the same path from Elastic Agent's perspective.
+Notice the use of the `{{{SERVICE_LOGS_DIR}}}` placeholder. This corresponds to
+the `${SERVICE_LOGS_DIR}` variable we saw in the `docker-compose.yml` file
+earlier. In the above example, the net effect is as if the
+`/usr/local/apache2/logs/access.log*` files located inside the Apache
+integration service container become available at the same path from Elastic
+Agent's perspective.
 
-When a data stream's manifest declares multiple streams with different inputs you can use the `input` option to select the stream to test. The first stream
-whose input type matches the `input` value will be tested. By default, the first stream declared in the manifest will be tested.
+When a data stream's manifest declares multiple streams with different inputs
+you can use the `input` option to select the stream to test. The first stream
+whose input type matches the `input` value will be tested. By default, the first
+stream declared in the manifest will be tested.
 
 #### Placeholders
 
@@ -306,7 +341,7 @@ The `SERVICE_LOGS_DIR` placeholder is not the only one available for use in a da
 | `Logs.Folder.Agent` | string | Path to integration service's logs folder, as addressable by the Agent. |
 | `SERVICE_LOGS_DIR` | string | Alias for `Logs.Folder.Agent`. Provided as a convenience. |
 
-Placeholders used in the `test-<test_name>-config.yml` must be enclosed in `{{` and `}}` delimiters, per Handlebars syntax.
+Placeholders used in the `test-<test_name>-config.yml` must be enclosed in `{{{` and `}}}` delimiters, per Handlebars syntax.
 
 
 **NOTE**: Terraform variables in the form of environment variables (prefixed with `TF_VAR_`) are not injected and cannot be used as placeholder (their value will always be empty).
@@ -372,4 +407,3 @@ The exposed environment variables are passed to the test runners through service
 The tests use the [default version](https://github.com/elastic/elastic-package/blob/main/internal/install/stack_version.go#L9) `elastic-package` provides.
 
 You can override this value by changing it in your PR if needed. To update the default version always create a dedicated PR.
-

internal/kibana/policies.go

@@ -172,10 +172,11 @@ type Stream struct {
 
 // Input represents a package-level input.
 type Input struct {
-	Type    string   `json:"type"`
-	Enabled bool     `json:"enabled"`
-	Streams []Stream `json:"streams"`
-	Vars    Vars     `json:"vars"`
+	PolicyTemplate string   `json:"policy_template,omitempty"` // Name of policy_template from the package manifest that contains this input. If not specified the Kibana uses the first policy_template.
+	Type           string   `json:"type"`
+	Enabled        bool     `json:"enabled"`
+	Streams        []Stream `json:"streams"`
+	Vars           Vars     `json:"vars"`
 }
 
 // PackageDataStream represents a request to add a single package's single data stream to a

internal/packages/packages.go

@@ -10,9 +10,10 @@ import (
 	"os"
 	"path/filepath"
 
+	"github.com/pkg/errors"
+
 	"github.com/elastic/go-ucfg"
 	"github.com/elastic/go-ucfg/yaml"
-	"github.com/pkg/errors"
 )
 
 const (
@@ -96,7 +97,9 @@ type Conditions struct {
 
 // PolicyTemplate is a configuration of inputs responsible for collecting log or metric data.
 type PolicyTemplate struct {
-	Inputs []Input `config:"inputs,omitempty" json:"inputs,omitempty" yaml:"inputs,omitempty"`
+	Name        string   `config:"name" json:"name" yaml:"name"`                                                       // Name of policy template.
+	DataStreams []string `config:"data_streams,omitempty" json:"data_streams,omitempty" yaml:"data_streams,omitempty"` // List of data streams compatible with the policy template.
+	Inputs      []Input  `config:"inputs,omitempty" json:"inputs,omitempty" yaml:"inputs,omitempty"`
 
 	// For purposes of "input packages"
 	Input        string `config:"input,omitempty" json:"input,omitempty" yaml:"input,omitempty"`

internal/testrunner/runners/system/runner.go

@@ -286,6 +286,14 @@ func (r *runner) runTest(config *testConfig, ctxt servicedeployer.ServiceContext
 		return result.WithError(errors.Wrap(err, "reading data stream manifest failed"))
 	}
 
+	policyTemplateName := config.PolicyTemplate
+	if policyTemplateName == "" {
+		policyTemplateName, err = findPolicyTemplateForInput(*pkgManifest, *dataStreamManifest, config.Input)
+		if err != nil {
+			return result.WithError(errors.Wrap(err, "failed to determine the associated policy_template"))
+		}
+	}
+
 	// Setup service.
 	logger.Debug("setting up service...")
 	serviceDeployer, err := servicedeployer.Factory(serviceOptions)
@@ -352,7 +360,7 @@ func (r *runner) runTest(config *testConfig, ctxt servicedeployer.ServiceContext
 	}
 
 	logger.Debug("adding package data stream to test policy...")
-	ds := createPackageDatastream(*policy, *pkgManifest, *dataStreamManifest, *config)
+	ds := createPackageDatastream(*policy, *pkgManifest, policyTemplateName, *dataStreamManifest, *config)
 	if err := kib.AddPackageDataStreamToPolicy(ds); err != nil {
 		return result.WithError(errors.Wrap(err, "could not add data stream config to policy"))
 	}
@@ -504,6 +512,7 @@ func checkEnrolledAgents(client *kibana.Client, ctxt servicedeployer.ServiceCont
 func createPackageDatastream(
 	p kibana.Policy,
 	pkg packages.PackageManifest,
+	policyTemplate string,
 	ds packages.DataStreamManifest,
 	c testConfig,
 ) kibana.PackageDataStream {
@@ -522,8 +531,9 @@ func createPackageDatastream(
 
 	r.Inputs = []kibana.Input{
 		{
-			Type:    streamInput,
-			Enabled: true,
+			PolicyTemplate: policyTemplate,
+			Type:           streamInput,
+			Enabled:        true,
 		},
 	}
 
@@ -601,6 +611,48 @@ func getDataStreamDataset(pkg packages.PackageManifest, ds packages.DataStreamMa
 	return fmt.Sprintf("%s.%s", pkg.Name, ds.Name)
 }
 
+// findPolicyTemplateForInput returns the name of the policy_template that
+// applies to the input under test. An error is returned if no policy template
+// matches or if multiple policy templates match and the response is ambiguous.
+func findPolicyTemplateForInput(pkg packages.PackageManifest, ds packages.DataStreamManifest, inputName string) (string, error) {
+	if inputName == "" {
+		if len(ds.Streams) == 0 {
+			return "", errors.New("no streams declared in data stream manifest")
+		}
+		inputName = ds.Streams[getDataStreamIndex(inputName, ds)].Input
+	}
+
+	var matchedPolicyTemplates []string
+
+	for _, policyTemplate := range pkg.PolicyTemplates {
+		// Does this policy_template include this input type?
+		if policyTemplate.FindInputByType(inputName) == nil {
+			continue
+		}
+
+		// Does the policy_template apply to this data stream (when data streams are specified)?
+		if len(policyTemplate.DataStreams) > 0 && !common.StringSliceContains(policyTemplate.DataStreams, ds.Name) {
+			continue
+		}
+
+		matchedPolicyTemplates = append(matchedPolicyTemplates, policyTemplate.Name)
+	}
+
+	switch len(matchedPolicyTemplates) {
+	case 1:
+		return matchedPolicyTemplates[0], nil
+	case 0:
+		return "", fmt.Errorf("no policy template was found for data stream %q "+
+			"with input type %q: verify that you have included the data stream "+
+			"and input in the package's policy_template list", ds.Name, inputName)
+	default:
+		return "", fmt.Errorf("ambiguous result: multiple policy templates ([%s]) "+
+			"were found that apply to data stream %q with input type %q: please "+
+			"specify the 'policy_template' in the system test config",
+			strings.Join(matchedPolicyTemplates, ", "), ds.Name, inputName)
+	}
+}
+
 func deleteDataStreamDocs(api *elasticsearch.API, dataStream string) error {
 	body := strings.NewReader(`{ "query": { "match_all": {} } }`)
 	_, err := api.DeleteByQuery([]string{dataStream}, body)

internal/testrunner/runners/system/runner_test.go

@@ -0,0 +1,160 @@
+// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package system
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/elastic/elastic-package/internal/packages"
+)
+
+func TestFindPolicyTemplateForInput(t *testing.T) {
+	const policyTemplateName = "my_policy_template"
+	const dataStreamName = "my_data_stream"
+	const inputName = "logfile"
+
+	var testCases = []struct {
+		testName string
+		err      string
+		pkg      packages.PackageManifest
+		input    string
+	}{
+		{
+			testName: "single policy_template",
+			pkg: packages.PackageManifest{
+				PolicyTemplates: []packages.PolicyTemplate{
+					{
+						Name:        policyTemplateName,
+						DataStreams: nil,
+						Inputs: []packages.Input{
+							{
+								Type: inputName,
+							},
+						},
+					},
+				},
+			},
+			input: inputName,
+		},
+		{
+			testName: "unspecified input name",
+			pkg: packages.PackageManifest{
+				PolicyTemplates: []packages.PolicyTemplate{
+					{
+						Name:        policyTemplateName,
+						DataStreams: nil,
+						Inputs: []packages.Input{
+							{
+								Type: inputName,
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			testName: "input matching",
+			pkg: packages.PackageManifest{
+				PolicyTemplates: []packages.PolicyTemplate{
+					{
+						Name:        policyTemplateName,
+						DataStreams: nil,
+						Inputs: []packages.Input{
+							{
+								Type: inputName,
+							},
+						},
+					},
+					{
+						Name:        policyTemplateName + "1",
+						DataStreams: nil,
+						Inputs: []packages.Input{
+							{
+								Type: "not_" + inputName,
+							},
+						},
+					},
+				},
+			},
+			input: inputName,
+		},
+		{
+			testName: "data stream not specified",
+			err:      "no policy template was found",
+			pkg: packages.PackageManifest{
+				PolicyTemplates: []packages.PolicyTemplate{
+					{
+						Name:        policyTemplateName,
+						DataStreams: []string{"not_" + dataStreamName},
+						Inputs: []packages.Input{
+							{
+								Type: inputName,
+							},
+						},
+					},
+				},
+			},
+			input: inputName,
+		},
+		{
+			testName: "multiple matches",
+			err:      "ambiguous result",
+			pkg: packages.PackageManifest{
+				PolicyTemplates: []packages.PolicyTemplate{
+					{
+						Name:        policyTemplateName,
+						DataStreams: []string{dataStreamName},
+						Inputs: []packages.Input{
+							{
+								Type: inputName,
+							},
+						},
+					},
+					{
+						Name:        policyTemplateName + "1",
+						DataStreams: []string{dataStreamName},
+						Inputs: []packages.Input{
+							{
+								Type: inputName,
+							},
+						},
+					},
+				},
+			},
+			input: inputName,
+		},
+	}
+
+	ds := packages.DataStreamManifest{
+		Name: dataStreamName,
+		Streams: []struct {
+			Input string              `config:"input" json:"input" yaml:"input"`
+			Vars  []packages.Variable `config:"vars" json:"vars" yaml:"vars"`
+		}{
+			{Input: inputName},
+		},
+	}
+
+	t.Parallel()
+	for _, tc := range testCases {
+		tc := tc
+
+		t.Run(tc.testName, func(t *testing.T) {
+			name, err := findPolicyTemplateForInput(tc.pkg, ds, inputName)
+
+			if tc.err != "" {
+				require.Errorf(t, err, "expected err containing %q", tc.err)
+				assert.Contains(t, err.Error(), tc.err)
+				return
+			}
+
+			require.NoError(t, err)
+			assert.Equal(t, policyTemplateName, name)
+		})
+	}
+}