bump github.com/elastic/elastic-integration-corpus-generator-tool to …

README.md

@@ -100,6 +100,22 @@ These benchmarks allow you to benchmark any Ingest Node Pipelines defined by you
 
 For details on how to configure pipeline benchmarks for a package, review the [HOWTO guide](./docs/howto/pipeline_benchmarking.md).
 
+### `elastic-package benchmark generate-corpus`
+
+_Context: package_
+
+
+*BEWARE*: this command is in beta and it's behaviour may change in the future.
+Use this command to generate benchmarks corpus data for a package.
+Currently, only data for what we have related assets on https://github.com/elastic/elastic-integration-corpus-generator-tool are supported.
+For details on how to run this command, review the [HOWTO guide](./docs/howto/generate_corpus.md).
+
+### `elastic-package benchmark pipeline`
+
+_Context: package_
+
+Run pipeline benchmarks for the package.
+
 ### `elastic-package build`
 
 _Context: package_
@@ -124,6 +140,19 @@ You can use this command to modify the changelog following the expected format a
 This can be useful when introducing changelog entries for changes done by automated processes.
 
 
+### `elastic-package changelog add`
+
+_Context: package_
+
+Use this command to add an entry to the changelog file.
+
+The entry added will include the given description, type and link. It is added on top of the
+last entry in the current version
+
+Alternatively, you can start a new version indicating the specific version, or if it should
+be the next major, minor or patch version.
+
+
 ### `elastic-package check`
 
 _Context: package_
@@ -150,18 +179,62 @@ The command can help bootstrap the first draft of a package using embedded packa
 
 For details on how to create a new package, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/create_new_package.md).
 
+### `elastic-package create data-stream`
+
+_Context: global_
+
+Use this command to create a new data stream.
+
+The command can extend the package with a new data stream using embedded data stream template and wizard.
+
+### `elastic-package create package`
+
+_Context: global_
+
+Use this command to create a new package.
+
+The command can bootstrap the first draft of a package using embedded package template and wizard.
+
 ### `elastic-package dump`
 
 _Context: global_
 
 Use this command as an exploratory tool to dump resources from Elastic Stack (objects installed as part of package and agent policies).
 
+### `elastic-package dump agent-policies`
+
+_Context: global_
+
+Use this command to dump agent policies created by Fleet as part of a package installation.
+
+Use this command as an exploratory tool to dump agent policies as they are created by Fleet when installing a package. Dumped agent policies are stored in files as they are returned by APIs of the stack, without any processing.
+
+If no flag is provided, by default this command dumps all agent policies created by Fleet.
+
+If --package flag is provided, this command dumps all agent policies that the given package has been assigned to it.
+
+### `elastic-package dump installed-objects`
+
+_Context: global_
+
+Use this command to dump objects installed by Fleet as part of a package.
+
+Use this command as an exploratory tool to dump objects as they are installed by Fleet when installing a package. Dumped objects are stored in files as they are returned by APIs of the stack, without any processing.
+
 ### `elastic-package export`
 
 _Context: package_
 
 Use this command to export assets relevant for the package, e.g. Kibana dashboards.
 
+### `elastic-package export dashboards`
+
+_Context: package_
+
+Use this command to export dashboards with referenced objects from the Kibana instance.
+
+Use this command to download selected dashboards and other associated saved objects from Kibana. This command adjusts the downloaded saved objects according to package naming conventions (prefixes, unique IDs) and writes them locally into folders corresponding to saved object types (dashboard, visualization, map, etc.).
+
 ### `elastic-package format`
 
 _Context: package_
@@ -196,6 +269,24 @@ Individual user profiles appear in ~/.elastic-package/stack, and contain all the
 Once a new profile is created, it can be specified with the -p flag, or the ELASTIC_PACKAGE_PROFILE environment variable.
 User profiles are not overwritten on upgrade of elastic-stack, and can be freely modified to allow for different stack configs.
 
+### `elastic-package profiles create`
+
+_Context: global_
+
+
+
+### `elastic-package profiles delete`
+
+_Context: global_
+
+
+
+### `elastic-package profiles list`
+
+_Context: global_
+
+
+
 ### `elastic-package promote`
 
 _Context: global_
@@ -228,6 +319,12 @@ The report will show performance differences between both runs.
 It is formatted as a Markdown Github comment to use as part of the CI results.
 
 
+### `elastic-package report benchmark`
+
+_Context: package_
+
+Generate a benchmark report comparing local results against ones from another benchmark run.
+
 ### `elastic-package service`
 
 _Context: package_
@@ -236,6 +333,12 @@ Use this command to boot up the service stack that can be observed with the pack
 
 The command manages lifecycle of the service stack defined for the package ("_dev/deploy") for package development and testing purposes.
 
+### `elastic-package service up`
+
+_Context: package_
+
+
+
 ### `elastic-package stack`
 
 _Context: global_
@@ -246,6 +349,50 @@ Be aware that a common issue while trying to boot up the stack is that your Dock
 
 For details on how to connect the service with the Elastic stack, see the [service command](https://github.com/elastic/elastic-package/blob/main/README.md#elastic-package-service).
 
+### `elastic-package stack down`
+
+_Context: global_
+
+
+
+### `elastic-package stack dump`
+
+_Context: global_
+
+
+
+### `elastic-package stack shellinit`
+
+_Context: global_
+
+
+
+### `elastic-package stack status`
+
+_Context: global_
+
+
+
+### `elastic-package stack up`
+
+_Context: global_
+
+Use this command to boot up the stack locally.
+
+By default the latest released version of the stack is spun up but it is possible to specify a different version, including SNAPSHOT versions by appending --version <version>.
+
+Be aware that a common issue while trying to boot up the stack is that your Docker environments settings are too low in terms of memory threshold.
+
+To ęxpose local packages in the Package Registry, build them first and boot up the stack from inside of the Git repository containing the package (e.g. elastic/integrations). They will be copied to the development stack (~/.elastic-package/stack/development) and used to build a custom Docker image of the Package Registry.
+
+For details on how to connect the service with the Elastic stack, see the [service command](https://github.com/elastic/elastic-package/blob/main/README.md#elastic-package-service).
+
+### `elastic-package stack update`
+
+_Context: global_
+
+
+
 ### `elastic-package status [package]`
 
 _Context: package_
@@ -282,6 +429,30 @@ These tests allow you to test a package's ability to ingest data end-to-end.
 
 For details on how to configure amd run system tests, review the [HOWTO guide](https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md).
 
+### `elastic-package test asset`
+
+_Context: package_
+
+Run asset loading tests for the package.
+
+### `elastic-package test pipeline`
+
+_Context: package_
+
+Run pipeline tests for the package.
+
+### `elastic-package test static`
+
+_Context: package_
+
+Run static files tests for the package.
+
+### `elastic-package test system`
+
+_Context: package_
+
+Run system tests for the package.
+
 ### `elastic-package uninstall`
 
 _Context: package_

cmd/benchmark.go

@@ -28,10 +28,10 @@ import (
 )
 
 const generateLongDescription = `
-BEWARE: this command is in beta and it's behaviour may change in the future.
+*BEWARE*: this command is in beta and it's behaviour may change in the future.
 Use this command to generate benchmarks corpus data for a package.
 Currently, only data for what we have related assets on https://github.com/elastic/elastic-integration-corpus-generator-tool are supported.
-`
+For details on how to run this command, review the [HOWTO guide](./docs/howto/generate_corpus.md).`
 
 const benchLongDescription = `Use this command to run benchmarks on a package. Currently, the following types of benchmarks are available:
 
@@ -226,9 +226,10 @@ func getGenerateCorpusCommand() *cobra.Command {
 	}
 
 	generateCorpusCmd.Flags().StringP(cobraext.PackageFlagName, cobraext.PackageFlagShorthand, "", cobraext.PackageFlagDescription)
-	generateCorpusCmd.Flags().StringP(cobraext.GenerateCorpusDataStreamFlagName, cobraext.GenerateCorpusDataStreamFlagShorthand, "", cobraext.GenerateCorpusDataStreamFlagDescription)
+	generateCorpusCmd.Flags().StringP(cobraext.GenerateCorpusDataSetFlagName, cobraext.GenerateCorpusDataSetFlagShorthand, "", cobraext.GenerateCorpusDataSetFlagDescription)
 	generateCorpusCmd.Flags().StringP(cobraext.GenerateCorpusSizeFlagName, cobraext.GenerateCorpusSizeFlagShorthand, "", cobraext.GenerateCorpusSizeFlagDescription)
 	generateCorpusCmd.Flags().StringP(cobraext.GenerateCorpusCommitFlagName, cobraext.GenerateCorpusCommitFlagShorthand, "main", cobraext.GenerateCorpusCommitFlagDescription)
+	generateCorpusCmd.Flags().StringP(cobraext.GenerateCorpusRallyTrackOutputDirFlagName, cobraext.GenerateCorpusRallyTrackOutputDirFlagShorthand, "", cobraext.GenerateCorpusRallyTrackOutputDirFlagDescription)
 
 	return generateCorpusCmd
 }
@@ -239,9 +240,9 @@ func generateDataStreamCorpusCommandAction(cmd *cobra.Command, _ []string) error
 		return cobraext.FlagParsingError(err, cobraext.PackageFlagName)
 	}
 
-	dataStreamName, err := cmd.Flags().GetString(cobraext.GenerateCorpusDataStreamFlagName)
+	dataSetName, err := cmd.Flags().GetString(cobraext.GenerateCorpusDataSetFlagName)
 	if err != nil {
-		return cobraext.FlagParsingError(err, cobraext.GenerateCorpusDataStreamFlagName)
+		return cobraext.FlagParsingError(err, cobraext.GenerateCorpusDataSetFlagName)
 	}
 
 	totSize, err := cmd.Flags().GetString(cobraext.GenerateCorpusSizeFlagName)
@@ -263,12 +264,19 @@ func generateDataStreamCorpusCommandAction(cmd *cobra.Command, _ []string) error
 		commit = "main"
 	}
 
-	generator, err := corpusgenerator.NewGenerator(packageName, dataStreamName, commit, totSizeInBytes)
+	rallyTrackOutputDir, err := cmd.Flags().GetString(cobraext.GenerateCorpusRallyTrackOutputDirFlagName)
+	if err != nil {
+		return cobraext.FlagParsingError(err, cobraext.GenerateCorpusRallyTrackOutputDirFlagName)
+	}
+
+	generator, err := corpusgenerator.NewGenerator(packageName, dataSetName, commit, totSizeInBytes)
 	if err != nil {
 		return errors.Wrap(err, "can't generate benchmarks data corpus for data stream")
 	}
 
-	err = corpusgenerator.RunGenerator(generator)
+	// TODO: we need a way to extract the type from the package and dataset, currently hardcode to `metrics`
+	dataStream := fmt.Sprintf("metrics-%s.%s-default", packageName, dataSetName)
+	err = corpusgenerator.RunGenerator(generator, dataStream, rallyTrackOutputDir)
 	if err != nil {
 		return errors.Wrap(err, "can't generate benchmarks data corpus for data stream")
 	}

docs/howto/generate_corpus.md

@@ -0,0 +1,41 @@
+# HOWTO: Generate corpus for a package dataset
+
+## Introduction
+
+The `elastic-package` tool can be used to generate a rally corpus for a package dataset.
+This feature is currently in beta and manual steps are required to create a valid rally track from the generated corpus.
+Currently, only data for what we have related assets on https://github.com/elastic/elastic-integration-corpus-generator-tool are supported.
+
+### Generate a corpus for a package dataset
+
+#### Steps
+
+1. Run the elastic-package command for generating the corpus of the package dataset:
+   `elastic-package benchmark generate-corpus --dataset sqs --package aws --size 100M`
+   1. replace the sample value for `--dataset` with the one of the dataset you want to generate a corpus for
+   2. replace the sample value for `--package` with the one of the package you want to generate a corpus for
+   3. replace the sample value for `--size` with the *approximate* size of the corpus you want to generate
+2. Choose a file where to redirect the output of the command if you want to save it:
+   `elastic-package benchmark generate-corpus --dataset sqs --package aws --size 100M > aws.sqs.100M.ndjson`
+    1. replace the sample value of the redirect file with the one you've chosen
+
+### Generate a rally track for a package dataset and run a rally benchmark
+
+*BEWARE*: this is only supported for `metrics` type data streams.
+
+#### Steps
+
+1. Run the elastic-package command for generating the corpus of the package dataset:
+   `elastic-package benchmark generate-corpus --dataset sqs --package aws --size 100M --rally-track-output-dir
+   ./track-output-dir`
+   1. replace the sample value for `--dataset` with the one of the dataset you want to generate a corpus for
+   2. replace the sample value for `--package` with the one of the package you want to generate a corpus for
+   3. replace the sample value for `--size` with the *approximate* size of the corpus you want to generate
+   4. replace the sample value for `--rally-track-output-dir` with the path to the folder where you want to save the rally track and the generated corpus (the folder will be created if it does not exist already)
+2. Go to the Kibana instance of the cluster you want to run the rally on and install the integration package that you have generated the rally track for. 
+3. Run the rally race with the generated track:
+   `esrally race --kill-running-processes --track-path=./track-output-dir --target-hosts=my-deployment.es.eastus2.azure.elastic-cloud.com:443 --pipeline=benchmark-only`
+   1. replace the sample value for `--track-path` with the path to the folder provided as `--rally-track-output-dir` at step 1
+   2. replace the sample value for `--target-hosts` with the host and port of the Elasticsearch instance(s) you want rally to connect to.
+   3. You might need to add the "client-options" parameter to rally in order to authenticate and use SSL: `--client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'changeme'"`
+      1. replace the sample value for `basic_auth_user` and `basic_auth_password` in `--client-options` to the credentials of the user in the cluster you want rally to use.

go.mod

@@ -10,7 +10,7 @@ require (
 	github.com/boumenot/gocover-cobertura v1.2.0
 	github.com/cespare/xxhash/v2 v2.2.0
 	github.com/dustin/go-humanize v1.0.1
-	github.com/elastic/elastic-integration-corpus-generator-tool v0.4.0
+	github.com/elastic/elastic-integration-corpus-generator-tool v0.4.3
 	github.com/elastic/go-elasticsearch/v7 v7.17.7
 	github.com/elastic/go-licenser v0.4.1
 	github.com/elastic/go-ucfg v0.8.6

go.sum

@@ -89,8 +89,8 @@ github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5/go.mod h1:qssHWj6
 github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
-github.com/elastic/elastic-integration-corpus-generator-tool v0.4.0 h1:wDScX6SQiF5rhwh9a1oecbMvBlaG7y+MzGXzXfhECtI=
-github.com/elastic/elastic-integration-corpus-generator-tool v0.4.0/go.mod h1:VIRTrT1Bj5PjRE+/RdNiR9O+Q9vg0DDaPLpvMKwvxj8=
+github.com/elastic/elastic-integration-corpus-generator-tool v0.4.3 h1:+QeArnxBKEP9t1GdEs4iEcZHVcrWXKndynJqITHrBi4=
+github.com/elastic/elastic-integration-corpus-generator-tool v0.4.3/go.mod h1:uf9N86y+UACGybdEhZLpwZ93XHWVhsYZAA4c2T2v6YM=
 github.com/elastic/go-elasticsearch/v7 v7.17.7 h1:pcYNfITNPusl+cLwLN6OLmVT+F73Els0nbaWOmYachs=
 github.com/elastic/go-elasticsearch/v7 v7.17.7/go.mod h1:OJ4wdbtDNk5g503kvlHLyErCgQwwzmDtaFC4XyOxXA4=
 github.com/elastic/go-licenser v0.4.1 h1:1xDURsc8pL5zYT9R29425J3vkHdt4RT5TNEMeRN48x4=

internal/cobraext/flags.go

@@ -92,14 +92,18 @@ const (
 	FailFastFlagName        = "fail-fast"
 	FailFastFlagDescription = "fail immediately if any file requires updates (do not overwrite)"
 
-	GenerateCorpusDataStreamFlagName        = "data-stream"
-	GenerateCorpusDataStreamFlagShorthand   = "D"
-	GenerateCorpusDataStreamFlagDescription = "data stream to generate benchmarks data corpus for"
+	GenerateCorpusDataSetFlagName        = "dataset"
+	GenerateCorpusDataSetFlagShorthand   = "D"
+	GenerateCorpusDataSetFlagDescription = "dataset to generate benchmarks data corpus for"
 
 	GenerateCorpusCommitFlagName        = "commit"
 	GenerateCorpusCommitFlagShorthand   = "C"
 	GenerateCorpusCommitFlagDescription = "commit to fetch assets from the corpus generator tool repo from"
 
+	GenerateCorpusRallyTrackOutputDirFlagName        = "rally-track-output-dir"
+	GenerateCorpusRallyTrackOutputDirFlagShorthand   = "R"
+	GenerateCorpusRallyTrackOutputDirFlagDescription = "output dir of the rally track: if present the command will generate a rally track instead of writing the generated data to stdout"
+
 	GenerateCorpusSizeFlagName        = "size"
 	GenerateCorpusSizeFlagShorthand   = "S"
 	GenerateCorpusSizeFlagDescription = "size of benchmarks data corpus to generate"