testbed: refactor TestCase with broken out LoadGenerator interface (#…

…30303)

**Description:**
Adding a feature - These changes break out the current
`testbed.LoadGenerator` to an interface of the same name, migrating its
(reference) implementation to a new `testbed.ProviderSender` type. This
will allow future test cases (in arbitrary distributions) to use the
existing* `TestCase` flow, but with arbitrary load generator
implementations beyond the capabilities of the current one. They also
provide a new `testbed.NewLoadGeneratorTestCase` similar to the existing
`testbed.NewTestCase` that is now called indirectly. I chose this route
to not force further signature changes.

In the process of this work, I found a few edge cases that can lead to*
races and hanging processes in error scenarios. They've been addressed
w/ some minimal synchronization.

Here's a partial* example of a LoadGenerator that can manage an
arbitrary number of `ProviderSender` instances, each of varying
telemetry types, that can now be used:

```go
type CompositeLoadGenerator struct {
	startTime      time.Time
	loadGenerators []testbed.LoadGenerator
}

func NewCompositeLoadGenerator(loadGenerators ...testbed.LoadGenerator) testbed.LoadGenerator {
	return &CompositeLoadGenerator{
		loadGenerators: loadGenerators,
	}
}

func (o *CompositeLoadGenerator) Start(options testbed.LoadOptions) {
	o.startTime = time.Now()
	for _, lg := range o.loadGenerators {
		lg.Start(options)
	}
}

func (o *CompositeLoadGenerator) Stop() {
	for _, lg := range o.loadGenerators {
		lg.Stop()
	}
}

func (o *CompositeLoadGenerator) DataItemsSent() uint64 {
	var dataItemsSent uint64
	for _, lg := range o.loadGenerators {
		dataItemsSent += lg.DataItemsSent()
	}
	return dataItemsSent
}

<...>
```

**Documentation:**
Updated readme.

.chloggen/testbedloadgenerator.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: breaking
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: testbed
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Expand TestCase capabilities with broken out LoadGenerator interface
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [30303]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [api]

testbed/README.md

@@ -12,7 +12,7 @@ Each test case within the suite should create a `testbed.TestCase` and supply im
 
 ## DataFlow
 
-`testbed.TestCase` uses `LoadGenerator` and `MockBackend` to further encapsulate pluggable components. `LoadGenerator` further encapsulates `DataProvider` and `DataSender` in order to generate and send data.  `MockBackend` further encapsulate `DataReceiver` and provide consume functionality.
+`testbed.TestCase` uses `LoadGenerator` and `MockBackend` to further encapsulate pluggable components. `LoadGenerator`'s reference implementation `ProviderSender`, created via `testbed.NewLoadGenerator()` further encapsulates `DataProvider` and `DataSender` in order to generate and send data. Any type satisfying the `LoadGenerator` interface can be used with `testbed.NewLoadGeneratorTestCase()` to exercise functionality beyond the single component limitation of `ProviderSender`. `MockBackend` further encapsulates `DataReceiver` and provides consume functionality.
 
 For instance, if using the existing end-to-end test, the general dataflow can be (Note that MockBackend does not really have a consumer instance, only to make it intuitive, this diagram draws it a separate module):