[Metricbeat] [GCP] Fix compute metadata

[gpop63]

Proposed commit message

A few things I noticed while debugging:

1. NewMetadataService is invoked based on the period set in the config e.g. 1m, therefore the computeInstances map will clear by itself, no need to remake it again in instance. If the period is 1m, are there any risks of having stale metadata?
   

   beats/x-pack/metricbeat/module/gcp/metrics/compute/metadata.go

   Lines 25 to 32 in 138e623

   	func NewMetadataService(projectID, zone string, region string, regions []string, opt ...option.ClientOption) (gcp.MetadataService, error) {
   	return &metadataCollector{
   	projectID: projectID,
   	zone: zone,
   	region: region,
   	regions: regions,
   	opt: opt,
   	computeInstances: make(map[uint64]*computepb.Instance),

2. ID is also calling Metadata, and in timeSeriesGrouped both are called
   

   beats/x-pack/metricbeat/module/gcp/metrics/compute/identity.go

   Lines 16 to 17 in 138e623

   	func (s *metadataCollector) ID(ctx context.Context, in *gcp.MetadataCollectorInputData) (string, error) {
   	metadata, err := s.Metadata(ctx, in.TimeSeries)

   
   

   beats/x-pack/metricbeat/module/gcp/metrics/timeseries.go

   Lines 33 to 43 in 138e623

   	id, err := metadataService.ID(ctx, sdCollectorInputData)
   	if err != nil {
   	m.Logger().Errorf("error trying to retrieve ID from metric event '%v'", err)
   	continue
   	}
   	metadataCollectorData, err := metadataService.Metadata(ctx, sdCollectorInputData.TimeSeries)
   	if err != nil {
   	m.Logger().Error("error trying to retrieve labels from metric event")
   	continue
   	}

3. ListTimeSeries call in Metric gets instances from other projects, not only from the project_id specified in the config. The AggregatedList call will not be able to find those instances.
   

   beats/x-pack/metricbeat/module/gcp/metrics/metrics_requester.go

   Lines 39 to 53 in 138e623

   	func (r *metricsRequester) Metric(ctx context.Context, serviceName, metricType string, timeInterval *monitoringpb.TimeInterval, aligner string) timeSeriesWithAligner {
   	timeSeries := make([]*monitoringpb.TimeSeries, 0)
   	req := &monitoringpb.ListTimeSeriesRequest{
   	Name: "projects/" + r.config.ProjectID,
   	Interval: timeInterval,
   	View: monitoringpb.ListTimeSeriesRequest_FULL,
   	Filter: r.getFilterForMetric(serviceName, metricType),
   	Aggregation: &monitoringpb.Aggregation{
   	PerSeriesAligner: gcp.AlignersMapToGCP[aligner],
   	AlignmentPeriod: r.config.period,
   	},
   	}
   	it := r.client.ListTimeSeries(ctx, req)

I think the problem is that there are many Metadata calls and clearing the computeInstances map would trigger the compute AggregatedList call a lot of times, and this would go on forever since that API call can't get metadata for all instances - there are also Kubernetes pods/nodes (we can't get metadata for them) that send compute metrics, not only VM instances.

By not remaking the computeInstances map, we only make 1 AggregatedList call per period specified in config (1m).

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gpop63? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-08-28T10:39:06.653+0000

• Duration: 53 min 11 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1533
Skipped	96
Total	1629

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[tdancheva]

@kaiyan-sheng I unskipped the test in a draft PR and the test fails cause it runs on 8.8.2. Do you think it is a good idea to backport this so that we can validate it now instead of waiting for the next release?

[kaiyan-sheng]

@tdancheva good point! I would consider this as a bug fix and let's backport it.