[CI/CD] Some integration tests are currently being skipped due to naming restriction

[crobert-1]

Component(s)

No response

Background

While investigating #32206 I found that running a basic integration test command was causing test failures in the receiver/mongodbatlasreceiver package:

go test -tags=integration

The failing tests (here's one for example) were referencing files whose names changed over a year ago, so I investigated a bit and found that these tests weren't running.

Recent successful CI run output:

Running target 'mod-integration-test' in module 'receiver/mongodbatlasreceiver' as part of group 'receiver-1'
make --no-print-directory -C receiver/mongodbatlasreceiver mod-integration-test
running go integration test ./... in /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/mongodbatlasreceiver
/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/.tools/gotestsum --rerun-fails=1 --packages="./..." -- -race -timeout 360s -parallel 4 -tags=integration,"" -run=Integration
∅  internal
∅  internal/model
∅  internal/metadata (1.018s)
✓  . (2.101s)

DONE 1 tests in 6.305s

However, the MongoDB Atlas receiver has 4 integration tests, one found here, and three found here.

Problem

I found that the reason not all integration tests are running is because of the -run=Integration option in the following makefile variable:

opentelemetry-collector-contrib/Makefile.Common

Line 32 in 191c2c9

GOTEST_OPT_WITH_INTEGRATION=$(GOTEST_INTEGRATION_OPT) -tags=integration,$(GO_BUILD_TAGS) -run=Integration

This was introduced here.

From Go's documentation this means that only tests containing Integration in their name are being run.

Impact

I used a script to scan through all of the integration test files and their tests names, only returning tests without Integration in their name. Here are the results:

./extension/healthcheckextension/integration_test.go
func Test_SimpleHealthCheck(t *testing.T) {

./testbed/tests/syslog_integration_test.go
func TestSyslogComplementaryRFC5424(t *testing.T) {
func TestSyslogComplementaryRFC3164(t *testing.T) {

./receiver/hostmetricsreceiver/integration_test.go
func Test_ProcessScrape(t *testing.T) {
func Test_ProcessScrapeWithCustomRootPath(t *testing.T) {
func Test_ProcessScrapeWithBadRootPathAndEnvVar(t *testing.T) {

./receiver/cloudflarereceiver/logs_integration_test.go
func TestReceiverTLS(t *testing.T) {

./receiver/mongodbatlasreceiver/alerts_integration_test.go
func TestAlertsReceiver(t *testing.T) {
func TestAlertsReceiverTLS(t *testing.T) {
func TestAtlasPoll(t *testing.T) {

./exporter/splunkhecexporter/integration_test.go
func TestSplunkHecExporter(t *testing.T) {

./exporter/opensearchexporter/integration_test.go
func TestOpenSearchTraceExporter(t *testing.T) {
func TestOpenSearchLogExporter(t *testing.T) {

./pkg/stanza/adapter/integration_test.go
func TestEmitterToConsumer(t *testing.T) {

Proposed solution

I think the main two options are:

1. Remove the -run=Integration option so that all tests in a file with the integration build tag are run.
2. Rename all integration tests to have Integration in their name.

I think 1 is the best option here, as it would be too easy to forget and miss that this is a requirement of integration tests, shown by how long this has been missed.

Side note: Removing the -run argument may have some side effects on running tests that are currently passing. I'm seeing there are four integration tests in the Clickhouse Exporter that are all running successfully right now, but without the argument they hit a panic. There might be some kind of parallelization impact here that may need investigated.

[crobert-1]

Removing needs triage based on PR being approved by project maintainers.