integ-runner: new integration tests always throw "Please use the IntegTest construct to configure the test"

[diranged]

Describe the bug

Every once in a while, it seems like a change to the manifests generated by the integ-runner code causes us to have to wipe out the .snapshot directory and start over. We'll do an update to the CDK libraries, and see an error like this:

Integration test failed: Error: Cannot read integ manifest 'test/integ/integ.dlq.ts.snapshot/manifest.json': Cloud assembly schema version mismatch: Maximum schema version supported is [35](XXX/actions/runs/7387728242/job/20097056043#step:10:36).0.0, but found 36.0.0

The thing is, when we wipe out the .snapshot directory to just start over:

% find test/integ 
test/integ
test/integ/integ.dlq.ts

We will get a fun new error:

👾 integ-dry-run » integ | npx integ-runner --update-on-failed --dry-run

Verifying integration test snapshots...

  NEW        integ.dlq 26.378s

Snapshot Results: 

Tests:    2 failed, 2 total
Failed: /Users/diranged/git/xxx/xxx/test/integ/integ.dlq.ts

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-west-2, us-east-2
Running test /Users/diranged/git/xxx/xxx/test/integ/integ.dlq.ts in us-east-1
  ERROR      /Users/diranged/git/xxx/xxx/test/integ/integ.dlq.ts (undefined/us-east-1) 24.809s
      Error during integration test: Error: integ.dlq is a new test. Please use the IntegTest construct to configure the test
https://github.com/aws/aws-cdk/tree/main/packages/%40aws-cdk/integ-tests-alpha

Test Results: 

Tests:    1 failed, 1 total
   --- Integration test metrics ---
Profile undefined + Region us-east-1 total time: 24.811
  /Users/diranged/git/xxx/xxx/test/integ/integ.dlq.ts: 24.811

After digging through the code, I see two lines that don't make sense to me...

At

aws-cdk/packages/@aws-cdk/integ-runner/lib/runner/integ-test-runner.ts

Line 78 in 166967f

if (!this.hasSnapshot() && this.isLegacyTest) {

, the code says (!this.hasSnapshot() && this.isLegacyTest) ... then throw error. Fine... we deleted the snapshot so that part makes sense, but then what is this isLegacyTest bit? Well according to

aws-cdk/packages/@aws-cdk/integ-runner/lib/runner/runner-base.ts

Lines 209 to 237 in 166967f

	/**
	* Load the integ manifest which contains information
	* on how to execute the tests
	* First we try and load the manifest from the integ manifest (i.e. integ.json)
	* from the cloud assembly. If it doesn't exist, then we fallback to the
	* "legacy mode" and create a manifest from pragma
	*/
	protected loadManifest(dir?: string): IntegTestSuite | LegacyIntegTestSuite {
	try {
	const testSuite = IntegTestSuite.fromPath(dir ?? this.snapshotDir);
	return testSuite;
	} catch {
	const testCases = LegacyIntegTestSuite.fromLegacy({
	cdk: this.cdk,
	testName: this.test.normalizedTestName,
	integSourceFilePath: this.test.fileName,
	listOptions: {
	...this.defaultArgs,
	all: true,
	app: this.cdkApp,
	profile: this.profile,
	output: path.relative(this.directory, this.cdkOutDir),
	},
	});
	this.legacyContext = LegacyIntegTestSuite.getPragmaContext(this.test.fileName);
	this.isLegacyTest = true;
	return testCases;
	}
	}

, isLegacyTest=true if there is no integ.json file .. but of course there isn't, because there's no .snapshot at all!

What am I missing here? It seems like this is designed to break if you try to generate a fresh snapshot?

Expected Behavior

I'd expect it to just create a new snapshot from scratch based on the current code base.

Current Behavior

See above.

Reproduction Steps

See above.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.116.1

Framework Version

No response

Node.js Version

18

OS

OSX

Language

TypeScript

Language Version

No response

Other information

No response

[diranged]

Closing. This is caused by a mismatch in the aws-cdk-lib and integ-runner libraries, which is really easy to do .

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.