Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking: [Integration Tests] #20384

Open
4 of 18 tasks
corymhall opened this issue May 17, 2022 · 1 comment
Open
4 of 18 tasks

Tracking: [Integration Tests] #20384

corymhall opened this issue May 17, 2022 · 1 comment
Labels
management/tracking Issues that track a subject or multiple issues

Comments

@corymhall
Copy link
Contributor

corymhall commented May 17, 2022 •
edited

Overview

Tracking issue for integ-tests and integ-runner

Link to the service’s CDK Construct Library API reference page.

https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html

Maturity: CloudFormation Resources Only

Experimental

Implementation

Integ Tests

Features

  • Add HttpApiCall similar to AwsApiCall (in progress)
  • Add the ability to have the assertions "wait" for something to occur (in progress)
    • Move assertion into the *ApiCall resources (in progress)
  • Access to addPolicyStatementFromSdkCall on AssertionsProvider (turns out there are a lot of permissions it can't figure out)

Integ Runner

Features

  • Automatically remove assets above a certain size (e.g. 20KB) - [high]
  • Add a tests remaining log (maybe only if # of tests is > something) - [low]
  • Allow filter by package name (or glob filter, etc) e.g. yarn integ 'aws-lambda/*' - [low]
  • Add flag to integ manifest to skip during automation - [low]
  • Add mass bootstrap of accounts/regions that are used - [low]
  • Add flag to disable snapshot creation
  • Add a flag to disable snapshots altogether (don't create or diff snapshots)
  • Report failed status on individual test cases/stacks

Bugs/Refactor

Issue list

  • [ ]
  • [ ]
@corymhall corymhall added the management/tracking Issues that track a subject or multiple issues label May 17, 2022
@corymhall corymhall self-assigned this May 17, 2022
@corymhall corymhall mentioned this issue May 26, 2022
Merged
4 tasks
mergify bot pushed a commit that referenced this issue May 27, 2022
@corymhall
fix(integ-runner): don't throw error if tests pass (#20511)
c274c2f 
If you run `integ-runner --update-on-failed` and the test succeeds, then
the cli should not return an exit code.

re #20384


----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/master/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@mrgrain
Copy link
Contributor

mrgrain commented Aug 17, 2022

TypeScript support
Out template for new CDK apps is configured to run jest via ts-node. Integ-runner needs files to be js.

@corymhall corymhall mentioned this issue Aug 25, 2022
Merged
4 tasks
mergify bot pushed a commit that referenced this issue Aug 25, 2022
@corymhall
fix(integ-runner): array arguments aren't recognizing multiple options (
d942324 
#21763)

I don't know why I was specifying `nargs: 1` for array arguments. This
tells yargs to only accept a single argument. You can no provide
multiple options, i.e.

```
yarn integ --regions us-east-1 us-east-2
```

re #20384


----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
josephedward pushed a commit to josephedward/aws-cdk that referenced this issue Aug 30, 2022
@corymhall @josephedward
fix(integ-runner): array arguments aren't recognizing multiple options (
856cff1 
aws#21763)

I don't know why I was specifying `nargs: 1` for array arguments. This
tells yargs to only accept a single argument. You can no provide
multiple options, i.e.

```
yarn integ --regions us-east-1 us-east-2
```

re aws#20384


----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hacker65536 pushed a commit to hacker65536/aws-cdk that referenced this issue Sep 30, 2022
@corymhall @hacker65536
fix(integ-runner): array arguments aren't recognizing multiple options (
9a43532 
aws#21763)

I don't know why I was specifying `nargs: 1` for array arguments. This
tells yargs to only accept a single argument. You can no provide
multiple options, i.e.

```
yarn integ --regions us-east-1 us-east-2
```

re aws#20384


----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@nbaillie nbaillie mentioned this issue Oct 10, 2022
Closed
2 tasks
@corymhall corymhall removed their assignment Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
management/tracking Issues that track a subject or multiple issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mrgrain @corymhall