Skip to content

Commit

Permalink
Merge branch 'master' into dependabot/npm_and_yarn/vue-3.3.8
Browse files Browse the repository at this point in the history
  • Loading branch information
@ozyx
ozyx committed Nov 9, 2023
2 parents 8fe8695 + 4b0abdf commit c72b25c
Show file tree
Hide file tree
Showing 9 changed files with 285 additions and 123 deletions.
2 changes: 2 additions & 0 deletions .github/dependabot.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ updates:
schedule:
interval: 'weekly'
open-pull-requests-limit: 10
rebase-strategy: 'disabled'
labels:
- 'pr:daveit'
- 'pr:e2e'
Expand Down Expand Up @@ -34,6 +35,7 @@ updates:
directory: '/'
schedule:
interval: 'daily'
rebase-strategy: 'disabled'
labels:
- 'pr:daveit'
- 'type:maintenance'
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ Each test suite generates a report in CircleCI. For a complete overview of testi

Our code coverage is generated during the runtime of our unit, e2e, and visual tests. The combination of those reports is published to [codecov.io](https://app.codecov.io/gh/nasa/openmct/)

For more on the specifics of our code coverage setup, [see](TESTING.md#code-coverage)

# Glossary

Certain terms are used throughout Open MCT with consistent meanings
Expand Down
83 changes: 77 additions & 6 deletions TESTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,85 @@ Documentation located [here](./e2e/README.md)

## Code Coverage

* 100% statement coverage is achievable and desirable.
It's up to the individual developer as to whether they want to add line coverage in the form of a unit test or e2e test.

Codecov.io will combine each of the above commands with [Codecov.io Flags](https://docs.codecov.com/docs/flags). Effectively, this allows us to combine multiple reports which are run at various stages of our CI Pipeline or run as part of a parallel process.
Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in GitHub PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage.

### Karma-istanbul

Line coverage is generated by our `karma-coverage-istanbul-reporter` package as defined in our `karma.conf.js` file:

```js
coverageIstanbulReporter: {
fixWebpackSourcePaths: true,
skipFilesWithNoCoverage: true,
dir: 'coverage/unit', //Sets coverage file to be consumed by codecov.io
reports: ['lcovonly']
},
```

Once the file is generated, it can be published to codecov with

```json
"cov:unit:publish": "codecov --disable=gcov -f ./coverage/unit/lcov.info -F unit",
```

### e2e
The e2e line coverage is a bit more complex than the karma implementation. This is the general sequence of events:

1. Each e2e suite will start webpack with the ```npm run start:coverage``` command with config `webpack.coverage.js` and the `babel-plugin-istanbul` plugin to generate code coverage during e2e test execution using our custom [baseFixture](./baseFixtures.js).
1. During testcase execution, each e2e shard will generate its piece of the larger coverage suite. **This coverage file is not merged**. The raw coverage file is stored in a `.nyc_report` directory.
1. [nyc](https://github.com/istanbuljs/nyc) converts this directory into a `lcov` file with the following command `npm run cov:e2e:report`
1. Most of the tests are run in the '@stable' configuration and focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:stable:publish`.
1. The rest of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR.

This e2e coverage is combined with our unit test report to give a comprehensive (if flawed) view of line coverage.

### Limitations in our code coverage reporting
Our code coverage implementation has some known limitations:
- [Variability](https://github.com/nasa/openmct/issues/5811)
- [Accuracy](https://github.com/nasa/openmct/issues/7015)
- [Vue instrumentation gaps](https://github.com/nasa/openmct/issues/4973)

## Troubleshooting CI
The following is an evolving guide to troubleshoot CI and PR issues.

### Github Checks failing
There are a few reasons that your GitHub PR could be failing beyond simple failed tests.
* Required Checks. We're leveraging required checks in GitHub so that we can quickly and precisely control what becomes and informational failure vs a hard requirement. The only way to determine the difference between a required vs information check is check for the `(Required)` emblem next to the step details in GitHub Checks.
* Not all required checks are run per commit. You may need to manually trigger addition GitHub checks with a `pr:<label>` label added to your PR.

### Flaky tests
There are two ways to know if a test on your branch is historically flaky:
1. `deploysentinel`'s PR comment bot to give an accurate and historical view of e2e flakiness. Check your PR for a view of the test failures and flakes (with link to the failing test). Note: only a 7 day window of flake is available.
2. (CircleCI's test insights feature)[https://circleci.com/blog/introducing-test-insights-with-flaky-test-detection/] collects historical data about the individual test results for both unit and e2e tests. Note: only a 14 day window of flake is available.

### Local=Pass and CI=Fail
Although rare, it is possible that your test can pass locally but fail in CI.

#### Busting Cache
In certain circumstances, the CircleCI cache can become stale. In order to bust the cache, we've implemented a runtime boolean parameter in Circle CI creatively name BUST_CACHE. To execute:
1. Navigate to the branch in Circle CI believed to have stale cache.
1. Click on the 'Trigger Pipeline' button.
1. Add Parameter -> Parameter Type = boolean , Name = BUST_CACHE ,Value = true
1. Click 'Trigger Pipeline'

#### Run tests in the same container as CI

In extreme cases, tests can fail due to the constraints of running within a container. To execute tests in exactly the same way as run in CircleCI.

```sh
// Replace {X.X.X} with the current Playwright version
// from our package.json or circleCI configuration file
docker run --rm --network host --cpus="2" -v $(pwd):/work/ -w /work/ -it mcr.microsoft.com/playwright:v{X.X.X}-focal /bin/bash
npm install
```

At this point, you're running inside the same container and with 2 cpu cores. You can specify the unit tests:
```sh
npm run test
```
or e2e tests:

Our code coverage implementation has two known limitations:
- [Variability and accuracy](https://github.com/nasa/openmct/issues/5811)
- [Vue instrumentation](https://github.com/nasa/openmct/issues/4973)
```sh
npx playwright test --config=e2e/playwright-ci.config.js --project=chrome --grep <the testcase name>
```
16 changes: 4 additions & 12 deletions e2e/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -490,15 +490,7 @@ Our e2e code coverage is captured and combined with our unit test coverage. For
#### Generating e2e code coverage
Code coverage is collected during test execution using our custom [baseFixture](./baseFixtures.js). The raw coverage files are stored in a `.nyc_report` directory to be converted into a lcov file with the following [nyc](https://github.com/istanbuljs/nyc) command:
```npm run cov:e2e:report```
At this point, the nyc linecov report can be published to [codecov.io](https://about.codecov.io/) with the following command:
```npm run cov:e2e:stable:publish``` for the stable suite running in ubuntu.
or
```npm run cov:e2e:full:publish``` for the full suite running against all available platforms.
Please read more about our code coverage [here](../TESTING.md#code-coverage)
## Other
Expand Down Expand Up @@ -548,10 +540,10 @@ A single e2e test in Open MCT is extended to run:
- How is Open MCT extending default Playwright functionality?
- What about Component Testing?
### Troubleshooting
### e2e Troubleshooting
Please follow the general guide troubleshooting in [the general troubleshooting doc](../TESTING.md#troubleshooting-ci)
- Why is my test failing on CI and not locally?
- How can I view the failing tests on CI?
- Tests won't start because 'Error: <http://localhost:8080/># is already used...'
This error will appear when running the tests locally. Sometimes, the webserver is left in an orphaned state and needs to be cleaned up. To clear up the orphaned webserver, execute the following from your Terminal:
```lsof -n -i4TCP:8080 | awk '{print$2}' | tail -1 | xargs kill -9```
Expand Down
116 changes: 105 additions & 11 deletions e2e/tests/functional/plugins/lad/lad.e2e.spec.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,21 +25,24 @@ const {
createDomainObjectWithDefaults,
setStartOffset,
setFixedTimeMode,
setRealTimeMode
setRealTimeMode,
openObjectTreeContextMenu
} = require('../../../../appActions');

test.describe('Testing LAD table configuration', () => {
let ladTable;
let sineWaveObject;
test.beforeEach(async ({ page }) => {
await page.goto('./', { waitUntil: 'domcontentloaded' });

// Create LAD table
const ladTable = await createDomainObjectWithDefaults(page, {
ladTable = await createDomainObjectWithDefaults(page, {
type: 'LAD Table',
name: 'Test LAD Table'
});

// Create Sine Wave Generator
await createDomainObjectWithDefaults(page, {
sineWaveObject = await createDomainObjectWithDefaults(page, {
type: 'Sine Wave Generator',
name: 'Test Sine Wave Generator',
parent: ladTable.uuid
Expand All @@ -50,31 +53,51 @@ test.describe('Testing LAD table configuration', () => {
test('in edit mode, LAD Tables provide ability to hide columns', async ({ page }) => {
// Edit LAD table
await page.locator('[title="Edit"]').click();

// // Expand the 'My Items' folder in the left tree
// await page.locator('.c-tree__item__view-control.c-disclosure-triangle').click();
// // Add the Sine Wave Generator to the LAD table and save changes
// await page.dragAndDrop('role=treeitem[name=/Test Sine Wave Generator/]', '.c-lad-table-wrapper');
// select configuration tab in inspector
await page.getByRole('tab', { name: 'LAD Table Configuration' }).click();

// make sure headers are visible initially
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Units' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Type' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// hide timestamp column
await page.getByLabel('Timestamp').uncheck();
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Units' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Type' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// hide units & type column
await page.getByLabel('Units').uncheck();
await page.getByLabel('Type').uncheck();
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Units' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Type' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// hide WATCH column
await page.getByLabel('WATCH').uncheck();
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Units' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Type' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// save and reload and verify they columns are still hidden
await page.locator('button[title="Save"]').click();
Expand All @@ -83,6 +106,11 @@ test.describe('Testing LAD table configuration', () => {
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Units' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Type' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// Edit LAD table
await page.locator('[title="Edit"]').click();
Expand All @@ -93,25 +121,41 @@ test.describe('Testing LAD table configuration', () => {
await expect(page.getByRole('cell', { name: 'Units' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Type' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// save and reload and make sure only timestamp is still visible
// save and reload and make sure timestamp is still visible
await page.locator('button[title="Save"]').click();
await page.locator('text=Save and Finish Editing').click();
await page.reload();
await expect(page.getByRole('cell', { name: 'Units' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Type' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// Edit LAD table
await page.locator('[title="Edit"]').click();
await page.getByRole('tab', { name: 'LAD Table Configuration' }).click();

// show units and type columns
// show units, type, and WATCH columns
await page.getByLabel('Units').check();
await page.getByLabel('Type').check();
await page.getByLabel('WATCH').check();
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Units' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Type' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// save and reload and make sure all columns are still visible
await page.locator('button[title="Save"]').click();
Expand All @@ -120,6 +164,56 @@ test.describe('Testing LAD table configuration', () => {
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Units' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Type' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();
});

test('When adding something without Units, do not show Units column', async ({ page }) => {
// Create Sine Wave Generator
await createDomainObjectWithDefaults(page, {
type: 'Event Message Generator',
parent: ladTable.uuid
});

await page.goto(ladTable.url);

// Edit LAD table
await page.getByLabel('Edit').click();
await page.getByRole('tab', { name: 'LAD Table Configuration' }).click();

// make sure Sine Wave headers are visible initially too
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Units' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Type' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeVisible();

// save and reload and verify they columns are still hidden
await page.getByLabel('Save').click();
await page.getByRole('listitem', { name: 'Save and Finish Editing' }).click();

// Remove Sin Wave Generator
openObjectTreeContextMenu(page, sineWaveObject.url);
await page.getByRole('menuitem', { name: /Remove/ }).click();
await page.getByRole('button', { name: 'OK' }).click();

// Ensure Units & Limit columns are gone
// as Event Generator don't have them
await page.goto(ladTable.url);
await expect(page.getByRole('cell', { name: 'Timestamp' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Type' })).toBeVisible();
await expect(page.getByRole('cell', { name: 'Units' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WATCH' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'WARNING' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'DISTRESS' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'CRITICAL' })).toBeHidden();
await expect(page.getByRole('cell', { name: 'SEVERE' })).toBeHidden();
});

test("LAD Tables don't allow selection of rows but does show context click menus", async ({
Expand Down
6 changes: 3 additions & 3 deletions package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"@percy/playwright": "1.0.4",
"@playwright/test": "1.39.0",
"@types/eventemitter3": "1.2.0",
"@types/jasmine": "4.3.4",
"@types/jasmine": "5.1.2",
"@types/lodash": "4.14.192",
"@vue/compiler-sfc": "3.3.8",
"babel-loader": "9.1.0",
Expand All @@ -30,7 +30,7 @@
"eslint-plugin-playwright": "0.12.0",
"eslint-plugin-prettier": "4.2.1",
"eslint-plugin-simple-import-sort": "10.0.0",
"eslint-plugin-unicorn": "48.0.1",
"eslint-plugin-unicorn": "49.0.0",
"eslint-plugin-vue": "9.18.1",
"eslint-plugin-you-dont-need-lodash-underscore": "6.13.0",
"eventemitter3": "1.2.0",
Expand Down Expand Up @@ -76,7 +76,7 @@
"vue": "3.3.8",
"vue-eslint-parser": "9.3.2",
"vue-loader": "16.8.3",
"webpack": "5.88.0",
"webpack": "5.89.0",
"webpack-cli": "5.1.1",
"webpack-dev-server": "4.15.1",
"webpack-merge": "5.10.0"
Expand Down

0 comments on commit c72b25c

Please sign in to comment.