Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: improve test filtering to use mocha hooks #3095

Merged
merged 11 commits into from Jan 10, 2022

Conversation

nbbeeken
Copy link
Contributor

@nbbeeken nbbeeken commented Jan 6, 2022

Description

What is changing?

Use a beforeEach hook to run test filters.

Also: remove docker stuff, move the sdam_viz tool to etc with out other helper scripts, corrected small type issue in unified test schema, enabled test sorting

What is the motivation for this change?

  • xuint files will now all have the same test names
  • tests are always searchable in logs
  • LOG_FILTER_REASON can be enabled to view which filter made a test skip

Follow up work

Our legacy spec runner still "multiplies" the tests by runOn requirement and then relies on the filter to skip the ones that don't apply. Fixing this would drastically reduce the test count without actually removing tests.
Now that we have a standardized runOn requirement documented in the unified format, it would be nice to refactor our filtering metadata to match that, and reuse the logic in topologySatisfies in the unified runner.

Double check the following

  • Ran npm run check:lint script
  • Self-review completed using the steps outlined here
  • PR title follows the correct format: <type>(NODE-xxxx)<!>: <description>
  • Changes are covered by tests
  • New TODOs have a related JIRA ticket

durran
durran previously requested changes Jan 7, 2022
Copy link
Member

@durran durran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests

describe('BSON size limits and batch splitting', function () {
need to be filtered to run on only server version 4.2 and higher to not fail with "Unknown $jsonSchema keyword: encrypt"

@durran durran added the Team Review Needs review from team label Jan 7, 2022
@nbbeeken nbbeeken marked this pull request as ready for review January 7, 2022 15:49
baileympearson
baileympearson previously approved these changes Jan 7, 2022
Copy link
Contributor

@baileympearson baileympearson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice work Neal!

This will also nice to be have for the test reorg work, because now we can clearly see that the correct number of tests are being considered with each test run 😄

Copy link
Contributor

@dariakp dariakp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few thoughts here

test/readme.md Outdated Show resolved Hide resolved
test/readme.md Show resolved Hide resolved
test/tools/runner/index.js Outdated Show resolved Hide resolved
@dariakp
Copy link
Contributor

dariakp commented Jan 7, 2022

@nbbeeken One more question, you mention the legacy spec runner follow up work - do we already have a ticket for that?

Co-authored-by: Daria Pardue <daria.pardue@mongodb.com>
@nbbeeken
Copy link
Contributor Author

nbbeeken commented Jan 7, 2022

@nbbeeken One more question, you mention the legacy spec runner follow up work - do we already have a ticket for that?

Yep! here: https://jira.mongodb.org/browse/NODE-3817

- Added type defintions that describe our metadata overloads, new tsconfig imports this, it makes it so vscode will auto discover the extensions
- Added skipReason to the unified runner
@nbbeeken nbbeeken requested a review from dariakp January 10, 2022 13:38
test/readme.md Show resolved Hide resolved
@dariakp dariakp dismissed durran’s stale review January 10, 2022 19:56

Review addressed

@dariakp dariakp merged commit 91c108e into main Jan 10, 2022
@dariakp dariakp deleted the chore/test-filter-with-hook branch January 10, 2022 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team Review Needs review from team
Projects
None yet
4 participants