-
Notifications
You must be signed in to change notification settings - Fork 387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: Updated versioned tests to support upcoming tooling #2147
Merged
bizob2828
merged 3 commits into
newrelic:main
from
jsumners-nr:versioned-tests-meta-update
Apr 16, 2024
Merged
Changes from 2 commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# Versioned Tests | ||
|
||
The subdirectories within this directory represent test suites that verify | ||
our agent compatibility across multiple versions of our supported modules. | ||
When these tests are run, through our versioned test runner tool, each suite | ||
is run multiple times utilizing different versions of the module that satisfy | ||
configured version constraints. | ||
|
||
For example, if a supported module has releases `1.0.0` through `1.10.0`, with | ||
a version for each minor between `0` and `10`, then our versioned test runner | ||
will run the test suite across a sampling of versions in that range, e.g. | ||
versions `1.0.0`, `1.3.0`, and `1.10.0`. | ||
|
||
## Versioned Tests `package.json` | ||
|
||
The versioned test runner reads a `package.json` in each test suite. This | ||
`package.json` describes the constraints for the suite and has a few properties | ||
that are specific to our versioned test runner. The following is a | ||
[jsonc](https://en.wikipedia.org/wiki/JSON#JSONC) representation of such a | ||
`package.json` that describes the unique properties: | ||
|
||
```jsonc | ||
{ | ||
// `name` is typically ignored. | ||
"name": "module-tests", | ||
|
||
// `targets` indicates which modules are being verified by the test suite. | ||
// This is utilized by tooling to build a compatibility document. There _must_ | ||
// be at least one value that matches a dependency name in a subsequent | ||
// `tests` block. If this suite is verifying the compatibility of multiple | ||
// modules, include all such module names in the array. | ||
"targets": ["module"], | ||
|
||
// `version` is ignored. | ||
"version": "0.0.0", | ||
|
||
// `private` should be set to `true`. | ||
"private": true, | ||
|
||
// `tests` contains blocks that describe the tests the versioned test runner | ||
// should run for this suite and under what constraints. Each block will | ||
// result in at least one test run by the test runner. | ||
"tests": [ | ||
{ | ||
// `engines` is a typical package.json engines block. | ||
"engines": { | ||
// `node` indicates which versions of Node.js should be used to run | ||
// this test block. Typically, a basic `>=` qualifier will be used, but | ||
// a static version is also likely. If the version of Node.js being | ||
// used to run the suite does not match the contraint, then the test | ||
// block will be skipped. | ||
"node": ">= 18" | ||
}, | ||
|
||
// `dependencies` lists dependencies that a needed in order to execute | ||
// the test block. In most cases, only the module under test will be | ||
// present. | ||
"dependencies": { | ||
// For the dependency named "module-name", run the suite with samples | ||
// from the provided semver range (https://docs.npmjs.com/cli/v6/using-npm/semver#advanced-range-syntax). | ||
// | ||
// The minimum version across all test blocks will be utilized to | ||
// indicate the minimum supported version of a module by our agent | ||
// if that module name is listed in the top-level `targets` property. | ||
// | ||
// Note: this may also be an object with a special format. See the next | ||
// example block. | ||
"module-name": ">=1.0.0 <2.0.0" | ||
}, | ||
|
||
// `files` lists out the test files that comprise the test suite for the | ||
// current block. | ||
"files": [ | ||
"test-one.tap.js", | ||
"test-two.tap.js" | ||
] | ||
}, | ||
|
||
// This example block will only run on Node.js 20.x. Pay attention | ||
// to the "dependencies" section for a special depedency declaration | ||
// supported by our versioned test runner. | ||
{ | ||
"engines": { "node": "20" }, | ||
"dependencies": { | ||
"module-name": { | ||
// Again, a standard semver range to indicate the versions of the | ||
// module to sample from. | ||
"versions": ">=1.0.0 <2.0.0", | ||
|
||
// How many samples, across the provided versions range, to conduct | ||
// when testing with this test block. It should be a string value, | ||
// although some of our tooling will likely coerce it to an integer. | ||
"samples": "2" | ||
} | ||
} | ||
} | ||
] | ||
} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "ampqlib-tests", | ||
"targets": ["amqplib"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "bunyan-tests", | ||
"targets": ["bunyan"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "connect-tests", | ||
"targets": ["connect"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "express-esm-tests", | ||
"targets": ["express"], | ||
"version": "0.0.0", | ||
"type": "module", | ||
"private": true, | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "express-tests", | ||
"targets": ["express"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "fastify-tests", | ||
"targets": ["fastify"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "generic-pool-tests", | ||
"targets": ["generic-pool"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "grpc-tests", | ||
"targets": ["@grpc/grpc-js"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "hapi-tests", | ||
"targets": ["@hapi/hapi"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "langchain-tests", | ||
"targets": ["@langchain/core"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"engines": { | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "mongodb-esm-tests", | ||
"targets": ["mongodb"], | ||
"version": "0.0.0", | ||
"type": "module", | ||
"private": true, | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "mongodb-tests", | ||
"targets": ["mongodb"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "mysql-tests", | ||
"targets": ["mysql"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "mysql2-tests", | ||
"targets": ["mysql2"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "nestjs-tests", | ||
"targets": ["@nestjs/cli"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "openai-tests", | ||
"targets": ["openai"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"engines": { | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "pino-tests", | ||
"targets": ["pino"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"tests": [ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
{ | ||
"name": "prisma-tests", | ||
"targets": ["@prisma/client"], | ||
"version": "0.0.0", | ||
"private": true, | ||
"engines": { | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we don't instrument cls, i think this is just a test to make sure we don't affect it. you can prob omit this one