Reorganize conformance tests #1079
Conversation
| cd packages/connect-conformance && PATH="$(abspath $(BIN)):$(PATH)" node18 ./bin/connectconformance --mode client --conf conformance-node.yaml -v ./bin/conformancenodeclient | ||
| cd packages/connect-conformance && PATH="$(abspath $(BIN)):$(PATH)" node20 ./bin/connectconformance --mode client --conf conformance-node.yaml -v ./bin/conformancenodeclient | ||
| cd packages/connect-conformance && PATH="$(abspath $(BIN)):$(PATH)" node21 ./bin/connectconformance --mode client --conf conformance-node.yaml -v ./bin/conformancenodeclient | ||
| testnodeconformance: $(BUILD)/connect-conformance $(BUILD)/connect-node $(BUILD)/connect-fastify $(BUILD)/connect-express |
There was a problem hiding this comment.
Note that we are intentionally not running these in all Node versions. A future PR will use a Github action matrix to run against multiple Node versions.
There was a problem hiding this comment.
We should also (as follow-ups):
- find a solution for test coverage for
createCallbackClient, - solve the
connectconformanceissue described in Reorganize conformance tests #1079 (review), - move tests we want to keep from connect-node-test to connect-node (similar for connect-web-test) and delete these packages,
- remove all remaining bits of the old crosstests
There was a problem hiding this comment.
Yep, sounds good. Created #1080 to track all these (plus the version-in-two-places issue).
smaye81
changed the title
|
@srikrsna-buf we can land #1066 now whenever and I can merge main into this. |
There was a problem hiding this comment.
Nice, this is taking shape.
I left some comments regarding documentation and code style below.
Regarding the test runner connectconformance:
- Currently, the version is maintained in two places, it would be better to have just a single one.
- The shim downloads the release on the fly and caches it. This won't work for parallel runs, and we'll use parallel runs with turborepo.
We've solved both problems in protobuf-es here. We should solve them here as well, but I don't think it's necessary to do it in this PR.
Co-authored-by: Timo Stamm <ts@timostamm.de>
This PR implements a reorganization of the conformance tests in the repo and how they are run.
The
connect-conformancepackage previously ran all the conformance tests on its own. Now, the conformance tests have been moved to their respective packages andconnect-conformancesimply houses the common functionality used by all the packages to run conformance tests. Conformance tests have been relocated to:connect-node- houses and runs both client and server conformance tests for Connect-Node.connect-web- houses and runs client conformance tests for Connect-Webconnect-express- houses and runs server conformance tests using theconnect-expressplugin.connect-fastify- houses and runs server conformance tests using theconnect-fastifyplugin.connect-cloudflare(new) - houses and runs server conformance tests forconnect-nodeon Cloudflare. This package exists solely for holding Cloudflare conformance tests (at least for now).The remainder of
connect-conformanceshould be no scripts, no tests running. Just a helper package.Note that the Makefile (intentionally) no longer runs the Node-related conformance tests in all supported versions of Node. A follow-up PR will add a Github action matrix that runs these tests in various Node versions.