Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify processing of known failing tests #754

Merged
merged 5 commits into from Jan 13, 2024
Merged

Conversation

smaye81
Copy link
Member

@smaye81 smaye81 commented Jan 12, 2024

Currently, when specifying a known-failing file, the tests that are listed in the file are matched against the list of test cases. If there are tests or patterns that do not match, an error is thrown.

However, this causes an issue when trying to mark one of the gRPC tests as known-failing because the gRPC test names are modified via the filterGRPCImplTestCases function. Tests have (grpc server impl), (grpc client impl) or (grpc impl) added as the penultimate value before the test name, depending on how they are run. But since this is done after the known-failing tests are matched, it is impossible to mark one of these gRPC tests as failing.

So, this PR modifies the calculation of the test names a bit by creating a list of all known test permutations and then using that to match against any known-failing tests listed.

Note that there is an example known-failing file and a test deliberately set up to fail just to see how this works in action. These two files will be removed from this PR before it is merged.

To run with these changes, use the command:

go run cmd/connectconformance/main.go -v --known-failing internal/app/connectconformance/testsuites/known_failing.txt --mode client -- .tmp/bin/referenceclient

@smaye81 smaye81 requested a review from jhump January 12, 2024 03:06
Copy link
Member

@jhump jhump left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a lot of this should really be the responsibility of the test case library, but this seems good for now.

@smaye81 smaye81 marked this pull request as ready for review January 13, 2024 00:25
@smaye81 smaye81 merged commit f920cb1 into main Jan 13, 2024
4 checks passed
@smaye81 smaye81 deleted the sayers/known_failing branch January 13, 2024 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants