Skip to content

Add Kubernetes API coverage test consuming traces #20181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jul 7, 2025
Merged

Add Kubernetes API coverage test consuming traces #20181

merged 4 commits into from
Jul 7, 2025

Conversation

AwesomePatrol
Copy link
Contributor

@AwesomePatrol AwesomePatrol commented Jun 17, 2025 •
edited
Loading

It pulls a file from testdata with traces collected by Jaeger and then iterates over them in order to group them into those which use the contract interface, are known (and usually valid) uses outside of it. Later it can further analyze the parameters used over the run of Kubernetes conformance tests.

=== RUN   TestInterfaceUse/otlp_etcd-2025-07-03.json
    coverage_test.go:55: Traces found: 64228
    coverage_test.go:66: API calls by gRPC method: map[etcdserverpb.KV/Compact:50 etcdserverpb.KV/R
ange:36761 etcdserverpb.KV/Txn:25450 etcdserverpb.Lease/LeaseGrant:1554 etcdserverpb.Maintenance/St
atus:412 etcdserverpb.Watch/Watch:1]
=== RUN   TestInterfaceUse/otlp_etcd-2025-07-04.json
    coverage_test.go:55: Traces found: 100000
    coverage_test.go:66: API calls by gRPC method: map[etcdserverpb.KV/Compact:84 etcdserverpb.KV/Range:56498 etcdserverpb.KV/Txn:40247 etcdserverpb.Lease/LeaseGrant:2503 etcdserverpb.Maintenance/Status:668]
--- FAIL: TestInterfaceUse (5.24s)
    --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json (2.06s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/etcdserverpb.KV/Range (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/etcdserverpb.KV/Txn (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/etcdserverpb.KV/Compact (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/etcdserverpb.Watch/Watch (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/etcdserverpb.Lease/LeaseGrant (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/etcdserverpb.Maintenance/Status (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-03.json/only_expected_methods_were_called (0.00s)
    --- FAIL: TestInterfaceUse/otlp_etcd-2025-07-04.json (3.19s)
        --- FAIL: TestInterfaceUse/otlp_etcd-2025-07-04.json/etcdserverpb.Watch/Watch (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-04.json/etcdserverpb.Lease/LeaseGrant (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-04.json/etcdserverpb.Maintenance/Status (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-04.json/etcdserverpb.KV/Range (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-04.json/etcdserverpb.KV/Txn (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-04.json/etcdserverpb.KV/Compact (0.00s)
        --- PASS: TestInterfaceUse/otlp_etcd-2025-07-04.json/only_expected_methods_were_called (0.00s)

@k8s-ci-robot
Copy link

Hi @AwesomePatrol. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@AwesomePatrol AwesomePatrol mentioned this pull request Jun 17, 2025
Open
9 tasks
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch from 270aa53 to b8356e4 Compare June 24, 2025 10:01
@AwesomePatrol AwesomePatrol marked this pull request as ready for review June 27, 2025 08:20
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch from b8356e4 to 4cd188f Compare June 27, 2025 09:14
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
serathius
serathius reviewed Jun 27, 2025
View reviewed changes
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch from 4cd188f to 8b7e4c0 Compare July 3, 2025 14:12
AwesomePatrol
AwesomePatrol commented Jul 3, 2025
View reviewed changes
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch from 8b7e4c0 to d4a9898 Compare July 3, 2025 14:15
serathius
serathius reviewed Jul 4, 2025
View reviewed changes
serathius
serathius reviewed Jul 4, 2025
View reviewed changes
tests/robustness/coverage/README.md Outdated
```

2\. Build and run Conformance tests from Kubernetes repository (this usually
takes 30-40m or will time out after 1 hour). It needs to be run from Kubernetes git repository:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add note that Conformance tests could be replaced with user prefered way to exercise K8s API. Conformance tests were just selected because ... (provide reasons).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to state that the goal is exercising Kubernetes API surface. Conformance tests are used, because they are a good starting point to achieve this, so I marked them as an example

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm saying we should document that. In the step.

serathius
serathius reviewed Jul 4, 2025
View reviewed changes
serathius
serathius reviewed Jul 4, 2025
View reviewed changes
serathius
serathius reviewed Jul 4, 2025
View reviewed changes
serathius
serathius reviewed Jul 4, 2025
View reviewed changes
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch 2 times, most recently from c08bde1 to edc8fa9 Compare July 4, 2025 11:09
serathius
serathius reviewed Jul 4, 2025
View reviewed changes
@AwesomePatrol
Add Kubernetes API coverage test consuming traces
f1a6777 
It pulls a file from testdata with traces collected by Jaeger and then
iterates over them in order to group them by Etcd gRPC method used.

Signed-off-by: Aleksander Mistewicz <amistewicz@google.com>
@AwesomePatrol
Use already imported library instead
b883f0d 
All tools needed to process traces are already available in etcd/tests.

Signed-off-by: Aleksander Mistewicz <amistewicz@google.com>
@AwesomePatrol
Split test into subtests by method name
dc7d95e 
It makes it easier to know what methods were used from the tests
outputs.

Signed-off-by: Aleksander Mistewicz <amistewicz@google.com>
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch from 9554f9c to 7daf63f Compare July 7, 2025 11:39
@AwesomePatrol
Print call stats for API coverage test in a nice table
587388f 
Signed-off-by: Aleksander Mistewicz <amistewicz@google.com>
@AwesomePatrol AwesomePatrol force-pushed the test-for-contract-usage branch from 7daf63f to 587388f Compare July 7, 2025 11:44
@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AwesomePatrol, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@serathius serathius merged commit a93551f into etcd-io:main Jul 7, 2025
5 checks passed
@MadhavJivrajani MadhavJivrajani mentioned this pull request Jul 7, 2025
Merged
@AwesomePatrol AwesomePatrol mentioned this pull request Jul 8, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@serathius serathius serathius approved these changes

Assignees
No one assigned
Labels
approved area/robustness-testing area/testing needs-ok-to-test size/L
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AwesomePatrol @k8s-ci-robot @serathius