-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MySQL + Kine Conformance Checks Fail #10023
Comments
This appears to be related to watch requests and response pagination, I suspect the fixes from these recent kine PRs may need some additional tweaking to properly group requests in the way the apiserver expects: |
@VestigeJ do you know if this can be reproduced on a single-node instance of k3s, or does it require 3+1 to fail? |
I haven't tried it on a single host at all we don't usually try to do conformance tests without a 3-1 cluster. |
It appears that the conformance tests now expect the datastore to list things sorted by key. Previously this wasn't required, and we closed out issues where people complained about it: There are now tests that create a large number (400) of resources, and then list them in smaller (17) chunks, and test that the items come out sorted by name (the 7th item returned should be named template-0007, for example). Ref: This fails because kine does not sort returned items by key, as etcd does. They are instead sorted by revision, as this is the native order returned by the SQL query. |
This error seems to stem from a similar problem, related to paginating with expired continue tokens. However in this case the problem seems to be related to consistency with the remaining item count. This was supposed to have been fixed by k3s-io/kine#271 but perhaps there are corner cases remaining with pagination and key order. Ref: |
An error is being bubbled up here, but I'm not sure if it's coming from the apiserver, or kine. I think the code in this test is small enough that we could just run it directly though. Ref: EDIT: This FeatureGate is still Alpha and is disabled by default, I'm not sure why this test is being run by default. The server is sending back:
With the FeatureGate enabled, the test passes:
|
All of the failing tests should be addressed by the linked kine PR. Note: any tests with |
Validated using the latest RCs on a single node running sonobuoy with the sig-api-machinery and conformance results. Attention to the fact that the shown failures are expected to fail when tested against a single node. $ kgn
$ sono_results
|
Environmental Info:
K3s Version:
Any current branch
Node(s) CPU architecture, OS, and Version:
not applicable the same bug is present regardless of the OS.
Cluster Configuration:
3 servers 1 agent
1 external MySQL DB mysql Ver 8.0.36 for Linux on x86_64 (MySQL Community Server - GPL)
Describe the bug:
Failing conformance checks only when MySQL is the external DB - the exact same setup with postgres passes conformance checks just fine.
Steps To Reproduce:
sonobuoy run --mode=certified-conformance // to expedite the tests you could run --e2e-focus=sig-api-machinery or the string of the failed tests --e2e-focus="Servers with support for API chunking should support continue listing from the last key if the original version has been compacted away, though the list is inconsistent"
Expected behavior:
all conformance checks pass
Actual behavior:
specifically it seems that various cascading failures can occur specifically with the sig-api-machinery suite of tests.
Additional context / logs:
These commands in the test suite when run manually pass
Important note that MySQL itself does not seem bottlenecked or overburdened by the cluster in this setup
The text was updated successfully, but these errors were encountered: