-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: RuntimePolicies: Fix flake when validating logs #14529
Conversation
retest-runtime |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get the intention, but it looks like this change will skip validating logs in the initial set up phases like -
Expect(vm.SetUpCiliumWithHubble()).To(BeNil())
ExpectCiliumReady(vm)
What if there is a panic in starting the cilium agent?
If there is a panic in starting the agent, the tests won't be able to pass anyway. They will fail with the usual "beforeAll failures". I think that's preferable given it highlights it's not the test itself failing but its preparation. |
But doesn't it defeat the purpose of having
The test should be marked as failed when there are any of these messages in the start-up phase too. What do you think? |
Yes, but in this case, we start Cilium with our default configuration, which we already do in a bunch of other places. So if any of these "bad messages" pop up during startup here, they also will in other places. |
If I understand correctly, we want to validate logs from This is what I have in mind (I'm not sure if you also suggested something similar) -
|
Yes. Your snippet of code is also what I had in mind :-) I'll update the branch. |
Hm, actually, if we do that, we'll get error from the previous shutdown again. We should stop Cilium, register |
For the first test executed as part of RuntimePolicies, the validation of logs may include logs from the BeforeAll block's execution. Given we restart Cilium in that block, it may lead to JoinEP messages in logs if Cilium was stopped while endpoints were being regenerated. An example of such a flake is visible at [1]. That test failed because JoinEP messages were found in logs. Using timestamps from test-output.log, we can check that the validation of logs included the BeforeAll block: 18:58:10 STEP: Running BeforeAll block for EntireTestsuite RuntimePolicies [...] 19:03:27 STEP: Running JustAfterEach block for EntireTestsuite RuntimePolicies time="2021-01-05T19:03:27Z" level=debug msg="running command: sudo journalctl -au cilium --since '316.7919274 seconds ago'" This commit changes our validation of logs in RuntimePolicies to ensure we only validate logs that correspond to the current test, excluding BeforeAll/BeforeEach blocks. 1 - https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/3207/testReport/junit/(root)/Suite-runtime/RuntimePolicies_L3_L4_Checks/ Related: e558100 ("test: RuntimeKVStoreTest: Fix flake when validating logs") Reported-by: Chris Tarazi <chris@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io>
7fdc41d
to
2770766
Compare
test-me-please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
We need to stop and start Cilium separately (vs. doing a restart)
Now that we are stopping cilium process, do we need to update the restart command that we were executing earlier?
This is where cilium is being restarted, but looks the function is called from couple of other places so we might just leave it as-is -
Line 918 in f0d509e
res = s.Exec("sudo systemctl restart cilium") |
The Runtime CI pipeline is passing and reviews are in, so I think this is ready to merge. |
When running runtime tests locally sometimes the test fail as level=error log entries are found that are the result of cilium-agent restarts during provisioning. This is similar to the fix done in cilium#14529. Signed-off-by: Jussi Maki <jussi@isovalent.com>
When running runtime tests locally sometimes the test fail as level=error log entries are found that are the result of cilium-agent restarts during provisioning. This is similar to the fix done in #14529. Signed-off-by: Jussi Maki <jussi@isovalent.com>
[ upstream commit 38994b0 ] When running runtime tests locally sometimes the test fail as level=error log entries are found that are the result of cilium-agent restarts during provisioning. This is similar to the fix done in cilium#14529. Signed-off-by: Jussi Maki <jussi@isovalent.com> Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
[ upstream commit 38994b0 ] When running runtime tests locally sometimes the test fail as level=error log entries are found that are the result of cilium-agent restarts during provisioning. This is similar to the fix done in cilium#14529. Signed-off-by: Jussi Maki <jussi@isovalent.com> Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
[ upstream commit 38994b0 ] When running runtime tests locally sometimes the test fail as level=error log entries are found that are the result of cilium-agent restarts during provisioning. This is similar to the fix done in #14529. Signed-off-by: Jussi Maki <jussi@isovalent.com> Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
[ upstream commit 38994b0 ] When running runtime tests locally sometimes the test fail as level=error log entries are found that are the result of cilium-agent restarts during provisioning. This is similar to the fix done in #14529. Signed-off-by: Jussi Maki <jussi@isovalent.com> Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
For the first test executed as part of RuntimePolicies, the validation of logs may include logs from the
BeforeAll
block's execution. Given we restart Cilium in that block, it may lead toJoinEP
messages in logs if Cilium was stopped while endpoints were being regenerated.An example of such a flake is visible at Cilium-PR-Runtime-4.9#3207. That test failed because
JoinEP
messages were found in logs. Using timestamps fromtest-output.log
, we can check that the validation of logs included theBeforeAll
block:This pull request changes our validation of logs in RuntimePolicies to ensure we only validate logs that correspond to the current test, excluding
BeforeAll
/BeforeEach
blocks.Related: e558100 ("test: RuntimeKVStoreTest: Fix flake when validating logs")