[FAIL] Kubevirt Virtual Machines when live migrated [It] with pre-copy should keep connectivity #3986

trozet · 2023-10-30T15:49:41Z

Seeing failures with this test still:

Kubevirt Virtual Machines when live migrated [It] with pre-copy should keep connectivity
/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/kubevirt.go:656

  [FAILED] Oct 25 23:45:54.083: Timed out after 60.010s.
  worker1: after live migrate, migration #1: Check tcp connection is not broken
  Expected success, but got an error:
      <*errors.errorString | 0xc0003bb4a0>: 
      http connection to virtual machine was broken
      {
          s: "http connection to virtual machine was broken",
      }
  In [It] at: /home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/kubevirt.go:296 @ 10/25/23 23:45:54.084

  There were additional failures detected.  To view them in detail run ginkgo -vv

https://github.com/ovn-org/ovn-kubernetes/actions/runs/6646915605/job/18062050736?pr=3979

The text was updated successfully, but these errors were encountered:

npinaeva · 2023-11-02T16:16:31Z

one more https://github.com/ovn-org/ovn-kubernetes/actions/runs/6734120950/job/18305537344?pr=3995

Ovn-kubernetes upstream issues labeled with "ci-flake" are not tracked in JIRA: let's create a jira user story under the SDN-4175 epic for each such open issue: - the assignee is kept whenever possible, otherwise bbennett is used. - the summary (title) of these jira cards will start with upstream-$GITHUB_ISSUE_ID, where $GITHUB_ISSUE_ID is the ID found in the URL of the issue itself (e.g. upstream-3986 for ovn-org/ovn-kubernetes#3986) A list of what jira stories have been created is printed to stdout, as well as a list of stories in the SDN-4175 epic whose status doesn't match the github issue they're tracking. The new "-g" ("--process-github-issues") input parameter executes this new functionality, or the script alone without any input parameters. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>

Ovn-kubernetes upstream issues labeled with "ci-flake" are not tracked in JIRA: let's create a jira user story under the SDN-4175 epic for each such open issue: - the assignee is kept whenever possible, otherwise bbennett is used. - the summary (title) of these jira cards will start with upstream-$GITHUB_ISSUE_ID, where $GITHUB_ISSUE_ID is the ID found in the URL of the issue itself (e.g. upstream-3986 for ovn-org/ovn-kubernetes#3986) A list of which jira stories have been created is printed to stdout, as well as a list of stories in the SDN-4175 epic whose status doesn't match the status of the github issue they're tracking. The new "-g" ("--process-github-issues") input parameter executes this new functionality, or the script alone without any input parameters. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>

martinkennelly · 2023-12-07T08:43:22Z

Flake seen here: #4040 ( https://github.com/ovn-org/ovn-kubernetes/actions/runs/7120153599/job/19387670459?pr=4040 )

flavio-fernandes · 2023-12-13T20:17:50Z

Flake seen here: #4050 ( https://github.com/ovn-org/ovn-kubernetes/actions/runs/7199631620/job/19612611460?pr=4050 )

tssurya · 2024-02-07T09:18:30Z

one more here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/7789676840/job/21242664047?pr=4100 :/

#4100

@maiqueb or @qinqon : can one of you please fix this?

qinqon · 2024-02-07T09:46:14Z

one more here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/7789676840/job/21242664047?pr=4100 :/

#4100

@maiqueb or @qinqon : can one of you please fix this?

@tssurya we are preparing a fix for it, it will be ready to merge today.

tssurya · 2024-02-07T12:27:22Z

one more: https://github.com/ovn-org/ovn-kubernetes/actions/runs/7813480320/job/21313563225?pr=4100#logs
ack thanks @qinqon !!

qinqon · 2024-02-07T13:56:00Z

@tssurya we are going to start small #4140

We have some other ideas but we will go step by step.

martinkennelly · 2024-02-08T10:21:13Z

Thanks @qinqon

seen again here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/7814875289/job/21318048354?pr=4061

qinqon · 2024-02-08T11:14:30Z

Thanks @qinqon

seen again here: https://github.com/ovn-org/ovn-kubernetes/actions/runs/7814875289/job/21318048354?pr=4061

Adding error so I can track if it's always the same.

 [FAILED] Timed out after 60.001s.
  worker1: after live migration to node owning the subnet: Check tcp connection is not broken
  Expected success, but got an error:
      <*errors.errorString | 0xc000f52b70>: 
      http connection to virtual machine was broken
      {
          s: "http connection to virtual machine was broken",
      }
  In [It] at: /home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/kubevirt.go:330 @ 02/07/24 13:22:07.982

qinqon · 2024-02-08T13:58:57Z

I am reproducing this error at my fork running 12 jobs in parallel and a more simple tcp server so is easier to debug
qinqon#10

qinqon#10 (comment)

qinqon · 2024-02-09T08:43:19Z

After around 25 runs show this failure with the stabilization PR
https://github.com/ovn-org/ovn-kubernetes/actions/runs/7840556162/job/21395793644?pr=4145

2024-02-09T07:20:48.1135328Z   Latency metrics for node ovn-worker3
2024-02-09T07:20:48.1136685Z   �[1mSTEP:�[0m Destroying namespace "kv-live-migration-923" for this suite. �[38;5;243m@ 02/09/24 07:20:48.113�[0m
2024-02-09T07:20:48.1181879Z �[38;5;9m• [FAILED] [419.375 seconds]�[0m
2024-02-09T07:20:48.1184612Z �[0mKubevirt Virtual Machines �[38;5;243mwhen live migration �[38;5;9m�[1m[It] with pre-copy succeeds, should keep connectivity�[0m
2024-02-09T07:20:48.1187469Z �[38;5;243m/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/kubevirt.go:836�[0m
2024-02-09T07:20:48.1188240Z 
2024-02-09T07:20:48.1190075Z   �[38;5;9m[FAILED] worker1: after live migration for the second time to node not owning subnet: Check connectivity is restored after delete deny all network policy
2024-02-09T07:20:48.1191653Z   Expected success, but got an error:
2024-02-09T07:20:48.1192369Z       <*fmt.wrapError | 0xc000763d20>: 
2024-02-09T07:20:48.1193746Z       failed Write to server: write tcp 172.18.0.1:41446->172.18.0.3:31702: write: broken pipe
2024-02-09T07:20:48.1194723Z       {
2024-02-09T07:20:48.1196036Z           msg: "failed Write to server: write tcp 172.18.0.1:41446->172.18.0.3:31702: write: broken pipe",
2024-02-09T07:20:48.1197218Z           err: <*net.OpError | 0xc00123f9f0>{
2024-02-09T07:20:48.1197908Z               Op: "write",
2024-02-09T07:20:48.1198471Z               Net: "tcp",
2024-02-09T07:20:48.1199748Z               Source: <*net.TCPAddr | 0xc0006ed9b0>{IP: [172, 18, 0, 1], Port: 41446, Zone: ""},
2024-02-09T07:20:48.1201354Z               Addr: <*net.TCPAddr | 0xc0006edaa0>{IP: [172, 18, 0, 3], Port: 31702, Zone: ""},
2024-02-09T07:20:48.1202601Z               Err: <*os.SyscallError | 0xc000763d00>{
2024-02-09T07:20:48.1203335Z                   Syscall: "write",
2024-02-09T07:20:48.1203895Z                   Err: <syscall.Errno>0x20,
2024-02-09T07:20:48.1204307Z               },
2024-02-09T07:20:48.1204642Z           },
2024-02-09T07:20:48.1204937Z       }�[0m
2024-02-09T07:20:48.1206276Z   �[38;5;9mIn �[1m[It]�[0m�[38;5;9m at: �[1m/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/kubevirt.go:401�[0m �[38;5;243m@ 02/09/24 07:20:46.845�[0m

I will try to reproduce this too with the testing PR.

qinqon · 2024-02-12T15:48:14Z

I have disable interconnect and remove the policy part of the test to make it simpler also printing all the echos
this is the result
Server logs

Server is running on: :9900
  2024/02/12 12:26:15 Handling connection 100.64.0.5:32870
  2024/02/12 12:26:15 Handling connection [fd98::5]:58614
  2024/02/12 12:28:33 failed copying data: readfrom tcp 10.244.1.8:9900->100.64.0.5:32870: splice: connection reset by peer
  2024/02/12 12:28:33 Closing connection 100.64.0.5:32870

Client logs

STEP: worker1: after live migration for the second time to node not owning subnet: Check tcp connection is not broken @ 02/12/24 13:07:39.455
  STEP: Writing 'Halo' @ 02/12/24 13:07:39.455
  STEP: Reading 'Halo' @ 02/12/24 13:07:39.455
  STEP: failed reading: read tcp 172.18.0.1:41882->172.18.0.4:30247: read: connection reset by peer  @ 02/12/24 13:07:39.465
  STEP: Writing 'Halo' @ 02/12/24 13:07:54.465
  STEP: failed writing: write tcp 172.18.0.1:41882->172.18.0.4:30247: write: broken pipe  @ 02/12/24 13:07:54.466
  STEP: Writing 'Halo' @ 02/12/24 13:08:09.466
  STEP: failed writing: write tcp 172.18.0.1:41882->172.18.0.4:30247: write: broken pipe  @ 02/12/24 13:08:09.467
  STEP: Writing 'Halo' @ 02/12/24 13:08:24.467
  STEP: failed writing: write tcp 172.18.0.1:41882->172.18.0.4:30247: write: broken pipe  @ 02/12/24 13:08:24.467

The test should not retry if first error is "connection reset by peer" since retrying will not fix anything.

I am going to attach a tcpdump to the client to see if we are receving at RST packet that triggers the "connection reset by peer" at first echo.

qinqon · 2024-02-26T11:45:54Z

This should appear way less often after
#4174

npinaeva · 2024-03-27T13:52:43Z

https://github.com/ovn-org/ovn-kubernetes/actions/runs/8449224256/job/23144057157?pr=4154

trozet · 2024-03-29T13:41:23Z

https://github.com/ovn-org/ovn-kubernetes/actions/runs/8478907825/job/23232584804?pr=4246

qinqon · 2024-04-01T06:31:54Z

https://github.com/ovn-org/ovn-kubernetes/actions/runs/8478907825/job/23232584804?pr=4246

@trozet adding this link to #4237 since is the specific error related to nameserver

martinkennelly · 2024-04-11T16:06:01Z

https://github.com/ovn-org/ovn-kubernetes/actions/runs/8648860156/job/23715049496?pr=4271

Seen in #4271

flavio-fernandes · 2024-06-27T15:15:31Z

Also seen in #4468

trozet added the kind/ci-flake Flakes seen in CI label Oct 30, 2023

ricky-rav self-assigned this Nov 29, 2023

flavio-fernandes mentioned this issue Dec 13, 2023

Fix Egress IP Deletion Handler to Prevent OVN Policy Leaks #4050

Merged

jcaamano mentioned this issue Jan 31, 2024

OCPBUGS-27215: Replace routes from kubevirt live migration to prevent ECMP #4094

Merged

npinaeva mentioned this issue Feb 6, 2024

Wait for ovnkube controller to start before checking result error. #4109

Merged

tssurya mentioned this issue Feb 7, 2024

Separate timeout for handler sync from informer sync and do not resync services during node tracker startup #4100

Merged

qinqon mentioned this issue Feb 8, 2024

Kubevirt e2e stabilizing refactor #4145

Closed

dceara mentioned this issue Feb 14, 2024

fedora: Update OVN to latest version ovn-23.09.0-112. #4114

Merged

qinqon mentioned this issue Feb 22, 2024

kubevirt: Disable guest console log container #4174

Merged

This was referenced Feb 22, 2024

Refactor ovnRunKube and fix some stop scenarios #4159

Merged

FLAKE: Kubevirt Virtual Machines when live migration [It] with pre-copy succeeds, should keep connectivity #4175

Closed

npinaeva mentioned this issue Mar 27, 2024

ipv6+all protocols conntrack flush #4154

Merged

trozet mentioned this issue Mar 29, 2024

vendor: bump safchain/ethtool to v0.3.1-0.20231027162144-83e5e0097c91 #4246

Merged

qinqon linked a pull request Jun 4, 2024 that will close this issue

kubevirt: Force neighbors cache update after live migrtation #4416

Open

1 task

This was referenced Jun 27, 2024

Docs: add new branch as part of release process #4468

Open

Add missing places when renaming ovn-kubernetes packages #4465

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FAIL] Kubevirt Virtual Machines when live migrated [It] with pre-copy should keep connectivity #3986

[FAIL] Kubevirt Virtual Machines when live migrated [It] with pre-copy should keep connectivity #3986

trozet commented Oct 30, 2023

npinaeva commented Nov 2, 2023

martinkennelly commented Dec 7, 2023 •

edited

Loading

flavio-fernandes commented Dec 13, 2023

tssurya commented Feb 7, 2024

qinqon commented Feb 7, 2024

tssurya commented Feb 7, 2024 •

edited

Loading

qinqon commented Feb 7, 2024

martinkennelly commented Feb 8, 2024

qinqon commented Feb 8, 2024

qinqon commented Feb 8, 2024 •

edited

Loading

qinqon commented Feb 9, 2024

qinqon commented Feb 12, 2024 •

edited

Loading

qinqon commented Feb 26, 2024

npinaeva commented Mar 27, 2024

trozet commented Mar 29, 2024

qinqon commented Apr 1, 2024

martinkennelly commented Apr 11, 2024

flavio-fernandes commented Jun 27, 2024

[FAIL] Kubevirt Virtual Machines when live migrated [It] with pre-copy should keep connectivity #3986

[FAIL] Kubevirt Virtual Machines when live migrated [It] with pre-copy should keep connectivity #3986

Comments

trozet commented Oct 30, 2023

npinaeva commented Nov 2, 2023

martinkennelly commented Dec 7, 2023 • edited Loading

flavio-fernandes commented Dec 13, 2023

tssurya commented Feb 7, 2024

qinqon commented Feb 7, 2024

tssurya commented Feb 7, 2024 • edited Loading

qinqon commented Feb 7, 2024

martinkennelly commented Feb 8, 2024

qinqon commented Feb 8, 2024

qinqon commented Feb 8, 2024 • edited Loading

qinqon commented Feb 9, 2024

qinqon commented Feb 12, 2024 • edited Loading

qinqon commented Feb 26, 2024

npinaeva commented Mar 27, 2024

trozet commented Mar 29, 2024

qinqon commented Apr 1, 2024

martinkennelly commented Apr 11, 2024

flavio-fernandes commented Jun 27, 2024

martinkennelly commented Dec 7, 2023 •

edited

Loading

tssurya commented Feb 7, 2024 •

edited

Loading

qinqon commented Feb 8, 2024 •

edited

Loading

qinqon commented Feb 12, 2024 •

edited

Loading