E2E tests instability #316

LionelJouin · 2022-11-01T16:23:45Z

Describe the bug
Tests are failing randomly around 19% of the time.

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/activity/

https://github.com/Nordix/Meridio/blob/master/docs/test.md

To Reproduce
/

Expected behavior
/

Logs

This report is from job run 24 to 172 (149 runs).

invalid character 'R' looking for beginning of value

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/71/pipeline/22/

�[38;5;13m•! [PANICKED] [190.727 seconds]�[0m
Scaling
�[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:30�[0m
  With one trench containing a stream with 2 VIP addresses and 4 target pods running
  �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:32�[0m
    when scaling targets up by 1
    �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:148�[0m
      �[38;5;13m�[1m[It] should receive the traffic correctly�[0m
      �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:152�[0m

  �[38;5;243mBegin Captured GinkgoWriter Output >>�[0m
    �[1mSTEP:�[0m Checking if all targets have receive ipv4 traffic with no traffic interruption (no lost connection) �[38;5;243m11/01/22 16:09:02.33�[0m
    �[1mSTEP:�[0m Checking if all targets have receive ipv6 traffic with no traffic interruption (no lost connection) �[38;5;243m11/01/22 16:09:03.53�[0m
  �[38;5;243m<< End Captured GinkgoWriter Output�[0m

  �[38;5;13mTest Panicked�[0m
  �[38;5;13mIn �[1m[It]�[0m�[38;5;13m at: �[1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/utils/trafficgenerator.go:96�[0m

  �[38;5;13minvalid character 'R' looking for beginning of value�[0m

  �[38;5;13mFull Stack Trace�[0m
    github.com/nordix/meridio/test/e2e/utils.(*MConnect).AnalyzeTraffic(0x15fa73d?, {0xc0000fb000, 0x6c9, 0x1000})
    	/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/utils/trafficgenerator.go:96 +0x1a6
    github.com/nordix/meridio/test/e2e/utils.(*TrafficGeneratorHost).SendTraffic(0xc0004ddf68?, {0x1821bd0, 0xc000133968}, {0x7fff08b74cb3?, 0x8}, {0x7fff08b74c82, 0x3}, {0xc000494450, 0xe}, {0x15f7463, ...})
    	/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/utils/trafficgenerator.go:45 +0x277
    github.com/nordix/meridio/test/e2e_test.glob..func6.1.5.2()
    	/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/71/test/e2e/scaling_test.go:161 +0x3cd

3 Targets are receiving traffic instead of 4 (most frequent) - #234

�[38;5;9m• [FAILED] [29.806 seconds]�[0m
Target
�[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:30�[0m
  With one trench containing a stream with 2 VIP addresses and 4 target pods running
  �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:32�[0m
    when a target is closing a stream
    �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:51�[0m
      �[38;5;9m�[1m[It] should receive traffic anymore�[0m
      �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/68/test/e2e/target_test.go:81�[0m

  �[38;5;243mBegin Captured GinkgoWriter Output >>�[0m
    �[1mSTEP:�[0m Checking the target has not receive ipv4 traffic �[38;5;243m11/01/22 15:40:36.855�[0m
  �[38;5;243m<< End Captured GinkgoWriter Output�[0m

  �[38;5;9mExpected
      <int>: 4
  to equal
      <int>: 3�[0m

unable to upgrade connection: container not found ("example-target")

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/2760/pipeline/22

�[38;5;9m• [FAILED] [0.050 seconds]�[0m
MultiTrenches
�[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:30�[0m
  With two trenches containing both a stream with 2 VIP addresses and 4 target pods running
  �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:32�[0m
    �[38;5;9m�[1mwhen a target disconnects from a trench and connect to another one [BeforeEach]�[0m
    �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:109�[0m
      should receive the traffic on the other trench
      �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:141�[0m

  �[38;5;9mUnexpected error:
      <*fmt.wrapError | 0xc0002bd4e0>: {
          msg: "unable to upgrade connection: container not found (\"example-target\"); ",
          err: <*errors.errorString | 0xc00050efc0>{
              s: "unable to upgrade connection: container not found (\"example-target\")",
          },
      }
      unable to upgrade connection: container not found ("example-target"); 
  occurred�[0m
  �[38;5;9mIn �[1m[BeforeEach]�[0m�[38;5;9m at: �[1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/64/test/e2e/multi_trenches_test.go:111�[0m

command terminated with exit code 137

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/165/pipeline/22
https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/81/pipeline/22/
https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/43/pipeline/22/

�[38;5;9m• [FAILED] [29.309 seconds]�[0m
MultiTrenches
�[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:30�[0m
  With two trenches containing both a stream with 2 VIP addresses and 4 target pods running
  �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:32�[0m
    �[38;5;9m�[1mwhen a target disconnects from a trench and connect to another one [BeforeEach]�[0m
    �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:109�[0m
      should receive the traffic on the other trench
      �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:141�[0m

  �[38;5;9mUnexpected error:
      <*fmt.wrapError | 0xc000548160>: {
          msg: "command terminated with exit code 137; ",
          err: <exec.CodeExitError>{
              Err: <*errors.errorString | 0xc0003ba310>{
                  s: "command terminated with exit code 137",
              },
              Code: 137,
          },
      }
      command terminated with exit code 137; 
  occurred�[0m
  �[38;5;9mIn �[1m[BeforeEach]�[0m�[38;5;9m at: �[1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/43/test/e2e/multi_trenches_test.go:113�[0m

pod not found

�[38;5;9m• [FAILED] [0.025 seconds]�[0m
MultiTrenches
�[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:30�[0m
  With two trenches containing both a stream with 2 VIP addresses and 4 target pods running
  �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:32�[0m
    �[38;5;9m�[1mwhen a target disconnects from a trench and connect to another one [BeforeEach]�[0m
    �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:109�[0m
      should receive the traffic on the other trench
      �[38;5;243m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:141�[0m

  �[38;5;9mUnexpected error:
      <*fmt.wrapError | 0xc0004f8980>: {
          msg: "pods \"target-a-878b7cb64-dxw9x\" not found; ",
          err: <*errors.StatusError | 0xc0004f2be0>{
              ErrStatus: {
                  TypeMeta: {Kind: "", APIVersion: ""},
                  ListMeta: {
                      SelfLink: "",
                      ResourceVersion: "",
                      Continue: "",
                      RemainingItemCount: nil,
                  },
                  Status: "Failure",
                  Message: "pods \"target-a-878b7cb64-dxw9x\" not found",
                  Reason: "NotFound",
                  Details: {
                      Name: "target-a-878b7cb64-dxw9x",
                      Group: "",
                      Kind: "pods",
                      UID: "",
                      Causes: nil,
                      RetryAfterSeconds: 0,
                  },
                  Code: 404,
              },
          },
      }
      pods "target-a-878b7cb64-dxw9x" not found; 
  occurred�[0m
  �[38;5;9mIn �[1m[BeforeEach]�[0m�[38;5;9m at: �[1m/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/124/test/e2e/multi_trenches_test.go:111�[0m

Timeout

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/364/pipeline/22/

Internal error occurred: error executing command in container: failed to exec in container: failed to create exec

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/2548/pipeline/22

• [FAILED] [394.636 seconds]
MultiTrenches
/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:30
  MT-Switch
  /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:110
    when Disconnect a target from target-a-deployment-name from trench-a and connect it to trench-b [BeforeEach]
    /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:112
      (Traffic) is received by the targets
      /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:144

  Unexpected error:
      <*fmt.wrapError | 0xc0001ee0c0>: {
          msg: "Internal error occurred: error executing command in container: failed to exec in container: failed to create exec \"fe45234377476bf7e74dabbe876cf7c7db8ca19dcd5723744357955ecad1c054\": task b63acb78a9f04481223ad020bf9fa7fba8297d01265934b930906360c91af8a3 not found: not found; ",
          err: <*errors.errorString | 0xc0003ae200>{
              s: "Internal error occurred: error executing command in container: failed to exec in container: failed to create exec \"fe45234377476bf7e74dabbe876cf7c7db8ca19dcd5723744357955ecad1c054\": task b63acb78a9f04481223ad020bf9fa7fba8297d01265934b930906360c91af8a3 not found: not found",
          },
      }
      Internal error occurred: error executing command in container: failed to exec in container: failed to create exec "fe45234377476bf7e74dabbe876cf7c7db8ca19dcd5723744357955ecad1c054": task b63acb78a9f04481223ad020bf9fa7fba8297d01265934b930906360c91af8a3 not found: not found; 
  occurred
  In [BeforeEach] at: /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/2548/test/e2e/multi_trenches_test.go:119

Interrupted by Timeout

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/2936/pipeline

command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/3154

• [FAILED] [451.770 seconds]
TAPA
/home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:30
  close-open
  /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:49
    when Close stream-a-I in one of the target from target-a-deployment-name and re-open it [BeforeEach]
    /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:55
      (Traffic) is received by the targets
      /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:118

  Unexpected error:
      <*fmt.wrapError | 0xc000424160>: {
          msg: "command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i\n",
          err: <exec.CodeExitError>{
              Err: <*errors.errorString | 0xc0005820c0>{
                  s: "command terminated with exit code 1",
              },
              Code: 1,
          },
      }
      command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i
      
  occurred
  In [BeforeEach] at: /home/jenkins/nordix/slave_root/workspace/meridio-e2e-test-kind/3154/test/e2e/tapa_test.go:76

2 tests fail

The text was updated successfully, but these errors were encountered:

LionelJouin · 2022-11-21T22:12:54Z

nfqlb configured, but no traffic:
https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/3103/pipeline/22

LionelJouin · 2023-02-20T15:20:01Z

Seed 1 ("Dead" NSM interface in the proxy with duplicated IP)

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/10948/pipeline/
https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/10944/pipeline/

A vip cannot be shared between 2 conduits in this version

https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/10952/pipeline/
https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/11158/pipeline/
https://jenkins.nordix.org/blue/organizations/jenkins/meridio-e2e-test-kind/detail/meridio-e2e-test-kind/11323/pipeline/

LionelJouin added kind/bug Something isn't working area/CI area/testing labels Nov 1, 2022

This was referenced Dec 6, 2022

Clean target IP from Neighbor table on add #334

Merged

Fix Scaling e2e test and Kubernetes 1.26 on Kind environment #336

Merged

LionelJouin self-assigned this Jan 11, 2023

LionelJouin mentioned this issue Jan 13, 2023

Fix TAPA concurrency issue #346

Merged

12 tasks

LionelJouin mentioned this issue Feb 20, 2023

New e2e tests #343

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E tests instability #316

E2E tests instability #316

LionelJouin commented Nov 1, 2022 •

edited

LionelJouin commented Nov 21, 2022

LionelJouin commented Feb 20, 2023 •

edited

E2E tests instability #316

E2E tests instability #316

Comments

LionelJouin commented Nov 1, 2022 • edited

invalid character 'R' looking for beginning of value

3 Targets are receiving traffic instead of 4 (most frequent) - #234

unable to upgrade connection: container not found ("example-target")

command terminated with exit code 137

pod not found

Timeout

Internal error occurred: error executing command in container: failed to exec in container: failed to create exec

Interrupted by Timeout

command terminated with exit code 1; FAILED mapSharedData: tshm-stream-a-i

2 tests fail

LionelJouin commented Nov 21, 2022

LionelJouin commented Feb 20, 2023 • edited

Seed 1 ("Dead" NSM interface in the proxy with duplicated IP)

A vip cannot be shared between 2 conduits in this version

LionelJouin commented Nov 1, 2022 •

edited

LionelJouin commented Feb 20, 2023 •

edited