Skip to content
This repository has been archived by the owner on Jun 28, 2024. It is now read-only.

devmapper CIs are broken #5682

Open
fidencio opened this issue Jun 6, 2023 · 1 comment
Open

devmapper CIs are broken #5682

fidencio opened this issue Jun 6, 2023 · 1 comment
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.

Comments

@fidencio
Copy link
Member

fidencio commented Jun 6, 2023

The CIs are consistently failing with:

03:06:31 not ok 2 Copy from pod to host
03:06:35 # (in test file k8s-copy-file.bats, line 72)
03:06:35 #   `cat "$file_name" | grep "$content"' failed
03:06:35 # INFO: k8s configured to use runtimeclass
03:06:35 # pod/pod-copy-file-to-host created
03:06:35 # pod/pod-copy-file-to-host condition met
03:06:35 # Name:         pod-copy-file-to-host
03:06:35 # Namespace:    default
03:06:35 # Priority:     0
03:06:35 # Node:         ubuntu20-37fdc0/10.0.0.5
03:06:35 # Start Time:   Sat, 03 Jun 2023 01:06:31 +0000
03:06:35 # Labels:       <none>
03:06:35 # Annotations:  <none>
03:06:35 # Status:       Running
03:06:35 # IP:           10.244.0.9
03:06:35 # IPs:
03:06:35 #   IP:  10.244.0.9
03:06:35 # Containers:
03:06:35 #   ctr-copy-file-to-host:
03:06:35 #     Container ID:  containerd://4d488d9cec03a6472252744148dab14728721b78f528bcc8110692aa94d2320e
03:06:35 #     Image:         quay.io/prometheus/busybox:latest
03:06:35 #     Image ID:      quay.io/prometheus/busybox@sha256:9a02a4a070d63298c2d3aaaa2551871e1fb40bba8c032f879a19246a273c2819
03:06:35 #     Port:          <none>
03:06:35 #     Host Port:     <none>
03:06:35 #     Command:
03:06:35 #       sleep
03:06:35 #       120
03:06:35 #     State:          Running
03:06:35 #       Started:      Sat, 03 Jun 2023 01:06:34 +0000
03:06:35 #     Ready:          True
03:06:35 #     Restart Count:  0
03:06:35 #     Environment:    <none>
03:06:35 #     Mounts:
03:06:35 #       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5kmmt (ro)
03:06:35 # Conditions:
03:06:35 #   Type              Status
03:06:35 #   Initialized       True
03:06:35 #   Ready             True
03:06:35 #   ContainersReady   True
03:06:35 #   PodScheduled      True
03:06:35 # Volumes:
03:06:35 #   kube-api-access-5kmmt:
03:06:35 #     Type:                    Projected (a volume that contains injected data from multiple sources)
03:06:35 #     TokenExpirationSeconds:  3607
03:06:35 #     ConfigMapName:           kube-root-ca.crt
03:06:35 #     ConfigMapOptional:       <nil>
03:06:35 #     DownwardAPI:             true
03:06:35 # QoS Class:                   BestEffort
03:06:35 # Node-Selectors:              <none>
03:06:35 # Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
03:06:35 #                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
03:06:35 # Events:
03:06:35 #   Type    Reason     Age   From               Message
03:06:35 #   ----    ------     ----  ----               -------
03:06:35 #   Normal  Scheduled  3s    default-scheduler  Successfully assigned default/pod-copy-file-to-host to ubuntu20-37fdc0
03:06:35 #   Normal  Pulling    1s    kubelet            Pulling image "quay.io/prometheus/busybox:latest"
03:06:35 #   Normal  Pulled     0s    kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 100.831595ms
03:06:35 #   Normal  Created    0s    kubelet            Created container ctr-copy-file-to-host
03:06:35 #   Normal  Started    0s    kubelet            Started container ctr-copy-file-to-host
03:06:37 # NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE
03:06:37 # default        pod-copy-file-to-host                     1/1     Running   0          3s
03:06:37 # kube-flannel   kube-flannel-ds-kgzr8                     1/1     Running   0          47s
03:06:37 # kube-system    coredns-64897985d-c7h6b                   1/1     Running   0          46s
03:06:37 # kube-system    coredns-64897985d-kdzwr                   1/1     Running   0          46s
03:06:37 # kube-system    etcd-ubuntu20-37fdc0                      1/1     Running   0          61s
03:06:37 # kube-system    kube-apiserver-ubuntu20-37fdc0            1/1     Running   0          61s
03:06:37 # kube-system    kube-controller-manager-ubuntu20-37fdc0   1/1     Running   0          61s
03:06:37 # kube-system    kube-proxy-66dp4                          1/1     Running   0          47s
03:06:37 # kube-system    kube-scheduler-ubuntu20-37fdc0            1/1     Running   0          61s
03:06:37 # I0603 01:06:34.744911   73126 log.go:184] (0xc0005c1080) (0xc00081f360) Create stream
03:06:37 # I0603 01:06:34.744991   73126 log.go:184] (0xc0005c1080) (0xc00081f360) Stream added, broadcasting: 1
03:06:37 # I0603 01:06:34.746221   73126 log.go:184] (0xc0005c1080) Reply frame received for 1
03:06:37 # I0603 01:06:34.746260   73126 log.go:184] (0xc0005c1080) (0xc0005b6d20) Create stream
03:06:37 # I0603 01:06:34.746267   73126 log.go:184] (0xc0005c1080) (0xc0005b6d20) Stream added, broadcasting: 3
03:06:37 # I0603 01:06:34.746813   73126 log.go:184] (0xc0005c1080) Reply frame received for 3
03:06:37 # I0603 01:06:34.746838   73126 log.go:184] (0xc0005c1080) (0xc00081f400) Create stream
03:06:37 # I0603 01:06:34.746844   73126 log.go:184] (0xc0005c1080) (0xc00081f400) Stream added, broadcasting: 5
03:06:37 # I0603 01:06:34.747331   73126 log.go:184] (0xc0005c1080) Reply frame received for 5
03:06:37 # I0603 01:06:34.763657   73126 log.go:184] (0xc0005c1080) Data frame received for 5
03:06:37 # I0603 01:06:34.763692   73126 log.go:184] (0xc00081f400) (5) Data frame handling
03:06:37 # I0603 01:06:34.763718   73126 log.go:184] (0xc0005c1080) Data frame received for 3
03:06:37 # I0603 01:06:34.763728   73126 log.go:184] (0xc0005b6d20) (3) Data frame handling
03:06:37 # I0603 01:06:34.768178   73126 log.go:184] (0xc0005c1080) Data frame received for 1
03:06:37 # I0603 01:06:34.768230   73126 log.go:184] (0xc00081f360) (1) Data frame handling
03:06:37 # I0603 01:06:34.768241   73126 log.go:184] (0xc00081f360) (1) Data frame sent
03:06:37 # I0603 01:06:34.768250   73126 log.go:184] (0xc0005c1080) (0xc00081f360) Stream removed, broadcasting: 1
03:06:37 # I0603 01:06:34.768261   73126 log.go:184] (0xc0005c1080) Go away received
03:06:37 # I0603 01:06:34.768952   73126 log.go:184] (0xc0005c1080) (0xc00081f360) Stream removed, broadcasting: 1
03:06:37 # I0603 01:06:34.768971   73126 log.go:184] (0xc0005c1080) (0xc0005b6d20) Stream removed, broadcasting: 3
03:06:37 # I0603 01:06:34.768986   73126 log.go:184] (0xc0005c1080) (0xc00081f400) Stream removed, broadcasting: 5
03:06:37 # Name:         pod-copy-file-to-host
03:06:37 # Namespace:    default
03:06:37 # Priority:     0
03:06:37 # Node:         ubuntu20-37fdc0/10.0.0.5
03:06:37 # Start Time:   Sat, 03 Jun 2023 01:06:31 +0000
03:06:37 # Labels:       <none>
03:06:37 # Annotations:  <none>
03:06:37 # Status:       Running
03:06:37 # IP:           10.244.0.9
03:06:37 # IPs:
03:06:37 #   IP:  10.244.0.9
03:06:37 # Containers:
03:06:37 #   ctr-copy-file-to-host:
03:06:37 #     Container ID:  containerd://4d488d9cec03a6472252744148dab14728721b78f528bcc8110692aa94d2320e
03:06:37 #     Image:         quay.io/prometheus/busybox:latest
03:06:37 #     Image ID:      quay.io/prometheus/busybox@sha256:9a02a4a070d63298c2d3aaaa2551871e1fb40bba8c032f879a19246a273c2819
03:06:37 #     Port:          <none>
03:06:37 #     Host Port:     <none>
03:06:37 #     Command:
03:06:37 #       sleep
03:06:37 #       120
03:06:37 #     State:          Running
03:06:37 #       Started:      Sat, 03 Jun 2023 01:06:34 +0000
03:06:37 #     Ready:          True
03:06:37 #     Restart Count:  0
03:06:37 #     Environment:    <none>
03:06:37 #     Mounts:
03:06:37 #       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5kmmt (ro)
03:06:37 # Conditions:
03:06:37 #   Type              Status
03:06:37 #   Initialized       True
03:06:37 #   Ready             True
03:06:37 #   ContainersReady   True
03:06:37 #   PodScheduled      True
03:06:37 # Volumes:
03:06:37 #   kube-api-access-5kmmt:
03:06:37 #     Type:                    Projected (a volume that contains injected data from multiple sources)
03:06:37 #     TokenExpirationSeconds:  3607
03:06:37 #     ConfigMapName:           kube-root-ca.crt
03:06:37 #     ConfigMapOptional:       <nil>
03:06:37 #     DownwardAPI:             true
03:06:37 # QoS Class:                   BestEffort
03:06:37 # Node-Selectors:              <none>
03:06:37 # Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
03:06:37 #                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
03:06:37 # Events:
03:06:37 #   Type    Reason     Age   From               Message
03:06:37 #   ----    ------     ----  ----               -------
03:06:37 #   Normal  Scheduled  3s    default-scheduler  Successfully assigned default/pod-copy-file-to-host to ubuntu20-37fdc0
03:06:37 #   Normal  Pulling    1s    kubelet            Pulling image "quay.io/prometheus/busybox:latest"
03:06:37 #   Normal  Pulled     0s    kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 100.831595ms
03:06:37 #   Normal  Created    0s    kubelet            Created container ctr-copy-file-to-host
03:06:37 #   Normal  Started    0s    kubelet            Started container ctr-copy-file-to-host
03:06:39 # NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE
03:06:39 # default        pod-copy-file-to-host                     1/1     Running   0          3s
03:06:39 # kube-flannel   kube-flannel-ds-kgzr8                     1/1     Running   0          47s
03:06:39 # kube-system    coredns-64897985d-c7h6b                   1/1     Running   0          46s
03:06:39 # kube-system    coredns-64897985d-kdzwr                   1/1     Running   0          46s
03:06:39 # kube-system    etcd-ubuntu20-37fdc0                      1/1     Running   0          61s
03:06:39 # kube-system    kube-apiserver-ubuntu20-37fdc0            1/1     Running   0          61s
03:06:39 # kube-system    kube-controller-manager-ubuntu20-37fdc0   1/1     Running   0          61s
03:06:39 # kube-system    kube-proxy-66dp4                          1/1     Running   0          47s
03:06:39 # kube-system    kube-scheduler-ubuntu20-37fdc0            1/1     Running   0          61s
03:06:39 # I0603 01:06:34.987391   73156 log.go:184] (0xc0001a7970) (0xc0004fcf00) Create stream
03:06:39 # I0603 01:06:34.987452   73156 log.go:184] (0xc0001a7970) (0xc0004fcf00) Stream added, broadcasting: 1
03:06:39 # I0603 01:06:34.990038   73156 log.go:184] (0xc0001a7970) Reply frame received for 1
03:06:39 # I0603 01:06:34.990068   73156 log.go:184] (0xc0001a7970) (0xc0004fc8c0) Create stream
03:06:39 # I0603 01:06:34.990075   73156 log.go:184] (0xc0001a7970) (0xc0004fc8c0) Stream added, broadcasting: 3
03:06:39 # I0603 01:06:34.990599   73156 log.go:184] (0xc0001a7970) Reply frame received for 3
03:06:39 # I0603 01:06:34.990625   73156 log.go:184] (0xc0001a7970) (0xc0003cd2c0) Create stream
03:06:40 # I0603 01:06:34.990632   73156 log.go:184] (0xc0001a7970) (0xc0003cd2c0) Stream added, broadcasting: 5
03:06:40 # I0603 01:06:34.991191   73156 log.go:184] (0xc0001a7970) Reply frame received for 5
03:06:40 # I0603 01:06:35.004225   73156 log.go:184] (0xc0001a7970) Data frame received for 3
03:06:40 # I0603 01:06:35.004246   73156 log.go:184] (0xc0004fc8c0) (3) Data frame handling
03:06:40 # I0603 01:06:35.004274   73156 log.go:184] (0xc0001a7970) Data frame received for 5
03:06:40 # I0603 01:06:35.004284   73156 log.go:184] (0xc0003cd2c0) (5) Data frame handling
03:06:40 # I0603 01:06:35.004293   73156 log.go:184] (0xc0003cd2c0) (5) Data frame sent
03:06:40 # tar: removing leading '/' from member names
03:06:40 # I0603 01:06:35.004520   73156 log.go:184] (0xc0001a7970) Data frame received for 5
03:06:40 # I0603 01:06:35.004537   73156 log.go:184] (0xc0003cd2c0) (5) Data frame handling
03:06:40 # I0603 01:06:35.006357   73156 log.go:184] (0xc0001a7970) Data frame received for 1
03:06:40 # I0603 01:06:35.006374   73156 log.go:184] (0xc0004fcf00) (1) Data frame handling
03:06:40 # I0603 01:06:35.006383   73156 log.go:184] (0xc0004fcf00) (1) Data frame sent
03:06:40 # I0603 01:06:35.006493   73156 log.go:184] (0xc0001a7970) (0xc0004fcf00) Stream removed, broadcasting: 1
03:06:40 # I0603 01:06:35.006708   73156 log.go:184] (0xc0001a7970) (0xc0003cd2c0) Stream removed, broadcasting: 5
03:06:40 # I0603 01:06:35.006756   73156 log.go:184] (0xc0001a7970) (0xc0004fcf00) Stream removed, broadcasting: 1
03:06:40 # I0603 01:06:35.006767   73156 log.go:184] (0xc0001a7970) (0xc0004fc8c0) Stream removed, broadcasting: 3
03:06:40 # Dropping out copy after 0 retries
03:06:40 # I0603 01:06:35.006781   73156 log.go:184] (0xc0001a7970) Go away received
03:06:40 # cat: file.txt: No such file or directory
03:06:40 # Name:         pod-copy-file-to-host
03:06:40 # Namespace:    default
03:06:40 # Priority:     0
03:06:40 # Node:         ubuntu20-37fdc0/10.0.0.5
03:06:40 # Start Time:   Sat, 03 Jun 2023 01:06:31 +0000
03:06:40 # Labels:       <none>
03:06:40 # Annotations:  <none>
03:06:40 # Status:       Running
03:06:40 # IP:           10.244.0.9
03:06:40 # IPs:
03:06:40 #   IP:  10.244.0.9
03:06:40 # Containers:
03:06:40 #   ctr-copy-file-to-host:
03:06:40 #     Container ID:  containerd://4d488d9cec03a6472252744148dab14728721b78f528bcc8110692aa94d2320e
03:06:40 #     Image:         quay.io/prometheus/busybox:latest
03:06:40 #     Image ID:      quay.io/prometheus/busybox@sha256:9a02a4a070d63298c2d3aaaa2551871e1fb40bba8c032f879a19246a273c2819
03:06:40 #     Port:          <none>
03:06:40 #     Host Port:     <none>
03:06:40 #     Command:
03:06:40 #       sleep
03:06:40 #       120
03:06:40 #     State:          Running
03:06:40 #       Started:      Sat, 03 Jun 2023 01:06:34 +0000
03:06:40 #     Ready:          True
03:06:40 #     Restart Count:  0
03:06:40 #     Environment:    <none>
03:06:40 #     Mounts:
03:06:40 #       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5kmmt (ro)
03:06:40 # Conditions:
03:06:40 #   Type              Status
03:06:40 #   Initialized       True
03:06:40 #   Ready             True
03:06:40 #   ContainersReady   True
03:06:40 #   PodScheduled      True
03:06:40 # Volumes:
03:06:40 #   kube-api-access-5kmmt:
03:06:40 #     Type:                    Projected (a volume that contains injected data from multiple sources)
03:06:40 #     TokenExpirationSeconds:  3607
03:06:40 #     ConfigMapName:           kube-root-ca.crt
03:06:40 #     ConfigMapOptional:       <nil>
03:06:40 #     DownwardAPI:             true
03:06:40 # QoS Class:                   BestEffort
03:06:40 # Node-Selectors:              <none>
03:06:40 # Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
03:06:40 #                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
03:06:40 # Events:
03:06:40 #   Type    Reason     Age   From               Message
03:06:40 #   ----    ------     ----  ----               -------
03:06:40 #   Normal  Scheduled  4s    default-scheduler  Successfully assigned default/pod-copy-file-to-host to ubuntu20-37fdc0
03:06:40 #   Normal  Pulling    2s    kubelet            Pulling image "quay.io/prometheus/busybox:latest"
03:06:40 #   Normal  Pulled     1s    kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 100.831595ms
03:06:40 #   Normal  Created    1s    kubelet            Created container ctr-copy-file-to-host
03:06:40 #   Normal  Started    1s    kubelet            Started container ctr-copy-file-to-host
03:06:40 # pod "pod-copy-file-to-host" deleted
03:06:40 [run_kubernetes_tests.sh:153] ERROR: bats k8s-copy-file.bats

Till someone has the time to debug it, the CI will be marked as non-required.
@fidencio fidencio added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels Jun 6, 2023
@fidencio
Copy link
Member Author

fidencio commented Jun 6, 2023

The last time a baseline CI passed as on June 2nd, starting to break on June 3rd.

@fidencio fidencio mentioned this issue Jun 6, 2023
Closed
@ananos ananos mentioned this issue Jun 21, 2023
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fidencio