Strange outward bound networking behavior on Ubuntu 22.04.2 LTS

[David-Igou]

Environmental Info:
K3s Version:

k3s -v
k3s version v1.26.4+k3s1 (8d0255a)
go version go1.19.

Node(s) CPU architecture, OS, and Version:
Master node: Linux master 5.15.80-rockchip64 #22.11.1 SMP PREEMPT Wed Nov 30 11:12:47 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Worker nodes: Linux worker-1 5.15.0-1029-raspi #31-Ubuntu SMP PREEMPT Sat Apr 22 12:26:40 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
1 server 4 agents

Agent OS is Ubuntu 22 LTS with very little modifications (k3s installed via https://github.com/PyratLabs/ansible-role-k3s)

OS booted with a cloud-init config that setup an ansible sudoer account

k3s was installed + the rpi extra kernel modules

Describe the bug:

It feels like in containers there is almost periodic rate limiting to "outside" (internet) connections. Here is the behavior observed in ArgoCD:

time="2023-05-28T20:42:22Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:42:22Z" grpc.time_ms=0.051 span.kind=server system=grpc
time="2023-05-28T20:42:52Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:42:52Z" grpc.time_ms=0.049 span.kind=server system=grpc
time="2023-05-28T20:43:22Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:43:22Z" grpc.time_ms=0.052 span.kind=server system=grpc
time="2023-05-28T20:43:52Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:43:52Z" grpc.time_ms=0.049 span.kind=server system=grpc
time="2023-05-28T20:44:22Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:44:22Z" grpc.time_ms=0.052 span.kind=server system=grpc
time="2023-05-28T20:44:31Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-05-28T20:44:31Z" grpc.time_ms=88.554 span.kind=server system=grpc
time="2023-05-28T20:44:52Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:44:52Z" grpc.time_ms=0.052 span.kind=server system=grpc
time="2023-05-28T20:45:22Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-05-28T20:45:22Z" grpc.time_ms=0.049 span.kind=server system=grpc
time="2023-05-28T20:45:24Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-05-28T20:45:24Z" grpc.time_ms=107.553 span.kind=server system=grpc
time="2023-05-28T20:45:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-05-28T20:45:25Z" grpc.time_ms=88.423 span.kind=server system=grpc
time="2023-05-28T20:45:26Z" level=error msg="finished unary call with code Unknown" error="error testing repository connectivity: Get \"https://github.com/igou-io/igou-kubernetes.git/info/refs?service=git-upload-pack\": EOF" grpc.code=Unknown grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-05-28T20:45:26Z" grpc.time_ms=303.756 span.kind=server system=grpc
time="2023-05-28T20:45:26Z" level=error msg="finished unary call with code Unknown" error="error testing repository connectivity: Get \"https://github.com/igou-io/igou-kubernetes.git/info/refs?service=git-upload-pack\": EOF" grpc.code=Unknown grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-05-28T20:45:26Z" grpc.time_ms=293.323 span.kind=server system=grpc
time="2023-05-28T20:45:31Z" level=error msg="finished unary call with code Unknown" error="error testing repository connectivity: Get \"https://github.com/igou-io/igou-kubernetes.git/info/refs?service=git-upload-pack\": EOF" grpc.code=Unknown grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-05-28T20:45:31Z" grpc.time_ms=294.396 span.kind=server system=grpc

I've also observed this when trying to deploy a Grafana pod that reaches out to the internet:

 k logs pod/release-name-grafana-7bd8dccc5f-6ckgq -n monitoring -c grafana
GF_PATHS_DATA='/var/lib/grafana/data' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
Error: ✗ Get "https://grafana.com/api/plugins/repo/natel-discrete-panel": EOF

Steps To Reproduce:
Installed K3s

Agent OS is Ubuntu 22 LTS with very little modifications (k3s installed via https://github.com/PyratLabs/ansible-role-k3s)

Expected behavior:
Pods can network externally

Actual behavior:
Periodic "EOF" returns from external networking

Additional context / logs:

It feels almost like limit rating, if you are familiar with ArgoCD - I can make a repo sync be pressing the button 10 seconds apart successfully, but if I click it more than once in 2ish seconds, then it starts to fail 100% of the time for a couple of minutes? I can't find any logs in the host OS..

I only have this problem with workloads running on Ubuntu 22.04.2 LTS Pi's

[David-Igou]

oh my god it was dns

[cwayne18]

The problem was DNS? Well now I've heard everything! 😄

[David-Igou]

It was a misconfiguration that only effected Ubuntu nodes, Armbian didn't seem to care. I don't completely understand the interworkings but fixing my configuration solved the problem immediately