the network commuication between k3s's pod and container failed #52

defims · 2023-01-16T19:24:54Z

I create a test repository defims/wasmedge-hyper-server to reproduct this problem:

environment

Ubuntu 22.04 LTS
k3s version v1.25.4+k3s1 (0dc63334) and go version go1.19.3
wasmedge version 0.11.2
containerd-shim-wasmedge-v1.zip build from second-state/runwasi (just cp it to /usr/local/bin/)

wasm pod failed

# run:
sudo kubectl apply -f wasm.yml
sudo curl localhost:30001

# got:
curl: (7) Failed to connect to localhost port 30001 after 1 ms: Connection refused

the img.tar and wasmedge-hyper-server.wasm:
img-and-wasmedge-hyper-server.zip

# unzip img-and-wasmedge-hyper-server.zip and import the image
sudo ctr image import --all-platforms img.tar

/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl:

version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true


[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime]
  runtime_type = "io.containerd.wasmtime.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
  runtime_type = "io.containerd.wasmedge.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

the wasm.yml file:

---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasmedge-deployment
  labels:
    app: wasmedge-hyper-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wasmedge-hyper-server
  template:
    metadata:
      labels:
        app: wasmedge-hyper-server
    spec:
      runtimeClassName: wasmedge
      containers:
      - name: wasmedge-hyper-server
        image: ghcr.io/containerd/runwasi/wasmedge-hyper-server:latest
        imagePullPolicy: Never
        ports:
        - containerPort: 8089
---
apiVersion: v1
kind: Service
metadata:
  name: wasmedge-service
  labels:
    app: wasmedge-hyper-server
spec:
  type: NodePort
  selector:
    app: wasmedge-hyper-server
  ports:
    - name: http
      protocol: TCP
      port: 8089
      targetPort: 8089
      nodePort: 30001

nginx works

# run:
sudo kubectl apply -f nginx.yml
sudo curl localhost:30000

# got:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

and the nginx.yml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.16.1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30000

single pod with hostNetwork failed:

# run:
sudo kubectl apply -f pod.yml
sudo curl localhost:8089

# got:
curl: (7) Failed to connect to localhost port 8089 after 0 ms: Connection refused

pod.yml file:

---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
---
apiVersion: v1
kind: Pod
metadata:
  name: wasmedge-hyper-pod
spec:
  hostNetwork: true
  runtimeClassName: wasmedge
  containers:
  - name: wasmedge-hyper-server
    image: ghcr.io/containerd/runwasi/wasmedge-hyper-server:latest
    imagePullPolicy: Never
    ports:
    - containerPort: 8089

wasm container works:

I'm sure anything oher than the network works

# run:
sudo ctr run --rm --net-host --runtime=io.containerd.wasmedge.v1 ghcr.io/containerd/runwasi/wasmedge-hyper-server:latest wasmedge-hyper-server
sudo curl localhost:8089

# got:
Try POSTing data to /echo such as: `curl localhost:8089/echo -XPOST -d 'hello world'`

hydai · 2023-01-19T06:37:09Z

Hi @defims
I would like to know if k8s/k3s produce any extra logs. I received other reports from the WasmEdge discord, and they have similar issues to yours. They tried to deploy runwasi+wasmedge or runwasi on their k8s cluster. Unfortunately, the application will keep running without producing any logs; however, this application should print lots of messages in the console(stdout).

defims · 2023-01-20T06:00:19Z

Hi @defims I would like to know if k8s/k3s produce any extra logs. I received other reports from the WasmEdge discord, and they have similar issues to yours. They tried to deploy runwasi+wasmedge or runwasi on their k8s cluster. Unfortunately, the application will keep running without producing any logs; however, this application should print lots of messages in the console(stdout).

@hydai
unfortunately, there's no logs either, so I post another issue 50.

defims · 2023-02-06T07:13:23Z

I test with new version and everything works!
runwasi git commit id is ec19a08c6800e8eccc2f6385f5389e2b8fb60855

# run
git clone https://github.com/second-state/wasmedge_wasi_socket.git
cd examples/http_server/
cargo build --target wasm32-wasi --release
sudo buildah build -t http_server . # need buildah installed and use Dockerfile below
sudo buildah push localhost/http_server:latest oci-archive:http_server.tar:localhost/http_server:latest
sudo ctr images import --all-platforms http_server.tar
sudo kubectl delete -f wasm.yml # need runwasi installed, wasmedge installed and containerd configured
sudo kubectl apply -f wasm.yml
sudo curl localhost:1234

# got
echo:

Dockerfile:

FROM scratch
ADD target/wasm32-wasi/release/http_server.wasm /
CMD ["/http_server.wasm"]

wasm.yml:

---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: http-server
  name: http-server
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  runtimeClassName: wasmedge
  containers:
  - image: localhost/http_server:latest
    imagePullPolicy: Never
    name: crud
    resources:
      limits:
        memory: 2048Mi
        cpu: "2"
      requests:
        memory: 512Mi
        cpu: ".5"

/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl is:

version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/84e02c125ef7644128a11b4ba4ebf72630b91abe0433e60a19586a247542e75b/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
  runtime_type = "io.containerd.wasmedge.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

defims changed the title ~~the network commuication between k8s's pod and container failed~~ the network commuication between k3s's pod and container failed Jan 17, 2023

defims closed this as completed Feb 6, 2023

defims mentioned this issue Feb 6, 2023

Document full set up with k3s #20

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

the network commuication between k3s's pod and container failed #52

the network commuication between k3s's pod and container failed #52

defims commented Jan 16, 2023 •

edited

Loading

hydai commented Jan 19, 2023

defims commented Jan 20, 2023 •

edited

Loading

defims commented Feb 6, 2023 •

edited

Loading

the network commuication between k3s's pod and container failed #52

the network commuication between k3s's pod and container failed #52

Comments

defims commented Jan 16, 2023 • edited Loading

environment

wasm pod failed

nginx works

single pod with hostNetwork failed:

wasm container works:

hydai commented Jan 19, 2023

defims commented Jan 20, 2023 • edited Loading

defims commented Feb 6, 2023 • edited Loading

defims commented Jan 16, 2023 •

edited

Loading

defims commented Jan 20, 2023 •

edited

Loading

defims commented Feb 6, 2023 •

edited

Loading