Document full set up with k3s #20

agracey · 2022-05-17T15:31:39Z

There is a large amount of assumed knowledge and set up in the current instructions so it would be useful to have documentation of a full run through of setup and usage with k3s.

I'm working on getting this running in my lab using k3s. If I get it all working, I can write up the commands I used.

Mossaka · 2022-05-18T03:38:56Z

I have an example that runs a shim developed on top of runwasi working on k3d.

Check out this here: https://github.com/Mossaka/k3d-wasm
Particularly containerd-shim-spin-v1 is a spin shim binary. Here is the source code.

agracey · 2022-09-05T00:20:25Z

Hey, I finally got a chance to play with this. I think I got it working on k3s but have a few issues:

I can see the shim running when the "pod" gets deployed! So I'm pretty sure my containerd config is read correctly.

I have to adapt a little since I can't use the example service as written. When I add a service and ingress, I get a "Bad Gateway" back from my ingress controller so something is weird with the networking. I'm assuming that the "pod" is exposing port 80, right? I see that in the code that it's listening on 0.0.0.0:80.

Also, I'm not seeing logs and when I delete the pod, it gets stuck in terminating.

Update submodule to fix CI build fail

defims · 2023-01-02T03:08:18Z

Hey, I finally got a chance to play with this. I think I got it working on k3s but have a few issues:

I can see the shim running when the "pod" gets deployed! So I'm pretty sure my containerd config is read correctly.

I have to adapt a little since I can't use the example service as written. When I add a service and ingress, I get a "Bad Gateway" back from my ingress controller so something is weird with the networking. I'm assuming that the "pod" is exposing port 80, right? I see that in the code that it's listening on 0.0.0.0:80.

Also, I'm not seeing logs and when I delete the pod, it gets stuck in terminating.

I have the same problem with microk8s

squillace · 2023-01-05T14:57:07Z

@defims do you have the walkthrough for bootstrapping and using in microk8s? That would be useful AND I can test your repro

defims · 2023-01-12T07:53:07Z

@defims do you have the walkthrough for bootstrapping and using in microk8s? That would be useful AND I can test your repro

@squillace I use macos and my steps is below:

install microk8s on macos:

brew install ubuntu/microk8s/microk8s
microk8s install
microk8s shell microk8s-vm
sudo usermod -aG microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
microk8s enable dns
microk8s stop
microk8s start

install wasmtime and build the shims

cd ~
curl https://wasmtime.dev/install.sh -sSf | bash
git clone https://github.com/containerd/runwasi.git runwasi-wasmtime
sudo apt-get install build-essential
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cd runwasi-wasmtime
make build
make install

configue containerd

edit containerd config(/var/snap/microk8s/4172/args/containerd-template.toml), add 5 line:

    # add 5 line blew
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
      runtime_type = "io.containerd.wasmedge.v1"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime]
      runtime_type = "io.containerd.wasmtime.v1"

and the total file is:

# Use config version 2 to enable new configuration fields.
version = 2
oom_score = 0

[grpc]
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

[cgroup]
  path = ""


# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
[plugins."io.containerd.grpc.v1.cri"]

  stream_server_address = "127.0.0.1"
  stream_server_port = "0"
  enable_selinux = false
  # sandbox_image = "k8s.gcr.io/pause:3.1"
  sandbox_image = "lank8s.cn/pause:3.7"
  stats_collect_period = 10
  enable_tls_streaming = false
  max_container_log_line_size = 16384

  # 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
  [plugins."io.containerd.grpc.v1.cri".containerd]

    # snapshotter is the snapshotter used by containerd.
    snapshotter = "${SNAPSHOTTER}"

    # no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.
    # This only works for runtime type "io.containerd.runtime.v1.linux".
    no_pivot = false

    # default_runtime_name is the default runtime name to use.
    default_runtime_name = "${RUNTIME}"

    # add 5 line blew
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
      runtime_type = "io.containerd.wasmedge.v1"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime]
      runtime_type = "io.containerd.wasmtime.v1"

    # 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
    # of runtime configurations, to the matching configurations.
    # In this example, 'runc' is the RuntimeHandler string to match.
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "${RUNTIME_TYPE}"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "${RUNTIME_TYPE}"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime.options]
        BinaryName = "nvidia-container-runtime"

   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
        BinaryName = "kata-runtime"

  # 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
  [plugins."io.containerd.grpc.v1.cri".cni]
    # bin_dir is the directory in which the binaries for the plugin is kept.
    bin_dir = "${SNAP_DATA}/opt/cni/bin"

    # conf_dir is the directory in which the admin places a CNI conf.
    conf_dir = "${SNAP_DATA}/args/cni-network"

  # 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to the registry
  [plugins."io.containerd.grpc.v1.cri".registry]

    # 'plugins."io.containerd.grpc.v1.cri".registry.mirrors' are namespace to mirror mapping for all namespaces.
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", ]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
        endpoint = ["http://localhost:32000"]

build wasi-demo-app and run it with ctr

cd crates/wasi-demo-app && cargo build --features oci-v1-tar
microk8s ctr image import --all-platforms img.tar
microk8s ctr run --rm --runtime=io.containerd.wasmtime.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest wasi-demo-app

test wasmedge server code

As I know wasi network standard still on the way, and wasmedge is similar to wasmtime, and second-state/runwasi is a fork of containerd/runwasi, but there is no way to post issue to second-state/runwasi, so the test code below is base on wasmedge

install wasmedge and shims

cd ~ 
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- -p /usr/local
git clone https://github.com/second-state/runwasi.git
cd runwasi
git submodule init
git submodule update
rustup default nightly
make build FEATURES=wasmedge
sudo make install RUNTIME=wasmedge

build hyper-demo and run it with ctr

sudo apt-get install docker-ce
sudo make demo/out/hyper_img.tar
microk8s ctr images --all-platforms import demo/out/hyper_img.tar
microk8s ctr run --rm --net-host --runtime=io.containerd.wasmedge.v1 docker.io/library/hyper-demo:latest testclient /client.wasm

run it with kubectl

microk8s kubectl apply -f wasm.yml

wasm.yml:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: discuzq
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wasmtest
            port:
              number: 3000
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
---
kind: Service
apiVersion: v1
metadata:
  name: wasmtest
  labels:
    name: wasmtest
spec:
  ports:
  - name: wasmtest3000
    protocol: TCP
    port: 3000
  selector:
    app: wasmtest
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: wasmtest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wasmtest
  template:
    metadata:
      labels:
        app: wasmtest
    spec:
      runtimeClassName: wasmedge
      containers:
      - name: wasmtest
        image: docker.io/library/hyper-demo:latest
        imagePullPolicy: Never 
        ports:
        - containerPort: 3000

visit 192.168.205.3:8080

got bad gateway

defims · 2023-02-06T07:19:21Z

@defims do you have the walkthrough for bootstrapping and using in microk8s? That would be useful AND I can test your repro

@squillace I use macos and my steps is below:

install microk8s on macos:

brew install ubuntu/microk8s/microk8s
microk8s install
microk8s shell microk8s-vm
sudo usermod -aG microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
microk8s enable dns
microk8s stop
microk8s start

install wasmtime and build the shims

cd ~
curl https://wasmtime.dev/install.sh -sSf | bash
git clone https://github.com/containerd/runwasi.git runwasi-wasmtime
sudo apt-get install build-essential
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cd runwasi-wasmtime
make build
make install

configue containerd

edit containerd config(/var/snap/microk8s/4172/args/containerd-template.toml), add 5 line:

    # add 5 line blew
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
      runtime_type = "io.containerd.wasmedge.v1"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime]
      runtime_type = "io.containerd.wasmtime.v1"

and the total file is:

# Use config version 2 to enable new configuration fields.
version = 2
oom_score = 0

[grpc]
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

[cgroup]
  path = ""


# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
[plugins."io.containerd.grpc.v1.cri"]

  stream_server_address = "127.0.0.1"
  stream_server_port = "0"
  enable_selinux = false
  # sandbox_image = "k8s.gcr.io/pause:3.1"
  sandbox_image = "lank8s.cn/pause:3.7"
  stats_collect_period = 10
  enable_tls_streaming = false
  max_container_log_line_size = 16384

  # 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
  [plugins."io.containerd.grpc.v1.cri".containerd]

    # snapshotter is the snapshotter used by containerd.
    snapshotter = "${SNAPSHOTTER}"

    # no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.
    # This only works for runtime type "io.containerd.runtime.v1.linux".
    no_pivot = false

    # default_runtime_name is the default runtime name to use.
    default_runtime_name = "${RUNTIME}"

    # add 5 line blew
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
      runtime_type = "io.containerd.wasmedge.v1"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmtime]
      runtime_type = "io.containerd.wasmtime.v1"

    # 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
    # of runtime configurations, to the matching configurations.
    # In this example, 'runc' is the RuntimeHandler string to match.
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "${RUNTIME_TYPE}"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "${RUNTIME_TYPE}"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime.options]
        BinaryName = "nvidia-container-runtime"

   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
        BinaryName = "kata-runtime"

  # 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
  [plugins."io.containerd.grpc.v1.cri".cni]
    # bin_dir is the directory in which the binaries for the plugin is kept.
    bin_dir = "${SNAP_DATA}/opt/cni/bin"

    # conf_dir is the directory in which the admin places a CNI conf.
    conf_dir = "${SNAP_DATA}/args/cni-network"

  # 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to the registry
  [plugins."io.containerd.grpc.v1.cri".registry]

    # 'plugins."io.containerd.grpc.v1.cri".registry.mirrors' are namespace to mirror mapping for all namespaces.
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", ]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
        endpoint = ["http://localhost:32000"]

build wasi-demo-app and run it with ctr

cd crates/wasi-demo-app && cargo build --features oci-v1-tar
microk8s ctr image import --all-platforms img.tar
microk8s ctr run --rm --runtime=io.containerd.wasmtime.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest wasi-demo-app

test wasmedge server code

As I know wasi network standard still on the way, and wasmedge is similar to wasmtime, and second-state/runwasi is a fork of containerd/runwasi, but there is no way to post issue to second-state/runwasi, so the test code below is base on wasmedge

install wasmedge and shims

cd ~ 
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- -p /usr/local
git clone https://github.com/second-state/runwasi.git
cd runwasi
git submodule init
git submodule update
rustup default nightly
make build FEATURES=wasmedge
sudo make install RUNTIME=wasmedge

build hyper-demo and run it with ctr

sudo apt-get install docker-ce
sudo make demo/out/hyper_img.tar
microk8s ctr images --all-platforms import demo/out/hyper_img.tar
microk8s ctr run --rm --net-host --runtime=io.containerd.wasmedge.v1 docker.io/library/hyper-demo:latest testclient /client.wasm

run it with kubectl

microk8s kubectl apply -f wasm.yml

wasm.yml:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: discuzq
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wasmtest
            port:
              number: 3000
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
---
kind: Service
apiVersion: v1
metadata:
  name: wasmtest
  labels:
    name: wasmtest
spec:
  ports:
  - name: wasmtest3000
    protocol: TCP
    port: 3000
  selector:
    app: wasmtest
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: wasmtest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wasmtest
  template:
    metadata:
      labels:
        app: wasmtest
    spec:
      runtimeClassName: wasmedge
      containers:
      - name: wasmtest
        image: docker.io/library/hyper-demo:latest
        imagePullPolicy: Never 
        ports:
        - containerPort: 3000

visit 192.168.205.3:8080

got bad gateway

I test the new vesion with k3s and everything works.
#52 (comment)

Mossaka · 2023-02-07T18:17:11Z

Thanks for sharing!! @defims

defims · 2023-02-09T05:20:14Z

I make a pull request #66, so k3s with runwasi can be easily tested by running make test/k3s

Mossaka · 2023-02-15T01:19:23Z

Closing this one as it's documented in #66

cpuguy83 assigned Mossaka May 17, 2022

cpuguy83 added question Further information is requested documentation Improvements or additions to documentation labels May 17, 2022

rumpl pushed a commit to rumpl/runwasi that referenced this issue Nov 23, 2022

Merge pull request containerd#20 from second-state/vincent/fix-ci-fail

32a28f7

Update submodule to fix CI build fail

Mossaka closed this as completed Feb 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document full set up with k3s #20

Document full set up with k3s #20

agracey commented May 17, 2022

Mossaka commented May 18, 2022

agracey commented Sep 5, 2022

defims commented Jan 2, 2023

squillace commented Jan 5, 2023

defims commented Jan 12, 2023 •

edited

Loading

defims commented Feb 6, 2023

install microk8s on macos:

install wasmtime and build the shims

configue containerd

build wasi-demo-app and run it with ctr

test wasmedge server code

install wasmedge and shims

build hyper-demo and run it with ctr

run it with kubectl

visit 192.168.205.3:8080

Mossaka commented Feb 7, 2023

defims commented Feb 9, 2023 •

edited

Loading

Mossaka commented Feb 15, 2023

Document full set up with k3s #20

Document full set up with k3s #20

Comments

agracey commented May 17, 2022

Mossaka commented May 18, 2022

agracey commented Sep 5, 2022

defims commented Jan 2, 2023

squillace commented Jan 5, 2023

defims commented Jan 12, 2023 • edited Loading

install microk8s on macos:

install wasmtime and build the shims

configue containerd

build wasi-demo-app and run it with ctr

test wasmedge server code

install wasmedge and shims

build hyper-demo and run it with ctr

run it with kubectl

visit 192.168.205.3:8080

defims commented Feb 6, 2023

install microk8s on macos:

install wasmtime and build the shims

configue containerd

build wasi-demo-app and run it with ctr

test wasmedge server code

install wasmedge and shims

build hyper-demo and run it with ctr

run it with kubectl

visit 192.168.205.3:8080

Mossaka commented Feb 7, 2023

defims commented Feb 9, 2023 • edited Loading

Mossaka commented Feb 15, 2023

defims commented Jan 12, 2023 •

edited

Loading

defims commented Feb 9, 2023 •

edited

Loading