wasm, kubernetes: support wasm for kubernetes infrastructure with side-cars with wasm-smart and compat-smart annotation

[flouthoc]

Complex kubernetes infrastructure contains pod and most likely pod with
side-cars. Therefore wasm integration is not useful for cases where
side-cars are almost there in most of the deployments example
infrastrcuture setups with service-mesh like linkerd, gloo and istio
and proxy like envoy.

Following use-case is solved by adding two smart annotations for only
wasm-handlers i.e compat-smart and wasm-smart

compat-smart, wasm-smart: annotation is a smart switch which only toggles wasm
runtime if its necessary otherwise fallbacks to native container runtime
hence in deployments with side-cars only containers which contains
valid wasm workload will be treated be executed by wasm-handlers and
regular containers are treated as it is.

Simple tryout example with module.wasm.image/variant: compat-smart

apiVersion: v1
kind: Pod
metadata:
  name: wasm-hello
  annotations:
    module.wasm.image/variant: compat-smart
spec:
  containers:
  - image: busybox
    name: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  - name: wasm-hello
    image: your-wasm-image:latest
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

Closes: #829

[flouthoc]

@giuseppe PTAL. I just need to confirm if wasmedge added support for wat or not other than that this PR should be good for review and makes wasm functional for kubernetes deployments in general i.e with side-cars and init-containers

[giuseppe]

just a typo, otherwise LGTM

[flouthoc]

Hi @0xE282B0 @gattytto I have tested this various side-car patterns and also added a tryout example. If someone wants to give this a try with k8s with any of the service-mesh like linkerd could you try this out ? otherwise we can get this merged.

[0xE282B0]

Thanks for working on that @flouthoc!
My Knative and DAPR examples are running fine with module.wasm.image/variant=compat-smart, LGTM.

The only thing I'm a bit concerned about is that a sidecars entrypoint maybe ends on wat. What do you think about checking on .wat? -Since this is a file extension, an entry point should be less likely to end with it.

[giuseppe]

LGTM

[flouthoc]

Thanks for working on that @flouthoc!
My Knative and DAPR examples are running fine with module.wasm.image/variant=compat-smart, LGTM.

Thanks for trying this out and confirming.

The only thing I'm a bit concerned about is that a sidecars entrypoint maybe ends on wat. What do you think about checking on .wat? -Since this is a file extension, an entry point should be less likely to end with it.

@0xE282B0 I did not understand this question completely. I have added .wat cause wasmer supports it and I can remove it from wasmedge if its a problem. But wasmer can compile .wat and can execute it on the fly and that is my intention behind adding a .wat and hopefully .wasmedge will also support it in future.

[0xE282B0]

Sorry for being imprecise. For me it looks like has_suffix just checks if the entrypoint ends on wat|wasm and I'd suggest to check on .wat|.wasm (just a dot added) just for the unlikely case that a sidecars entrypoint accidentally ends on one of the strings.
I'm totally fine with supporting wat files 😊

[flouthoc]

Sorry for being imprecise. For me it looks like has_suffix just checks if the entrypoint ends on wat|wasm and I'd suggest to check on .wat|.wasm (just a dot added) just for the unlikely case that a sidecars entrypoint accidentally ends on one of the strings. I'm totally fine with supporting wat files blush

Oh fair request. Let me amend the the commit.

[flouthoc]

@rhatdan @giuseppe Pushed refactor to a new commit could you PTAL.

[rhatdan]

@giuseppe Might want to extend your copyright...

[rhatdan]

LGTM

[giuseppe]

LGTM

[flouthoc]

Hi @juntao @hydai, I think it would be nice if wasmedge docs could also point to these new annotations since they might be useful for dapr and other k8s infra.

Newly added annotations are

• module.wasm.image/variant: compat-smart
• run.oci.handler: wasm-smart

[0xE282B0]

@flouthoc I can open an issue in the wasmedge repo to add the annotations to the docs.

[flouthoc]

@0xE282B0 Thanks.

[rhatdan]

We really need to have a nice blog/article/demo showing us using crun/wasm to run containers.

Would it make sense to have entrypoint oci-runtimes named wasm-smart and compat-smart and just specify the oci runtime rather then setting annotations?

[flouthoc]

@rhatdan I think the convention was initially set by solo.io and it gives freedom to always not opt for wasm and use crun as regular runtime but yes I agree we need more comprehensive examples and write-up on this. I'll pick it up.