Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Execute OpenWhisk action via Ignite and Firecracker VM #4556

Draft
wants to merge 23 commits into
base: master
from

Conversation

@chetanmeh
Copy link
Member

chetanmeh commented Jul 12, 2019

This is a proof of concept to execute OpenWhisk actions via Weave Ignite which uses Firecracker MicroVMs

Recently Weave works announced Ignite project which supports running compatible docker images as Firecracker MicroVMs. Such an execution can provide better security compared to simple container based execution.

Objective

Some of the key aspects here are

  1. Rebuild OpenWhisk runtime as Ignote compatible docker image
  2. Execute action via vm build from those images

Usage

1. Install Ignite

Follow the Ignite installation steps and ensure that required dependencies are met. OpenWhisk would check for ignite cli at /usr/local/bin/ignite

2. Build a Ignite compatible docker image for nodejs.

Currently a docker file is provided for nodejs 12 here

$ cd tools/ignite/nodejs
$ docker build -t whisk/ignite-nodejs-v12:latest  .

3. Build the standalone OpenWhisk

$ ./gradlew :core:standalone:build

4. Start the server

Start the server passing a custom runtime manifest which refers to new nodejs docker image build in previous steps

$ sudo java -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.ignite.IgniteContainerFactoryProvider \
      -jar bin/openwhisk-standalone.jar \
      -m tools/ignite/ignite-runtimes.json 

Note that this is run with sudo as ignite needs root access to run. Once started it would launch 2 pre warm containers (vms!)

In startup logs you can see the vm being created

[2019-07-12T18:12:07.811Z] [INFO] [#tid_sid_unknown] [IgniteClient] Detected ignite client version Ignite version: version.Info{Major:"0", Minor:"4", GitVersion:"v0.4.0", GitCommit:"e0e7e8c50cd3c6532486625393dff4d415081829", GitTreeState:"clean", BuildDate:"2019-07-10T12:30:11Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Firecracker version: v0.17.0
[2019-07-12T18:12:08.080Z] [INFO] [#tid_sid_invokerWarmup] [IgniteClient] running /usr/local/bin/ignite -q image import whisk/ignite-nodejs-v12:latest (timeout: 10 minutes) [marker:invoker_ignite.image_start:1384]
[2019-07-12T18:12:08.101Z] [INFO] [#tid_sid_invokerWarmup] [IgniteClient] running /usr/local/bin/ignite -q run whisk/ignite-nodejs-v12:latest --cpus 1 --memory 256m --size 1GB --name wsk0_1_prewarm_nodejs12 (timeout: 5 minutes) [marker:invoker_ignite.run_start:1406]

Checking running vms

# ignite vm ps 
VM ID                   IMAGE                           KERNEL                                  CREATED SIZE            CPUS    MEMORY          STATE   IPS             PORTS       NAME
296c4f2fd75bf834        whisk/ignite-nodejs-v12:latest  weaveworks/ignite-kernel:4.19.47        17s ago 1024.0 MB       1       256.0 MB        Running 172.17.0.10        wsk0_1_prewarm_nodejs12
e9c77eb3f95e7d7c        whisk/ignite-nodejs-v12:latest  weaveworks/ignite-kernel:4.19.47        17s ago 1024.0 MB       1       256.0 MB        Running 172.17.0.11        wsk0_2_prewarm_nodejs12

There would also be 2 "proxy" docker container for these 2 vm

# docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                                                 NAMES
f94d7c15ed90        weaveworks/ignite:v0.4.0              "/usr/local/bin/igni…"   28 seconds ago      Up 24 seconds                                                                                                             ignite-e9c77eb3f95e7d7c
be939a1e2d5a        weaveworks/ignite:v0.4.0              "/usr/local/bin/igni…"   29 seconds ago      Up 25 seconds    

Now you can invoke js action

{
    "namespace": "guest",
    "name": "hello",
    "version": "0.0.1",
    "subject": "guest",
    "activationId": "1b86c507e4df4b6286c507e4df3b62ea",
    "start": 1562937307810,
    "end": 1562937323341,
    "duration": 15531,
    "statusCode": 0,
    "response": {
        "status": "success",
        "statusCode": 0,
        "success": true,
        "result": {
            "LANG": "en_US.UTF-8",
            "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin",
            "__OW_ACTION_NAME": "/guest/hello",
            "__OW_ACTIVATION_ID": "1b86c507e4df4b6286c507e4df3b62ea",
            "__OW_DEADLINE": "1562937383336",
            "__OW_NAMESPACE": "guest",
            "payload": "hello, world"
        }
    },
    "logs": [],
    "annotations": [
        {
            "key": "path",
            "value": "guest/hello"
        },
        {
            "key": "waitTime",
            "value": 100
        },
        {
            "key": "kind",
            "value": "nodejs:12"
        },
        {
            "key": "timeout",
            "value": false
        },
        {
            "key": "limits",
            "value": {
                "concurrency": 1,
                "logs": 10,
                "memory": 256,
                "timeout": 60000
            }
        },
        {
            "key": "initTime",
            "value": 15526
        }
    ],
    "publish": false
}

Design

This PR adds a new IgniteContainerFactory which internally launches the vm and route calls to it. See Getting started with Ignite to get a quick overview of how ignite works

Upon first launch it would

  1. Import the image via ignite image import whisk/ignite-nodejs-v12:latest. This would implicitly pull the image if not present
  2. Create and run the vm ignite run whisk/ignite-nodejs-v12:latest --cpus 1 --memory 256m --size 1GB --name wsk0_1_prewarm_nodejs12
  3. Connect to the 8080 port of launched vm for actual execution

IgniteContainer would also figure out the matching docker container running for given vm and use that to make calls as being done with standard Docker implementation.

Docker Image

For any docker image to be usable with ignite it must have init system present. For this it provides some base images. This PR uses there centos image to build a custom nodejs action runtime image

FROM openwhisk/action-nodejs-v12:1.14.0-incubating AS ownode

FROM weaveworks/ignite-centos

RUN curl -sL https://rpm.nodesource.com/setup_10.x | bash -
RUN sudo yum -y install nodejs

COPY --from=ownode /nodejsAction /nodejsAction
COPY --from=ownode /node_modules /node_modules
COPY ./action.service /etc/systemd/system/

RUN chmod 664 /etc/systemd/system/action.service \
    && systemctl enable action.service

Service Startup

Despite ignite use of OCI Image the end result is a vm being launched. So for our action runtime to be launches this docker image also adds a systemd service

[Unit]
Description=OpenWhisk Nodejs action server
Wants=network-online.target
After=network-online.target

[Service]
WorkingDirectory=/nodejsAction
ExecStart=/usr/bin/node --expose-gc app.js

[Install]
WantedBy=multi-user.target

So upon startup of vm the node server would be launched and it would bind to 8080 port.

Logging

Currently logs support does not work. For that we need to figure out how to route nodejs logs to vm logs. These logs are then routed to docker logging sub system so we can then fetch it via standard docker log access.

Things to note

Currently the cold start takes long time as for some reason even if vm launches quickly the nodejs server does not respond for some time post start. This aspect would need to be investigated

@sven-lange-last

This comment has been minimized.

Copy link
Member

sven-lange-last commented Jul 14, 2019

@chetanmeh thanks a lot for providing this Ignite / Firecracker starter.

  • Would you know how network connectivity is realized for the created VMs?
  • Is it possible to run Firecracker VMs within a virtualization like Xen / KVM / VirtualBox or is a bare metal Linux system required?
@chetanmeh

This comment has been minimized.

Copy link
Member Author

chetanmeh commented Jul 19, 2019

Would you know how network connectivity is realized for the created VMs

Currently it supports 2 options docker-bridge and cni with docker bridge as default

Is it possible to run Firecracker VMs within a virtualization like Xen / KVM / VirtualBox or is a bare metal Linux system required?

I tested it on bare metal linux (ubuntu) box. Per firecracker docs either a bare-metal machine (with hardware virtualization), or a virtual machine that supports nested virtualization can be used

protected val igniteCmd: Seq[String] = {
val alternatives = List("/usr/bin/ignite", "/usr/local/bin/ignite")

val dockerBin = Try {

This comment has been minimized.

Copy link
@gohar94
} getOrElse {
throw new FileNotFoundException(s"Couldn't locate ignite binary (tried: ${alternatives.mkString(", ")}).")
}
Seq(dockerBin, "-q")

This comment has been minimized.

Copy link
@gohar94
@130B848

This comment has been minimized.

Copy link

130B848 commented Nov 21, 2019

When I try to start the server by

sudo java -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.ignite.IgniteContainerFactoryProvider \
      -jar bin/openwhisk-standalone.jar \
      -m tools/ignite/ignite-runtimes.json 

Errors occurr in ignite ps command:

[2019-11-21T14:02:11.322Z] [ERROR] [#tid_sid_invoker] [IgniteClient] info: command was unsuccessful, code: 1 (unsuccessful), stdout: , stderr: Error: unknown flag: --no-trunc

And even if I delete these options (--notrunc --format=...), there is still no pre warm containers launched.

Is there anything wrong with my environment configuration?

@chetanmeh

This comment has been minimized.

Copy link
Member Author

chetanmeh commented Nov 21, 2019

Have not tried the flow with current ignite version so may be it does not work with current impl and need to be adapted. --no-trunc was used to get full ps output. May be we need to use a different way

@130B848

This comment has been minimized.

Copy link

130B848 commented Nov 21, 2019

Have not tried the flow with current ignite version so may be it does not work with current impl and need to be adapted. --no-trunc was used to get full ps output. May be we need to use a different way

Thanks for the quick reply!

Since I am interested in this PR and want to reproduce this, could you please tell me some more detailed information about the environment configuration with which the ignite works well? (e.g., the version of ignite when you succeeded)

Thanks!

@chetanmeh

This comment has been minimized.

Copy link
Member Author

chetanmeh commented Nov 21, 2019

Per logs its 0.4.0

[2019-07-12T18:12:07.811Z] [INFO] [#tid_sid_unknown] [IgniteClient] Detected ignite client version Ignite version: version.Info{Major:"0", Minor:"4", GitVersion:"v0.4.0", GitCommit:"e0e7e8c50cd3c6532486625393dff4d415081829", GitTreeState:"clean", BuildDate:"2019-07-10T12:30:11Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Firecracker version: v0.17.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.