Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run SGX in HW mode in k8s #612

Merged
merged 12 commits into from
Mar 14, 2022
Merged

Run SGX in HW mode in k8s #612

merged 12 commits into from
Mar 14, 2022

Conversation

csegarragonz
Copy link
Collaborator

@csegarragonz csegarragonz commented Mar 11, 2022

In this PR I Introduce the necessary changes to allow running SGX functions in hardware mode in a kubernetes cluster. Most importantly, we need to build different image tags for workers with a hardware-mode SGX build.

In addition, when deploying the KNative services we need to: use the new worker docker image, and set the environment variable WASM_VM=sgx for both the upload and worker pods.
For the upload service, we can either use templated deployment files/env. var substitution or have separate deployment files. For the worker service I think we need a different file, as we have to add sgx-specific reources and we may have to add further options in the worker deployment file for attestation.

I have manually checked that with the current setup, and deploying as follows, we can run SGX functions in HW mode in AKS:

cd ~/experiment-base
inv cluster.provision --vm Standard_DC4s_v3 --nodes 4 --location eastus2 --sgx
inv cluster.credentials

cd ~/faasm
inv knative.install # run twice
inv knative.deploy --replicas=4 --sgx

inv upload demo hello <path_to_wasm>
inv invoke demo hello

I also bump the code version to test the new "Release" workflow file.

@csegarragonz csegarragonz added the wasm/wamr-sgx SGX related stuff. label Mar 11, 2022
@csegarragonz csegarragonz self-assigned this Mar 11, 2022
RUN cmake \
-GNinja \
-DCMAKE_CXX_COMPILER=/usr/bin/clang++-13 \
-DCMAKE_C_COMPILER=/usr/bin/clang-13 \
-DCMAKE_BUILD_TYPE=Release \
-DFAASM_SGX_MODE=Simulation \
-DFAASM_SGX_MODE=${FAASM_SGX_MODE:-Simulation} \
Copy link
Collaborator Author

@csegarragonz csegarragonz Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If FAASM_SGX_MODE is not set, use Simulation, if it is set, use ${FAASM_SGX_MODE}. Link to bash wiki..

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually don't need to link to syntax in a PR unless it's some obscure library that you're introducing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I always get confused with bash variable's expansion, so added the link and the explanation more for me than for you 🤣

value: "info"
- name: LD_LIBRARY_PATH
value: "/build/faasm/third-party/lib:/usr/local/lib"
- name: WASM_VM
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is a copy of deploy/k8s/upload.yml with just one extra environment variable.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than mixing these in the same directory I feel like having a totally separate deploy/k8s-sgx directory would be cleaner, then we avoid all the filtering logic in Python. Anything that's shared between vanilla and SGX would go into another new directory, deploy/k8s-common.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this is cleaner. We now have: deploy/{k8s-common,k8s-sgx,k8s}.

@csegarragonz csegarragonz requested review from Shillaker and removed request for Shillaker March 11, 2022 17:31
@csegarragonz csegarragonz marked this pull request as ready for review March 11, 2022 17:32
@@ -92,6 +92,34 @@ jobs:
context: .
tags: faasm/base:${{ env.TAG_VERSION }}

build-base-sgx:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is getting quite long and verbose, we could redo this with a matrix build like we do the tests. Doesn't need to be in this PR but will add to the todo list.

value: "info"
- name: LD_LIBRARY_PATH
value: "/build/faasm/third-party/lib:/usr/local/lib"
- name: WASM_VM
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than mixing these in the same directory I feel like having a totally separate deploy/k8s-sgx directory would be cleaner, then we avoid all the filtering logic in Python. Anything that's shared between vanilla and SGX would go into another new directory, deploy/k8s-common.

deploy/k8s/upload-sgx.yml Outdated Show resolved Hide resolved
deploy/k8s/worker-sgx.yml Outdated Show resolved Hide resolved
RUN cmake \
-GNinja \
-DCMAKE_CXX_COMPILER=/usr/bin/clang++-13 \
-DCMAKE_C_COMPILER=/usr/bin/clang-13 \
-DCMAKE_BUILD_TYPE=Release \
-DFAASM_SGX_MODE=Simulation \
-DFAASM_SGX_MODE=${FAASM_SGX_MODE:-Simulation} \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually don't need to link to syntax in a PR unless it's some obscure library that you're introducing.

faasmcli/faasmcli/tasks/sgx.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@Shillaker Shillaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, looks good.

Can you triple check that running inv version.bump still works across all the k8s files? Looks like the task does whatever it finds in deploy, but just want to make sure.

ports:
- port: 8002
selector:
role: upload
Copy link
Collaborator

@Shillaker Shillaker Mar 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@csegarragonz I think this is now a duplicate of what's in upload-lb.yml. Missed it in my review.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, thanks. i copied the file before removing this 😓

@csegarragonz csegarragonz merged commit 9cca59b into main Mar 14, 2022
@csegarragonz csegarragonz deleted the sgx-hw-k8s branch March 14, 2022 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wasm/wamr-sgx SGX related stuff.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants