Feature/gpu support extended #87
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nding values.yml capability definition example
…GPU related tests
iakkus
reviewed
Oct 12, 2020
Sandbox/Dockerfile_gpu
Outdated
Comment on lines
46
to
57
| # Install dlib for CUDA | ||
| RUN git clone https://github.com/davisking/dlib.git | ||
| RUN mkdir -p /dlib/build | ||
|
|
||
| RUN cmake -H/dlib -B/dlib/build -DDLIB_USE_CUDA=1 -DUSE_AVX_INSTRUCTIONS=1 | ||
| RUN cmake --build /dlib/build | ||
|
|
||
| RUN cd /dlib; python3 /dlib/setup.py install | ||
|
|
||
| # Install the face recognition package and tensorflow | ||
| RUN pip3 install face_recognition | ||
| RUN pip3 install tensorflow==2.1.0 |
There was a problem hiding this comment.
I am not sure why we need to install all these custom libraries for the GPU usage.
If these are needed by the workflows, then they should specify it in the function requirements.
iakkus
reviewed
Oct 12, 2020
mfn_sdk/mfn_sdk/mfnclient.py
Outdated
| @@ -449,7 +449,7 @@ def _get_state_names_and_resource(self, desired_state_type, wf_dict): | |||
| return state_list | |||
|
|
|||
|
|
|||
| def add_workflow(self,name,filename=None): | |||
| def add_workflow(self,name,filename=None, gpu_usage="None"): | |||
iakkus
reviewed
Oct 12, 2020
deploy/ansible/Makefile
Outdated
| @@ -21,7 +21,7 @@ NAMES := $(YAML:%.yaml=%) | |||
| .PHONY: $(NAMES) | |||
| default: prepare_packages install | |||
|
|
|||
| install: init_once riak elasticsearch fluentbit datalayer sandbox management nginx | |||
| install: init_once installnvidiadocker riak elasticsearch fluentbit datalayer frontend sandbox management nginx | |||
There was a problem hiding this comment.
I think 'frontend' component does not exist anymore.
What happens if the host does not have any Nvidia GPUs? Will the 'installnvidiadocker' still succeed?
iakkus
reviewed
Oct 12, 2020
| @@ -107,6 +118,7 @@ image_java: \ | |||
|
|
|||
| push: image image_java | |||
| $(call push_image,microfn/sandbox) | |||
| $(call push_image,microfn/sandbox_gpu) | |||
There was a problem hiding this comment.
microfn/sandbox_java_gpu?
Need to also update the dependencies for the Makefile target.
iakkus
reviewed
Oct 12, 2020
| gpu_hosts[hostname] = hostip | ||
|
|
||
| # instruct hosts to start the sandbox and deploy workflow | ||
| if runtime=="Java" or sandbox_image_name == "microfn/sandbox": # can use any host |
There was a problem hiding this comment.
I thought we had the "microfn/sandbox_java_gpu" image?
…x-microfunctions/knix into feature/GPU_support_extended
This reverts commit 7a1b157.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the capability to execute Python KNIX functions in sandboxes using NVIDIA GPU resources for both ansible and helm deployments of KNIX. GPU nodes are detected and configured automatically. The required kubernetes configuration for deployments with GPU nodes is described in README_GPU_Installation.md
Subsumes #11, and fixes #79.