feat: GPU Docker Image?

[Elsayed91]

Problem
A well crafted docker image would go a long way making the tool production ready. One can start running it in kubernetes pods and see how it goes.

Currently I have successfully run the CPU version on multiple systems, however the GPU version has not worked for me locally, nor on GCP VM, nor on Runpod. on GCP VM i had Cuda 12.1, on runpod 11.8.
on Runpod I had 'libcudart.so.12: cannot open shared object file: No such file or directory` and on GCP I am having

nitro: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by nitro)
nitro: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by nitro)
nitro: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by nitro)
nitro: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by nitro)
nitro: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by nitro)

attempting to manually install libraries has not helped me at all.

Success Criteria

1. A dockerfile that containerizes the environment needed to run with GPU

Additional Context

strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX  # For libstdc++
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBCXX_DEBUG_MESSAGE_LENGTH

[dpkirchner]

I don't know if this is the perfect solution, but try this (using nvidia's cuda base image):

FROM nvidia/cuda:12.3.1-runtime-ubuntu22.04

RUN apt-get update && \
    apt-get install -y curl

RUN curl -sfL https://raw.githubusercontent.com/janhq/nitro/main/install.sh | bash

EXPOSE 3928

ENTRYPOINT ["nitro"]
CMD ["1", "0.0.0.0", "3928"]

and then run with something like:

docker build -t nitro-image . && docker run --rm -it --gpus all -p 8081:3928

This seems to work for me, but I can't say for sure that it's actually using the GPU.

[tikikun]

hi @Elsayed91 i have a working CUDA example dockerfile in my homelab, i will update that for everyone to try also

[tikikun]

#355

see example

[Elsayed91]

Hi, I tested out the dockerfile, gave me an error. Fixed by adding the line below

ENV LD_LIBRARY_PATH=/usr/local/cuda-12.3/compat:$LD_LIBRARY_PATH

Error: /app/nitro/build/nitro: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory error.