feat: Implement Remote server and Dokploy Server - GPU Support for Docker Swarm #590

vishalkadam47 · 2024-10-24T21:57:49Z

This PR introduces GPU support in Dokploy by leveraging Docker Swarm's native GPU resource management capabilities. The implementation follows the Docker Swarm GPU configuration guidelines, enabling seamless integration of GPU resources across the infrastructure.

Key aspects of the implementation:

Configured Docker Swarm to advertise GPU resources using DOCKER_RESOURCE_GPU

Integrated GPU resource allocation for container deployments

Added API endpoint for GPU setup management

Implemented system-wide GPU constants for consistent resource handling

The GPU support work seamlessly at the server level, allowing applications to utilize GPU resources when available.

This enhancement makes Dokploy more versatile for GPU-intensive workloads while maintaining compatibility with non-GPU environments.

This feature significantly expands Dokploy's capabilities, enabling efficient deployment and management of GPU-dependent applications through Docker Swarm's native resource management system.

…PI endpoint for setup

Siumauricio

The implementation looks good, I haven tested yet, we have two way of deploying applications:

Dopkloy Server(Done)
Remote servers(Missing)

Can you add support for remote servers?

packages/server/src/constants/index.ts

apps/dokploy/templates/blender/docker-compose.yml

vishalkadam47 · 2024-10-27T16:26:10Z

The current PR of GPU support in dokploy works at both the server level and the remote server level. This means that the GPU-enabled containers can be deployed on both the dokploy server and on remote servers that are connected to the dokploy platform.

Remote Server GPU Support: The user needs to ensure that the necessary GPU hardware and drivers are installed and configured on the remote server. Once this is done, the user can simply add the GPU configuration to their
docker-compose.yml file, and the GPU-enabled containers will be able to utilize the GPU resources on the remote server.

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu

Additional Implementation: if required we can add an option under new server Deployment Setup Server when checked run a script verify that the necessary GPU hardware and drivers are installed and configured on the remote server.

For Reference: You can review the recording here

… GPU constants from index.ts

Siumauricio

I don't know where to activate it because I don't see any way to trigger the setupGPU method other than through the API, you need to expose some way to do it trought the UI

apps/dokploy/server/api/trpc.ts

packages/server/src/utils/gpu-setup.ts

…us checks, including API endpoints and utility functions

…tions

…ate environment variables

vishalkadam47

fixed the changes as suggested

Siumauricio

I tested and works great and the UI Looks awesome

I rent a GPU Droplet in digitalocean and I tried to see the GPU Setup and I get this:

Which Is fine

Then I tried to setup the GPU and I got this error

"NVIDIA drivers or runtime not installed. Please install them first.",

Then I look in to the code and commented this code, I think here we need to do something because here are validating if we already have enabled the docker gpu runtime in docker swarm, but that is wrong because we are not setup yet, so first we need to setup and then we can check that validation

GPU Status Check Result: {
  driverInstalled: true,
  driverVersion: '535.183.06',
  runtimeInstalled: false,
  runtimeConfigured: false,
  availableGPUs: 1,
  swarmEnabled: false,
  gpuResources: 0,
  gpuModel: 'NVIDIA H100 80GB HBM3',
  memoryInfo: '81559 MiB',
  cudaSupport: true,
  cudaVersion: '12.2'
}

Then it works

Also before i setup the GPU Setup I tried to deploy the blender template and I have this error

But when I activate the gpu it works now

I think it works very good, just need a few validations as I mentioned above

apps/dokploy/components/dashboard/settings/servers/gpu-support.tsx

apps/dokploy/server/api/routers/settings.ts

packages/server/src/utils/gpu-setup.ts

vishalkadam47 · 2024-11-07T13:56:33Z

I tested and works great and the UI Looks awesome

I rent a GPU Droplet in digitalocean and I tried to see the GPU Setup and I get this:

Which Is fine

Then I tried to setup the GPU and I got this error

"NVIDIA drivers or runtime not installed. Please install them first.",

Then I look in to the code and commented this code, I think here we need to do something because here are validating if we already have enabled the docker gpu runtime in docker swarm, but that is wrong because we are not setup yet, so first we need to setup and then we can check that validation
GPU Status Check Result: {
  driverInstalled: true,
  driverVersion: '535.183.06',
  runtimeInstalled: false,
  runtimeConfigured: false,
  availableGPUs: 1,
  swarmEnabled: false,
  gpuResources: 0,
  gpuModel: 'NVIDIA H100 80GB HBM3',
  memoryInfo: '81559 MiB',
  cudaSupport: true,
  cudaVersion: '12.2'
}
Then it works

Also before i setup the GPU Setup I tried to deploy the blender template and I have this error

But when I activate the gpu it works now

I think it works very good, just need a few validations as I mentioned above

NVIDIA Container Runtime is not installed. Please follow these steps:

Install NVIDIA Container Runtime:
Ubuntu/Debian:

curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | \
  sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
sudo apt-get update
sudo apt-get install nvidia-container-runtime

Note: for some VPS providers may have nvidia-container-runtime pre-installed, @Siumauricio Will update proper checks

NVIDIA Container Runtime is installed but not configured.

Configuration is handled when we click the Enable GPU button in the UI.

Siumauricio · 2024-11-07T18:35:17Z

I tested and works great and the UI Looks awesome
I rent a GPU Droplet in digitalocean and I tried to see the GPU Setup and I get this:
Which Is fine
Then I tried to setup the GPU and I got this error
"NVIDIA drivers or runtime not installed. Please install them first.",
Then I look in to the code and commented this code, I think here we need to do something because here are validating if we already have enabled the docker gpu runtime in docker swarm, but that is wrong because we are not setup yet, so first we need to setup and then we can check that validation
GPU Status Check Result: {
  driverInstalled: true,
  driverVersion: '535.183.06',
  runtimeInstalled: false,
  runtimeConfigured: false,
  availableGPUs: 1,
  swarmEnabled: false,
  gpuResources: 0,
  gpuModel: 'NVIDIA H100 80GB HBM3',
  memoryInfo: '81559 MiB',
  cudaSupport: true,
  cudaVersion: '12.2'
}
Then it works

Also before i setup the GPU Setup I tried to deploy the blender template and I have this error
But when I activate the gpu it works now
I think it works very good, just need a few validations as I mentioned above
NVIDIA Container Runtime is not installed. Please follow these steps:
Install NVIDIA Container Runtime:
Ubuntu/Debian:
curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | \
  sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
sudo apt-get update
sudo apt-get install nvidia-container-runtime
Note: for some VPS providers may have nvidia-container-runtime pre-installed, @Siumauricio Will update proper checks

NVIDIA Container Runtime is installed but not configured.

Configuration is handled when we click the Enable GPU button in the UI.

I don't think that's the solution, because I didn't have to install anything in that gpu droplet, there is only one validation that you don't have to do when you want to do the gpu configuration, and that was the portion of the code that I commented

vishalkadam47 · 2024-11-07T20:08:05Z

I tested and works great and the UI Looks awesome
I rent a GPU Droplet in digitalocean and I tried to see the GPU Setup and I get this:
Which Is fine
Then I tried to setup the GPU and I got this error
"NVIDIA drivers or runtime not installed. Please install them first.",
Then I look in to the code and commented this code, I think here we need to do something because here are validating if we already have enabled the docker gpu runtime in docker swarm, but that is wrong because we are not setup yet, so first we need to setup and then we can check that validation
GPU Status Check Result: {
  driverInstalled: true,
  driverVersion: '535.183.06',
  runtimeInstalled: false,
  runtimeConfigured: false,
  availableGPUs: 1,
  swarmEnabled: false,
  gpuResources: 0,
  gpuModel: 'NVIDIA H100 80GB HBM3',
  memoryInfo: '81559 MiB',
  cudaSupport: true,
  cudaVersion: '12.2'
}
Then it works

Also before i setup the GPU Setup I tried to deploy the blender template and I have this error
But when I activate the gpu it works now
I think it works very good, just need a few validations as I mentioned above
NVIDIA Container Runtime is not installed. Please follow these steps:
Install NVIDIA Container Runtime:
Ubuntu/Debian:
curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | \
  sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
sudo apt-get update
sudo apt-get install nvidia-container-runtime
Note: for some VPS providers may have nvidia-container-runtime pre-installed, @Siumauricio Will update proper checks

NVIDIA Container Runtime is installed but not configured.

Configuration is handled when we click the Enable GPU button in the UI.
I don't think that's the solution, because I didn't have to install anything in that gpu droplet, there is only one validation that you don't have to do when you want to do the gpu configuration, and that was the portion of the code that I commented

I understand your point. Some GPU VM instance comes with everything pre-installed, so there's no additional setup needed. However, for daemon.json configuration, it's necessary to have the NVIDIA Container Runtime installed. This is why we need a check to notify users to install it if it’s missing.

To improve this process, I’ve separated out the check, which should make it more effective. I’ll push an update shortly, and would appreciate your feedback.

This check is also valuable for local GPU setups, as we’re using the same components across both Dokploy Server and Remote Server.

… error handling

- Add gpu status refresh with useEffect - Update docker-compose.yml configuration - Modify gpu setup scripts - Improve gpu support checks

apps/dokploy/server/api/routers/settings.ts

packages/server/src/utils/gpu-setup.ts

vishalkadam47

removed console logs and error handling

vishalkadam47

removed sleep function and updated import

Siumauricio

Thank you for this huge PR ❤️

vishalkadam47 and others added 7 commits October 17, 2024 15:45

feat: Add server-level GPU support for Docker Swarm deployments and A…

15a76d2

…PI endpoint for setup

feat: Added Blender template

e52a0fc

Merge branch 'Dokploy:canary' into feature/gpu-support-blender-template

6d3ea8d

Merge branch 'Dokploy:canary' into feature/gpu-support-blender-template

4334301

Merge branch 'Dokploy:canary' into feature/gpu-support-blender-template

96c5176

fix: Remove privileged mode and seccomp option, update runtime to nvidia

5a440d9

Merge branch 'Dokploy:canary' into feature/gpu-support-blender-template

b309269

vishalkadam47 requested a review from Siumauricio as a code owner October 24, 2024 21:57

Siumauricio self-assigned this Oct 27, 2024

Siumauricio requested changes Oct 27, 2024

View reviewed changes

packages/server/src/constants/index.ts Outdated Show resolved Hide resolved

apps/dokploy/templates/blender/docker-compose.yml Outdated Show resolved Hide resolved

refactor: Update docker-compose.yml to remove port mapping and remove…

3e46795

… GPU constants from index.ts

Siumauricio requested changes Oct 27, 2024

View reviewed changes

apps/dokploy/server/api/trpc.ts Outdated Show resolved Hide resolved

packages/server/src/utils/gpu-setup.ts Outdated Show resolved Hide resolved

vishalkadam47 changed the title ~~Implement Server-Level GPU Support for Docker Swarm and Add Blender Template~~ Implement Remote server and Dokploy Server - GPU Support for Docker Swarm Nov 2, 2024

vishalkadam47 added 4 commits November 2, 2024 15:15

feat: Added GPU support feature for Remote Server with setup and stat…

1b6d8d8

…us checks, including API endpoints and utility functions

fix: Remove unused imports and interfaces from gpu-setup.ts

ed7150f

fix: resolve merge conflicts with upstream/canary

bc097c7

feat: Add GPU configuration and Update import path for gpu-setup func…

7306d8c

…tions

vishalkadam47 requested a review from Siumauricio November 3, 2024 16:12

vishalkadam47 and others added 4 commits November 5, 2024 12:01

Merge branch 'Dokploy:canary' into feature/gpu-support-blender-template

3b5e892

refactor: gpu support component and related api routers; update templ…

b53da82

…ate environment variables

feat: add dokploy server gpu setup

2e6d9c3

Merge branch 'Dokploy:canary' into feature/gpu-support-blender-template

a331020

vishalkadam47 commented Nov 6, 2024

View reviewed changes

Siumauricio requested changes Nov 7, 2024

View reviewed changes