[Runpod] Require CUDA 12.8+ on the host

[diagonalge]

Original request:

### Problem

There is no option currently to specify the cuda version of the instance on runpod backend.

### Solution

Use allowed_cuda_versions parameter on the runpod api to have the option.

### Workaround

_No response_

### Would you like to help us implement this feature by sending a PR?

No

Updated description:

Problem

Currently, gpuhut returns all offers, including the offers that come with too old CUDA version, e.g. 11.x. Because dstack doesn't allow to filter that, it can be that sometimes a user is using 12.4 CUDA version inside the Docker image but the host can have, e.g. 11.x version.

Solution

Filter out the offers withe the hosts that have too old CUDA version

[peterschmidt85]

@diagonalge Would it also work for you if we set the default min CUDA version to 12.8? This is the version we require on the host for other backends at the moment.

[diagonalge]

Yes that could work i guess. Even though they should be backwards compatible, but I am getting a mismatch. Setting a min of 12.8 could work and then I could use 12.8 in my images.

[peterschmidt85]

Even though they should be backwards compatible, but I am getting a mismatch.

Just for information, could you please share which host version is there and which version are you using inside the Docker image when you get a mismatch?

[diagonalge]

I am not exactly sure about the host version really. But I was using 12.4 in the image.