How to pass a different CUDA version when containerizing the Bento?

[justas147]

I'm trying to build a docker image with any model. However, after succesfull Bento build the containerize command fails with this error:

$ bentoml containerize facebook-opt-1-3b-service:8c7b10754972749675d22364c25c428b29face51
Building OCI-compliant image for facebook-opt-1-3b-service:8c7b10754972749675d22364c25c428b29face51 with docker

[+] Building 2.3s (3/3) FINISHED                                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                        0.1s
 => => transferring dockerfile: 3.36kB                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.2s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => ERROR [internal] load metadata for docker.io/nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04                                                                                                              1.9s
------
 > [internal] load metadata for docker.io/nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04:
------
Dockerfile:6
--------------------
   4 |
   5 |     # Block SETUP_BENTO_BASE_IMAGE
   6 | >>> FROM nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04 as base-container
   7 |
   8 |     ENV LANG=C.UTF-8
--------------------
ERROR: failed to solve: nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04: docker.io/nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04: not found

The docker tag does not exist. I saw in the documentation that it is possible to set the cuda_version in bentofile.yaml:

docker:
    cuda_version: "11.7.1"

But after adding the bentofile.yaml to the same directory I run the containerizing command from it still uses the non-existing version. So how can I set the CUDA version when containerizing the Bento?

[aarnphm]

I haven't released a new version yet, but you can try out the main build

pip install "git+https://github.com/bentoml/openllm"

I think the responsibility of CUDA should be at OpenLLM

[justas147]

Thank you very much for the help. The main build used 12.0.1-cudnn8-runtime-ubuntu20 tag and the containerize command worked.

Just now I run into a problem when starting the docker container:

$ docker run -it --rm -p 3000:3000 facebook-opt-125m-service:3d2b5f275bdf882b8775f902e1bfdb790e2cfc32 serve
Error: [bentoml-cli] `serve` failed: Model 'pt-3d2b5f275bdf882b8775f902e1bfdb790e2cfc32:9f03ddb706784703177e8a40d39a714889157507' is not found in BentoML store <osfs '/home/bentoml/bento/models'>

[ksteimel]

I was getting that error when my host file system was nearly full. Clearing out some docker images I was no longer using did the trick. Maybe something similar is happening in your case.

[aarnphm]

Can you try out 0.2.0?

quick note you can do docker system prune -a -f to just vacate all cache and overlay from docker

[justas147]

Cleared all cache and installed version 0.2.0. Tried buildint bento, but got an error about cuda version 11.8.0:

$ openllm build opt --model-id facebook/opt-125m
__tag__:pt-facebook-opt-125m:3d2b5f275bdf882b8775f902e1bfdb790e2cfc32

Exception caught during building LLM opt:
Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/openllm/cli.py", line 1894, in build_command
    bento = bentoml.get(bento_tag)
  File ".../lib/python3.8/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File ".../lib/python3.8/site-packages/bentoml/bentos.py", line 66, in get
    return _bento_store.get(tag)
  File ".../lib/python3.8/site-packages/bentoml/_internal/store.py", line 146, in get
    raise NotFound(
bentoml.exceptions.NotFound: Bento 'facebook-opt-125m-service:3d2b5f275bdf882b8775f902e1bfdb790e2cfc32' is not found in BentoML store <osfs '/home/jusrad/bentoml/bentos'>, you may need to run `bentoml models pull` first

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/openllm/cli.py", line 1914, in build_command
    bento = openllm.bundle.create_bento(
  File ".../lib/python3.8/site-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File ".../lib/python3.8/site-packages/openllm/bundle/_package.py", line 297, in create_bento
    docker=construct_docker_options(
  File ".../lib/python3.8/site-packages/openllm/bundle/_package.py", line 226, in construct_docker_options
    return DockerOptions(
  File "<attrs generated init bentoml._internal.bento.build_config.DockerOptions>", line 5, in __init__
    _setattr('cuda_version', __attr_converter_cuda_version(cuda_version))
  File ".../lib/python3.8/site-packages/bentoml/_internal/bento/build_config.py", line 91, in _convert_cuda_version
    raise BentoMLException(
bentoml.exceptions.BentoMLException: Unsupported cuda version: "11.8.0". Supported cuda versions are ['12', '12.1', '12.1.1', '12.1.0', '12.0', '12.0.1', '12.0.0', '11', '11.7', '11.7.0', '11.6', '11.6.2', '11.4', '11.4.3', '11.2', '11.2.2']

[aarnphm]

Sorry for the inconvenience. I'm currently working on this atm. I will make a patch promptly.