bug: Missing Dependency when running

[IbrahimMCode]

Describe the bug

Hello,
I followed the instructions on github (nothing more) and when I try to run it with the following command:
sudo docker run -it --rm -p 3000:3000 google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce serve
I'm getting the following error.

Error: [bentoml-cli] serve failed: Failed loading Bento from directory /home/bentoml/bento: Failed to import module "generated_flan_t5_service": No module named 'orjson'

To reproduce

No response

Logs

`sudo docker run -it --rm -p 3000:3000 google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce serve
`

`Error: [bentoml-cli] `serve` failed: Failed loading Bento from directory /home/bentoml/bento: Failed to import module "generated_flan_t5_service": No module named 'orjson'`

Environment

python:3.9.17
bentoml:1.0.22
openllm:0.1.8

[aarnphm]

This is strange. Did you run into any issue when running containerize?

[IbrahimMCode]

Here is the Docker Build ouput
`sudo openllm build flan-t5 --model-id google/flan-t5-xl
Packing 'flan-t5' into a Bento with kwargs={}...
Building Bento for LLM 'flan-t5'
Building Bento for LLM 'flan-t5'

██████╗ ██████╗ ███████╗███╗ ██╗██╗ ██╗ ███╗ ███╗
██╔═══██╗██╔══██╗██╔════╝████╗ ██║██║ ██║ ████╗ ████║
██║ ██║██████╔╝█████╗ ██╔██╗ ██║██║ ██║ ██╔████╔██║
██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║██║ ██║ ██║╚██╔╝██║
╚██████╔╝██║ ███████╗██║ ╚████║███████╗███████╗██║ ╚═╝ ██║
╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝╚══════╝╚══════╝╚═╝ ╚═╝

Successfully built Bento(tag="google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce").

Possible next steps:

• Push to BentoCloud with bentoml push:
  $ bentoml push google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce
• Containerize your Bento with bentoml containerize:
  $ bentoml containerize google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce
  Tip: To enable additional BentoML feature for 'containerize', use '--enable-features=FEATURE[,FEATURE]' [see 'bentoml containerize -h' for more advanced usage]

sudo bentoml containerize google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce
Building OCI-compliant image for google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce with docker

[+] Building 300.4s (19/19) FINISHED
=> [internal] load build definition from Dockerfile 0.5s
=> => transferring dockerfile: 3.36kB 0.0s
=> [internal] load .dockerignore 0.5s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/nvidia/cuda:11.6.2-cudnn8-runtime-ubuntu20.04 0.5s
=> [base-container 1/14] FROM docker.io/nvidia/cuda:11.6.2-cudnn8-runtime-ubuntu20.04@sha256:3e1c8a9754bb530b0 40.2s
=> => resolve docker.io/nvidia/cuda:11.6.2-cudnn8-runtime-ubuntu20.04@sha256:3e1c8a9754bb530b01c21be43f65fcc4a74 0.0s
=> => sha256:3e1c8a9754bb530b01c21be43f65fcc4a74c9bec42f61183112e936dda5cbe43 743B / 743B 0.0s
=> => sha256:462ba0e2badc619e410d3a10f0471056553b71a9c48187920d5ec4b92031d713 2.42kB / 2.42kB 0.0s
=> => sha256:2e33cbfb80bf58ced106d63a60762db7d436deb6c2cc793c08cca42aa1f4f017 12.77kB / 12.77kB 0.0s
=> => sha256:ca1778b6935686ad781c27472c4668fc61ec3aeb85494f72deb1921892b9d39e 27.50MB / 27.50MB 0.4s
=> => sha256:65966f22ed3d88416d069fddf13902e152ae54b6c9a04e2787938f831014f9af 7.94MB / 7.94MB 0.5s
=> => sha256:ad6b7dd7c65f1dabb00d95a454994d3664d03efbd0d64ae34fef79a3ec25893a 23.61MB / 23.61MB 0.6s
=> => extracting sha256:ca1778b6935686ad781c27472c4668fc61ec3aeb85494f72deb1921892b9d39e 0.9s
=> => sha256:037a5c0ebcaab7c1b8cfc3c82745d94c7d394e874f8629a33609c9b46360d75f 184B / 184B 0.8s
=> => sha256:7921c0ffa124d1669cc033a479b9ef5a4817f57cb1a089caca43a4ef8b5ab6a1 1.12GB / 1.12GB 11.0s
=> => sha256:ea504443a0386bfe9f1f778e0318641268d763b0415beace78b81b49cee43262 6.89kB / 6.89kB 1.0s
=> => sha256:ca95f35017dce66c17b318dd49cf041bfe6b4705e84014031c4aa5058fa14d57 62.68kB / 62.68kB 0.9s
=> => sha256:651795e98cbbf0ffe858eb5000428947c028af902bb9ef0410a7da9b9488dc69 1.68kB / 1.68kB 1.0s
=> => sha256:b753b50441b1c0c7c235e859ba57459a755d421b828aaecd09b15f77f1c6701d 1.52kB / 1.52kB 1.3s
=> => sha256:6703542d8e6b9b8e8b19e4f5079960adab08b0a7026fa541bd72a11b51b8d588 727.16MB / 727.16MB 9.2s
=> => extracting sha256:65966f22ed3d88416d069fddf13902e152ae54b6c9a04e2787938f831014f9af 0.3s
=> => extracting sha256:ad6b7dd7c65f1dabb00d95a454994d3664d03efbd0d64ae34fef79a3ec25893a 0.6s
=> => extracting sha256:037a5c0ebcaab7c1b8cfc3c82745d94c7d394e874f8629a33609c9b46360d75f 0.0s
=> => extracting sha256:ea504443a0386bfe9f1f778e0318641268d763b0415beace78b81b49cee43262 0.0s
=> => extracting sha256:7921c0ffa124d1669cc033a479b9ef5a4817f57cb1a089caca43a4ef8b5ab6a1 16.3s
=> => extracting sha256:ca95f35017dce66c17b318dd49cf041bfe6b4705e84014031c4aa5058fa14d57 0.0s
=> => extracting sha256:651795e98cbbf0ffe858eb5000428947c028af902bb9ef0410a7da9b9488dc69 0.0s
=> => extracting sha256:b753b50441b1c0c7c235e859ba57459a755d421b828aaecd09b15f77f1c6701d 0.0s
=> => extracting sha256:6703542d8e6b9b8e8b19e4f5079960adab08b0a7026fa541bd72a11b51b8d588 10.7s
=> [internal] load build context 66.7s
=> => transferring context: 11.40GB 66.7s
=> [base-container 2/14] RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Pa 3.6s
=> [base-container 3/14] RUN --mount=type=cache,target=/var/lib/apt --mount=type=cache,target=/var/cache/apt s 26.5s
=> [base-container 4/14] RUN --mount=type=cache,target=/var/lib/apt --mount=type=cache,target=/var/cache/apt 55.7s
=> [base-container 5/14] RUN ln -sf /usr/bin/python3.9 /usr/bin/python3 && ln -sf /usr/bin/pip3.9 /usr/bin/ 0.4s
=> [base-container 6/14] RUN curl -O https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && rm - 8.9s
=> [base-container 7/14] RUN groupadd -g 1034 -o bentoml && useradd -m -u 1034 -g 1034 -o -r bentoml 0.5s
=> [base-container 8/14] RUN mkdir /home/bentoml/bento && chown bentoml:bentoml /home/bentoml/bento -R 0.3s
=> [base-container 9/14] WORKDIR /home/bentoml/bento 0.0s
=> [base-container 10/14] COPY --chown=bentoml:bentoml ./env/python ./env/python/ 0.0s
=> [base-container 11/14] RUN --mount=type=cache,target=/root/.cache/pip bash -euxo pipefail /home/bentoml/bent 84.3s
=> [base-container 12/14] COPY --chown=bentoml:bentoml . ./ 31.4s
=> [base-container 13/14] RUN rm -rf /var/lib/{apt,cache,log} 0.3s
=> [base-container 14/14] RUN chmod +x /home/bentoml/bento/env/docker/entrypoint.sh 0.4s
=> exporting to image 46.7s
=> => exporting layers 46.7s
=> => writing image sha256:64188b623d999137c14601d043c4f5fdd5d66416d5799070e9f8c4745b7c4e7a 0.0s
=> => naming to docker.io/library/google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce 0.0s
Successfully built Bento container for "google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce" with tag(s) "google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce"
To run your newly built Bento container, run:
docker run -it --rm -p 3000:3000 google-flan-t5-xl-service:53fd1e22aa944eee1fd336f9aee8a437e01676ce serve`

[aarnphm]

You shouldn't run openllm with sudo first and foremost, try again without sudo

[IbrahimMCode]

It’s not related to sudo.
Having same error without sudo

[aarnphm]

This seems like the weirdest bug to me. orjson is a core dependency and should already include during install openllm in the container?

Would you mind running bentoml containerize ... --opt progress=plain and dump all of the output to a text file and send it here? Thanks.

[aarnphm]

I discover a bug on main and I will release a new version patch version promptly

[aarnphm]

can you try with 0.1.9 to see if you still run into this issue?

[IbrahimMCode]

Still the same issue

[aarnphm]

you need to provide bentoml containerize --opt no-cache to remove the previous cache layer. It will still use the previous cache hit afaik

[IbrahimMCode]

Still the same issue.
We installed everything from scratch on another vm as well, doesn’t work

[aarnphm]

Hey there, I have identified the issue and will have a patch release soon. sorry for the trouble

[aarnphm]

v0.1.10 will address this issue. Thanks for spotting this!

[aarnphm]

Did you still run into this issue?

[MartynasZilinskas]

I was having the same error. Upgraded to latest version (openllm, version 0.1.17). Build step with --overwrite and containerize step with --opt no-cache. It was solved.