Official arm64 build does not work on Jetson Nano Orin

[gab0220]

What is the issue?

Hello everyone, thank you for your work.

I'm using a Jetson Nano Orin. Following #3098, some days ago I done a git checkout using #2279 commit and install this version on my device. It works.

Today I tried to:

• Install the v0.1.30 using this tutorial
• Clean ollama list
• Run ollama pull <model>
• Run OLLAMA_DEBUG="1" ollama run <model>
  Output:

Error: Post "http://127.0.0.1:11434/api/chat": EOF

I also attach the output of journalctl -u ollama:

Mar 29 11:16:09 ubuntu ollama[4168]: time=2024-03-29T11:16:09.687+01:00 level=INFO source=gpu.go:115 msg="Detecting GPU type"
Mar 29 11:16:09 ubuntu ollama[4168]: time=2024-03-29T11:16:09.687+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library [libcudart.so](https://libcudart.so/)*"
Mar 29 11:16:09 ubuntu ollama[4168]: time=2024-03-29T11:16:09.692+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/tmp/ollama3349183846/runners/cuda_v11/[libcudart.so](https://libcudart.so/).11.0 /usr/local/cuda/lib64/libcudart.so.12.2.140 /usr/local/cuda/targets/aarch64-linux/lib/[libcudart.so](https://libcudart.so/).12.2.140 /usr/local/cuda-12/targets/aarch64-linux/lib/[libcudart.so](https://libcudart.so/).12.2.140 /usr/local/cuda-12.2/targets/aarch64-linux/lib/[libcudart.so](https://libcudart.so/).12.2.140]"
Mar 29 11:16:09 ubuntu ollama[4168]: time=2024-03-29T11:16:09.714+01:00 level=INFO source=gpu.go:120 msg="Nvidia GPU detected via cudart"
Mar 29 11:16:09 ubuntu ollama[4168]: time=2024-03-29T11:16:09.714+01:00 level=INFO source=cpu_common.go:18 msg="CPU does not have vector extensions"
Mar 29 11:16:09 ubuntu ollama[4168]: time=2024-03-29T11:16:09.801+01:00 level=INFO source=gpu.go:188 msg="[cudart] CUDART CUDA Compute Capability detected: 8.7"
Mar 29 11:16:17 ubuntu systemd[1]: Stopping Ollama Service...
Mar 29 11:16:17 ubuntu systemd[1]: ollama.service: Deactivated successfully.
Mar 29 11:16:17 ubuntu systemd[1]: Stopped Ollama Service.
Mar 29 11:16:17 ubuntu systemd[1]: ollama.service: Consumed 9.601s CPU time.

What did you expect to see?

So the I can't use model.

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

Architecture

Other

Platform

No response

Ollama version

v0.1.30

GPU

Nvidia

GPU info

No response

CPU

No response

Other software

No response

[remy415]

@gab0220 thank you for reporting this. The issue right now is the OS Jetsons run on aren’t able to use the CUDA libraries bundled by the process they use to compile the binary. We’re still trying to pinpoint the exact issue to see if there’s a way to continue using the same process with minor adjustments.

You should be able to quickly build the binary on your Jetson, note that it is no longer necessary to follow the referenced tutorial, though it should still work if you compile yourself.

First, set up environment variables

export GOLANG_VERSION=1.21.3
export GO_ARCH=arm64
export CMAKE_VERSION=3.22.1
export LD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/lib64/:/usr/local/cuda/include
export OLLAMA_SKIP_CPU_GENERATE="1"
export CGO_ENABLED="1"
export CMAKE_CUDA_ARCHITECTURES="72;87"

Ensure required tools are installed

sudo apt update && sudo apt install -y build-essentials
curl -s -L https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-$(uname -m).tar.gz | tar -zx -C /usr --strip-components 1
rm /usr/local/bin/cmake && update-alternatives --install /usr/local/bin/cmake cmake /usr/bin/cmake 30
curl -s -L https://dl.google.com/go/go${GOLANG_VERSION}.linux-${GO_ARCH}.tar.gz | tar xz -C /usr/local
ln -s /usr/local/go/bin/go /usr/local/bin/go
ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt

Clone repo and build. Ensure you first cd <project folder>

git clone https://github.com/ollama/ollama.git && cd ollama
go clean
go generate ./… && go build .

This will compile the Ollama binary for your Jetson and save it to your current directory. Remove the old Ollama binarysudo rm /usr/local/bin/ollama then copy the new one withsudo cp ollama /usr/local/bin/ollama. You can then restart your Ollama service.

[dhiltgen]

I've adjusted the behavior of the system with the upcoming 0.1.32 release so that we'll load the cuda library from the LD_LIBRARY_PATH before our bundled version, which should help mitigate this. As long as you include the cuda lib dir in your LD_LIBRARY_PATH for the ollama server, it should work. Ultimately I'd still like to get an older glibc based build setup defined that has a cuda library that works on Jetson, so I'll keep this issue open for now.

[CesarCalvoCobo]

Hi , thanks again all for your work

I am trying to compile new version and getting always the same error :
/usr/local/go/pkg/tool/linux_arm64/link: running gcc failed: exit status 1
/usr/bin/ld: cannot find ollama/llm/build/linux/arm64_static/libllama.a: No such file or directory

Also trying to install the bundled version directly including LD_LIBRARY_PATH and it runs but it does not load the models

[remy415]

@CesarCalvoCobo are you setting OLLAMA_SKIP_CPU_GENERATE=1? If so, you should set it to OLLAMA_SKIP_CPU_GENERATE="". I've submitted a PR to fix this but in the mean time, you need to compile the CPU and ensure you also don't set OLLAMA_CPU_TARGET

[remy415]

@CesarCalvoCobo Okay my PR got merged so you should be able to just pull the latest ollama repo and run the compile again

[CesarCalvoCobo]

Thank you so much @remy415 - I compiled it succesfully now

[remy415]

@dhiltgen yea everything is working well as of a couple weeks ago

[dhiltgen]

Sounds like we can close this as resolved. Please speak up if you have any lingering issues on Jetsons.