Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ollama ROCm Docker container crashing on RX 6650 #2870

Closed
Zambito1 opened this issue Mar 2, 2024 · 2 comments
Closed

Ollama ROCm Docker container crashing on RX 6650 #2870

Zambito1 opened this issue Mar 2, 2024 · 2 comments

Comments

@Zambito1
Copy link

Zambito1 commented Mar 2, 2024 •
edited

I am running GNU Guix with the following ROCm packages installed:

$ guix package -I rocm
rocm-device-libs   	5.6.0	out	/gnu/store/jrkc3924g178yfvqlwqzq9d3pmxc9jlg-rocm-device-libs-5.6.0
rocm-opencl-runtime	5.6.0	out	/gnu/store/wnq6v9cqfsbrm7w0y3c610vifbdbdb8x-rocm-opencl-runtime-5.6.0
rocm-comgr         	5.6.0	out	/gnu/store/p5cnvrb78d6pmxh1fzh5wz1pw18rcism-rocm-comgr-5.6.0
rocminfo           	5.6.0	out	/gnu/store/bbgzkrszjqvwz5d8b3qimkzjzk9h1r7r-rocminfo-5.6.0

My upstream Guix commit is 8c0282c. Here is the output of rocminfo:

�[37mROCk module is loaded�[0m
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 7 3700X 8-Core Processor 
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 7 3700X 8-Core Processor 
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   4050                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            16                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    32790324(0x1f45734) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    32790324(0x1f45734) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    32790324(0x1f45734) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx1032                            
  Uuid:                    GPU-XX                             
  Marketing Name:          AMD Radeon RX 6650 XT              
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      2048(0x800) KB                     
    L3:                      32768(0x8000) KB                   
  Chip ID:                 29679(0x73ef)                      
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   2765                               
  BDFID:                   2816                               
  Internal Node ID:        1                                  
  Compute Unit:            32                                 
  SIMDs per CU:            2                                  
  Shader Engines:          2                                  
  Shader Arrs. per Eng.:   2                                  
  WatchPts on Addr. Ranges:4                                  
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          32(0x20)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        32(0x20)                           
  Max Work-item Per CU:    1024(0x400)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    8372224(0x7fc000) KB               
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx1032         
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***

Here I am running rocm-smi inside a Docker container:

$ docker run --rm -it --device /dev/kfd --device /dev/dri rocm/rocm-terminal
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

rocm-user@fcd0a80ee8e7:~$ rocm-smi


===================================== ROCm System Management Interface =====================================
=============================================== Concise Info ===============================================
Device  [Model : Revision]    Temp    Power  Partitions      SCLK    MCLK   Fan  Perf  PwrCap  VRAM%  GPU%  
        Name (20 chars)       (Edge)  (Avg)  (Mem, Compute)                                                 
============================================================================================================
0       [0x5026 : 0xc1]       40.0°C  5.0W   N/A, N/A        700Mhz  96Mhz  0%   auto  164.0W   10%   5%    
        0x73ef                                                                                              
============================================================================================================
=========================================== End of ROCm SMI Log ============================================

So I am fairly sure I have ROCm setup correctly with Docker. When I run the following, I get a crash:

$ docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:0.1.27-rocm 
c466d9c7494992fd813b1b38337715247a323a5d15a7cd53a1cf987e173df7ba
$ docker exec -it ollama ollama run gemma:2b

I have tried with several different models, gemma:2b being the smallest, to see if size was the issue. This is the log from the container when I crash:

time=2024-03-02T00:34:13.785Z level=INFO source=images.go:710 msg="total blobs: 16"
time=2024-03-02T00:34:13.785Z level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-03-02T00:34:13.786Z level=INFO source=routes.go:1019 msg="Listening on [::]:11434 (version 0.1.27)"
time=2024-03-02T00:34:13.786Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-03-02T00:34:16.164Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v6 cuda_v11 cpu cpu_avx rocm_v5 cpu_avx2]"
time=2024-03-02T00:34:16.164Z level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-03-02T00:34:16.164Z level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-03-02T00:34:16.166Z level=INFO source=gpu.go:311 msg="Discovered GPU libraries: []"
time=2024-03-02T00:34:16.166Z level=INFO source=gpu.go:265 msg="Searching for GPU management library librocm_smi64.so"
time=2024-03-02T00:34:16.166Z level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/opt/rocm/lib/librocm_smi64.so.5.0.50701 /opt/rocm-5.7.1/lib/librocm_smi64.so.5.0.50701]"
time=2024-03-02T00:34:16.168Z level=INFO source=gpu.go:109 msg="Radeon GPU detected"
time=2024-03-02T00:34:16.168Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[GIN] 2024/03/02 - 00:34:23 | 200 |      35.141µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/03/02 - 00:34:23 | 200 |     414.706µs |       127.0.0.1 | POST     "/api/show"
[GIN] 2024/03/02 - 00:34:23 | 200 |     286.014µs |       127.0.0.1 | POST     "/api/show"
time=2024-03-02T00:34:24.590Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-02T00:34:24.590Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-02T00:34:24.590Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-02T00:34:24.628Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama2381532766/rocm_v5/libext_server.so"
time=2024-03-02T00:34:24.628Z level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"

rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx1032
 List of available TensileLibrary Files : 
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx941.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx940.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx906.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx803.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx908.dat"
loading library /tmp/ollama2381532766/rocm_v5/libext_server.so
SIGSEGV: segmentation violation
PC=0x7f7e54233bc7 m=12 sigcode=128
signal arrived during cgo execution

goroutine 21 [syscall]:
runtime.cgocall(0x9bcdd0, 0xc0000d66c8)
	/usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0000d66a0 sp=0xc0000d6668 pc=0x409b0b
github.com/jmorganca/ollama/llm._Cfunc_dyn_llama_server_init({0x7f7de0000e20, 0x7f7dea064bf0, 0x7f7dea065410, 0x7f7dea0654a0, 0x7f7dea065710, 0x7f7dea065950, 0x7f7dea0662b0, 0x7f7dea066290, 0x7f7dea0663c0, 0x7f7dea0669d0, ...}, ...)
	_cgo_gotypes.go:282 +0x45 fp=0xc0000d66c8 sp=0xc0000d66a0 pc=0x7c5485
github.com/jmorganca/ollama/llm.newDynExtServer.func7(0xaf20c4?, 0xc?)
	/go/src/github.com/jmorganca/ollama/llm/dyn_ext_server.go:153 +0xef fp=0xc0000d67b8 sp=0xc0000d66c8 pc=0x7c69cf
github.com/jmorganca/ollama/llm.newDynExtServer({0xc000616000, 0x2e}, {0xc00007a690, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...)
	/go/src/github.com/jmorganca/ollama/llm/dyn_ext_server.go:153 +0xa65 fp=0xc0000d6a58 sp=0xc0000d67b8 pc=0x7c6665
github.com/jmorganca/ollama/llm.newLlmServer({{_, _, _}, {_, _}, {_, _}}, {_, _}, {0xc00007a690, ...}, ...)
	/go/src/github.com/jmorganca/ollama/llm/llm.go:158 +0x425 fp=0xc0000d6c18 sp=0xc0000d6a58 pc=0x7c2dc5
github.com/jmorganca/ollama/llm.New({0xc00002a108, 0x15}, {0xc00007a690, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...)
	/go/src/github.com/jmorganca/ollama/llm/llm.go:123 +0x713 fp=0xc0000d6e98 sp=0xc0000d6c18 pc=0x7c2733
github.com/jmorganca/ollama/server.load(0xc000176d80?, 0xc000176d80, {{0x0, 0x800, 0x200, 0x1, 0xffffffffffffffff, 0x0, 0x0, 0x1, ...}, ...}, ...)
	/go/src/github.com/jmorganca/ollama/server/routes.go:85 +0x3a5 fp=0xc0000d7018 sp=0xc0000d6e98 pc=0x996945
github.com/jmorganca/ollama/server.ChatHandler(0xc00022e100)
	/go/src/github.com/jmorganca/ollama/server/routes.go:1173 +0xa37 fp=0xc0000d7748 sp=0xc0000d7018 pc=0x9a1f77
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc00022e100)
	/go/src/github.com/jmorganca/ollama/server/routes.go:943 +0x68 fp=0xc0000d7780 sp=0xc0000d7748 pc=0x9a07a8
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc00022e100)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x7a fp=0xc0000d77d0 sp=0xc0000d7780 pc=0x97803a
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc00022e100)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xde fp=0xc0000d7980 sp=0xc0000d77d0 pc=0x9771de
github.com/gin-gonic/gin.(*Context).Next(...)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0005924e0, 0xc00022e100)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x65b fp=0xc0000d7b08 sp=0xc0000d7980 pc=0x97629b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0005924e0, {0x11403a20?, 0xc00007c0e0}, 0xc00022e300)
	/root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc0000d7b48 sp=0xc0000d7b08 pc=0x975a5d
net/http.serverHandler.ServeHTTP({0x11401d40?}, {0x11403a20?, 0xc00007c0e0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc0000d7b78 sp=0xc0000d7b48 pc=0x6ced4e
net/http.(*conn).serve(0xc000174240, {0x11405088, 0xc0003eb380})
	/usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc0000d7fb8 sp=0xc0000d7b78 pc=0x6cac34
net/http.(*Server).Serve.func3()
	/usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc0000d7fe0 sp=0xc0000d7fb8 pc=0x6cf568
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000d7fe8 sp=0xc0000d7fe0 pc=0x46e2c1
created by net/http.(*Server).Serve in goroutine 1
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

goroutine 1 [IO wait]:
runtime.gopark(0x480f10?, 0xc0005b5850?, 0xa0?, 0x58?, 0x4f711d?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005b5830 sp=0xc0005b5810 pc=0x43e7ee
runtime.netpollblock(0x46c332?, 0x4092a6?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0005b5868 sp=0xc0005b5830 pc=0x437277
internal/poll.runtime_pollWait(0x7f7e53e99e28, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0005b5888 sp=0xc0005b5868 pc=0x468a05
internal/poll.(*pollDesc).wait(0xc000482000?, 0x4?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0005b58b0 sp=0xc0005b5888 pc=0x4efd67
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000482000)
	/usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0005b5958 sp=0xc0005b58b0 pc=0x4f524c
net.(*netFD).accept(0xc000482000)
	/usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0005b5a10 sp=0xc0005b5958 pc=0x56be29
net.(*TCPListener).accept(0xc000453560)
	/usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0005b5a38 sp=0xc0005b5a10 pc=0x580c3e
net.(*TCPListener).Accept(0xc000453560)
	/usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0005b5a68 sp=0xc0005b5a38 pc=0x57fdf0
net/http.(*onceCloseListener).Accept(0xc000174240?)
	<autogenerated>:1 +0x24 fp=0xc0005b5a80 sp=0xc0005b5a68 pc=0x6f1ae4
net/http.(*Server).Serve(0xc00007e000, {0x11403810, 0xc000453560})
	/usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc0005b5bb0 sp=0xc0005b5a80 pc=0x6cf1a4
github.com/jmorganca/ollama/server.Serve({0x11403810, 0xc000453560})
	/go/src/github.com/jmorganca/ollama/server/routes.go:1046 +0x454 fp=0xc0005b5c98 sp=0xc0005b5bb0 pc=0x9a0c54
github.com/jmorganca/ollama/cmd.RunServer(0xc000480300?, {0x1184c8c0?, 0x4?, 0xad9d6a?})
	/go/src/github.com/jmorganca/ollama/cmd/cmd.go:706 +0x1b9 fp=0xc0005b5d30 sp=0xc0005b5c98 pc=0x9b3d99
github.com/spf13/cobra.(*Command).execute(0xc00043f500, {0x1184c8c0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c fp=0xc0005b5e68 sp=0xc0005b5d30 pc=0x764d9c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00043e900)
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc0005b5f20 sp=0xc0005b5e68 pc=0x7655c5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
	/go/src/github.com/jmorganca/ollama/main.go:11 +0x4d fp=0xc0005b5f40 sp=0xc0005b5f20 pc=0x9bbeed
runtime.main()
	/usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0005b5fe0 sp=0xc0005b5f40 pc=0x43e39b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005b5fe8 sp=0xc0005b5fe0 pc=0x46e2c1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006afa8 sp=0xc00006af88 pc=0x43e7ee
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc00006afe0 sp=0xc00006afa8 pc=0x43e673
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x46e2c1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000066778 sp=0xc000066758 pc=0x43e7ee
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000667c8 sp=0xc000066778 pc=0x42a73f
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000667e0 sp=0xc0000667c8 pc=0x41f865
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x46e2c1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 19 [GC scavenge wait]:
runtime.gopark(0x8b6c3c?, 0x878546?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000066f70 sp=0xc000066f50 pc=0x43e7ee
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1181cc40)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000066fa0 sp=0xc000066f70 pc=0x427f69
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000066fc8 sp=0xc000066fa0 pc=0x428519
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000066fe0 sp=0xc000066fc8 pc=0x41f805
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x46e2c1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 34 [finalizer wait]:
runtime.gopark(0xad2d20?, 0x10043f901?, 0x0?, 0x0?, 0x4469a5?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006a628 sp=0xc00006a608 pc=0x43e7ee
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00006a7e0 sp=0xc00006a628 pc=0x41e8e7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006a7e8 sp=0xc00006a7e0 pc=0x46e2c1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x33a6ea91a4e?, 0x1?, 0x46?, 0x50?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049c750 sp=0xc00049c730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049c7e0 sp=0xc00049c750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049c7e8 sp=0xc00049c7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 3 [GC worker (idle)]:
runtime.gopark(0x1184e5e0?, 0x1?, 0xc?, 0xfd?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006b750 sp=0xc00006b730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006b7e0 sp=0xc00006b750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006b7e8 sp=0xc00006b7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 4 [GC worker (idle)]:
runtime.gopark(0x33a6ea917ff?, 0x3?, 0x59?, 0x1e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006bf50 sp=0xc00006bf30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006bfe0 sp=0xc00006bf50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x33a6ea9197b?, 0x3?, 0x73?, 0xf?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049cf50 sp=0xc00049cf30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049cfe0 sp=0xc00049cf50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049cfe8 sp=0xc00049cfe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x33a6ea9174b?, 0x1?, 0xd3?, 0x50?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049d750 sp=0xc00049d730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049d7e0 sp=0xc00049d750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049d7e8 sp=0xc00049d7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x33a6ea9179b?, 0x3?, 0xc9?, 0xd1?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049df50 sp=0xc00049df30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049dfe0 sp=0xc00049df50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049dfe8 sp=0xc00049dfe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x1184e5e0?, 0x1?, 0xf2?, 0x7a?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049e750 sp=0xc00049e730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049e7e0 sp=0xc00049e750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049e7e8 sp=0xc00049e7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x33a6ea91827?, 0x3?, 0x1a?, 0xa2?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049ef50 sp=0xc00049ef30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049efe0 sp=0xc00049ef50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049efe8 sp=0xc00049efe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x33a6ea917d7?, 0x3?, 0xdc?, 0x5?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049f750 sp=0xc00049f730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049f7e0 sp=0xc00049f750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049f7e8 sp=0xc00049f7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 42 [GC worker (idle)]:
runtime.gopark(0x33a6ea91daa?, 0x3?, 0xe?, 0x4c?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049ff50 sp=0xc00049ff30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00049ffe0 sp=0xc00049ff50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049ffe8 sp=0xc00049ffe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 43 [GC worker (idle)]:
runtime.gopark(0x1184e5e0?, 0x3?, 0x94?, 0x2?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000498750 sp=0xc000498730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0004987e0 sp=0xc000498750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004987e8 sp=0xc0004987e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 44 [GC worker (idle)]:
runtime.gopark(0x33a6ea91093?, 0x3?, 0x78?, 0xf6?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000498f50 sp=0xc000498f30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000498fe0 sp=0xc000498f50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000498fe8 sp=0xc000498fe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 45 [GC worker (idle)]:
runtime.gopark(0x1184e5e0?, 0x1?, 0x0?, 0x6d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000499750 sp=0xc000499730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0004997e0 sp=0xc000499750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004997e8 sp=0xc0004997e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 46 [GC worker (idle)]:
runtime.gopark(0x1184e5e0?, 0x3?, 0x88?, 0x3b?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000499f50 sp=0xc000499f30 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000499fe0 sp=0xc000499f50 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000499fe8 sp=0xc000499fe0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x33a6ea9192b?, 0x3?, 0xea?, 0x1?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006c750 sp=0xc00006c730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006c7e0 sp=0xc00006c750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006c7e8 sp=0xc00006c7e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x33a6ea918be?, 0x3?, 0xb2?, 0x2a?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000067750 sp=0xc000067730 pc=0x43e7ee
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0000677e0 sp=0xc000067750 pc=0x4213e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000677e8 sp=0xc0000677e0 pc=0x46e2c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1217 +0x1c

goroutine 6 [select, locked to thread]:
runtime.gopark(0xc000069fa8?, 0x2?, 0x89?, 0xea?, 0xc000069fa4?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000069e38 sp=0xc000069e18 pc=0x43e7ee
runtime.selectgo(0xc000069fa8, 0xc000069fa0, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000069f58 sp=0xc000069e38 pc=0x44e325
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc000069fe0 sp=0xc000069f58 pc=0x46535f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x46e2c1
created by runtime.ensureSigM in goroutine 1
	/usr/local/go/src/runtime/signal_unix.go:997 +0xc8

goroutine 50 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
	/usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc0005a87a0 sp=0xc0005a8768 pc=0x411349
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0005a87c0 sp=0xc0005a87a0 pc=0x46ac89
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0005a87e0 sp=0xc0005a87c0 pc=0x6f4513
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005a87e8 sp=0xc0005a87e0 pc=0x46e2c1
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 51 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005a8f18 sp=0xc0005a8ef8 pc=0x43e7ee
runtime.chanrecv(0xc0000b4b40, 0x0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0005a8f90 sp=0xc0005a8f18 pc=0x40beed
runtime.chanrecv1(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0005a8fb8 sp=0xc0005a8f90 pc=0x40baf2
github.com/jmorganca/ollama/server.Serve.func2()
	/go/src/github.com/jmorganca/ollama/server/routes.go:1028 +0x25 fp=0xc0005a8fe0 sp=0xc0005a8fb8 pc=0x9a0ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005a8fe8 sp=0xc0005a8fe0 pc=0x46e2c1
created by github.com/jmorganca/ollama/server.Serve in goroutine 1
	/go/src/github.com/jmorganca/ollama/server/routes.go:1027 +0x3c7

goroutine 48 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xc?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005a75a0 sp=0xc0005a7580 pc=0x43e7ee
runtime.netpollblock(0x47f078?, 0x4092a6?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0005a75d8 sp=0xc0005a75a0 pc=0x437277
internal/poll.runtime_pollWait(0x7f7e53e99d30, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0005a75f8 sp=0xc0005a75d8 pc=0x468a05
internal/poll.(*pollDesc).wait(0xc000038a00?, 0xc0003eb661?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0005a7620 sp=0xc0005a75f8 pc=0x4efd67
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000038a00, {0xc0003eb661, 0x1, 0x1})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0005a76b8 sp=0xc0005a7620 pc=0x4f105a
net.(*netFD).Read(0xc000038a00, {0xc0003eb661?, 0xc0005a7740?, 0x46a990?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc0005a7700 sp=0xc0005a76b8 pc=0x569e05
net.(*conn).Read(0xc00059e178, {0xc0003eb661?, 0x1?, 0xc000160780?})
	/usr/local/go/src/net/net.go:179 +0x45 fp=0xc0005a7748 sp=0xc0005a7700 pc=0x5780a5
net.(*TCPConn).Read(0xc0003eb650?, {0xc0003eb661?, 0xc000160780?, 0x0?})
	<autogenerated>:1 +0x25 fp=0xc0005a7778 sp=0xc0005a7748 pc=0x589fa5
net/http.(*connReader).backgroundRead(0xc0003eb650)
	/usr/local/go/src/net/http/server.go:683 +0x37 fp=0xc0005a77c8 sp=0xc0005a7778 pc=0x6c4ab7
net/http.(*connReader).startBackgroundRead.func2()
	/usr/local/go/src/net/http/server.go:679 +0x25 fp=0xc0005a77e0 sp=0xc0005a77c8 pc=0x6c49e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005a77e8 sp=0xc0005a77e0 pc=0x46e2c1
created by net/http.(*connReader).startBackgroundRead in goroutine 21
	/usr/local/go/src/net/http/server.go:679 +0xba

rax    0x6
rbx    0x7f7e029fee60
rcx    0x7f7e54232387
rdx    0x6
rdi    0x1
rsi    0x11
rbp    0x0
rsp    0x7f7e029fed30
r8     0x0
r9     0x7f7e029fec80
r10    0x8
r11    0x202
r12    0x7f7de08d4460
r13    0x7f7de08d2f60
r14    0x7f7e029fef98
r15    0x7f7e029ff2d8
rip    0x7f7e54233bc7
rflags 0x10246
cs     0x33
fs     0x0
gs     0x0

Edit: and FWIW this error occured both when I tried to run the model from the ollama cli like I demonstrated, and when I tried to connect via open-webui, so it's not an issue specific to using the cli.
@Zambito1
Copy link
Author

Zambito1 commented Mar 2, 2024 •
edited

Actually give me one moment, I am doing to try installing one more thing to see if it fixes it.

Edit: Scratch that. I realized there is an amdgpu-firmware package in nonguix that I did not have installed before. I installed it now, but I am running into the same issue.

@Zambito1
Copy link
Author

Zambito1 commented Mar 4, 2024

Sick I was able to figure it out. I found this comment ROCm/Tensile#1936 which suggested setting the environment variable HSA_OVERRIDE_GFX_VERSION=10.3.0. That worked for me. The full command I ran to get it to work is: docker run -d -e HSA_OVERRIDE_GFX_VERSION=10.3.0 --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:0.1.27-rocm

@Zambito1 Zambito1 closed this as completed Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Zambito1