-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Description
LocalAI version:
v3.5.2 with docker image localai/localai:latest-gpu-nvidia-cuda-12
Environment, CPU architecture, OS, and Version:
Linux 6.11.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
RTX 5090
Describe the bug
Qwen-image generation does not work
To Reproduce
Install the Qwen-image model,
Install the diffusers cuda12 backend
Expected behavior
Logs
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stdout Initializing libbackend for cuda12-diffusers
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stdout Using portable Python
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr /backends/cuda12-diffusers/venv/lib/python3.10/site-packages/transformers/utils/hub.py:111: FutureWarning: UsingTRANSFORMERS_CACHE
is deprecated and will be removed in v5 of Transformers. UseHF_HOME
instead.
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr warnings.warn(
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr /backends/cuda12-diffusers/venv/lib/python3.10/site-packages/torch/cuda/init.py:235: UserWarning:
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr warnings.warn(
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Server started. Listening on: 127.0.0.1:34563
10:30AM DBG GRPC Service Ready
10:30AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:0xc00082f958} sizeCache:0 unknownFields:[] Model:Qwen/Qwen-Image ContextSize:1024 Seed:1840332066 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:true Embeddings:false NUMA:false NGPULayers:9999999 MainGPU: TensorSplit: Threads:8 RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/Qwen/Qwen-Image PipelineType:DiffusionPipeline SchedulerType: CUDA:true CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 LoadFormat: DisableLogStatus:false DType: LimitImagePerPrompt:0 LimitVideoPerPrompt:0 LimitAudioPerPrompt:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:auto NoKVOffload:false ModelPath://models LoraAdapters:[] LoraScales:[] Options:[true_cfg_scale:4.0 torch_dtype:bf16] CacheTypeKey: CacheTypeValue: GrammarTriggers:[] Reranking:false Overrides:[]}
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Loading model Qwen/Qwen-Image...
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Request Model: "Qwen/Qwen-Image"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr ContextSize: 1024
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Seed: 1840332066
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr NBatch: 512
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr F16Memory: true
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr MMap: true
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr LowVRAM: true
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr NGPULayers: 9999999
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Threads: 8
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr ModelFile: "/models/Qwen/Qwen-Image"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr PipelineType: "DiffusionPipeline"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr CUDA: true
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr FlashAttention: "auto"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr ModelPath: "//models"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Options: "true_cfg_scale:4.0"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr Options: "torch_dtype:bf16"
10:30AM DBG GRPC(qwen-image-127.0.0.1:34563): stderr
10:30AM ERR Failed to load model qwen-image with backend diffusers error="failed to load model with internal loader: could not load model (no success): Unexpected err=AttributeError("'str' object has no attribute 'is_integer'"), type(err)=<class 'AttributeError'>" modelID=qwen-image
10:30AM ERR Server error error="failed to load model with internal loader: could not load model (no success): Unexpected err=AttributeError("'str' object has no attribute 'is_integer'"), type(err)=<class 'AttributeError'>" ip=192.168.2.33 latency=6.041281123s method=POST status=500 url=/v1/images/generations
Additional context
I try to uninstall then reinstall the diffuser backend from the interface
I seem that there is a python error 'str' object has no attribute 'is_integer'