[Issue]: Ideogram 4 second image with prompt enhancer failed

[liutyi]

Issue Description

1st image generation after SD.Next restart with Ideogram 4 PE enabled (default settings), offloading is None

Looks like PE done on GPU, but TE on CPU

than Image generation is on GPU again

results - ok

trying second prompt "home". i looks like both PE and TE done on CPU this time

PE successful, but image generation fail

Version Platform Description

latest dev

Relevant log output

11:12:18-984500 TRACE    Server: alive=True requests=359 memory=77.11/125.32
                         status='running' task='Inference'
                         timestamp='20260611111155'
                         current='task(96pvrumdglbnxky)' id='5e18c6cc7bbd436'
                         job=0 jobs=1 total=20 step=0 steps=0 queued=0
                         uptime=311 elapsed=23.04 eta=None progress=0
11:14:01-185785 DEBUG    Encode: prompt="['{"high_level_description":"A cozy
                         home interior, a warm ambient photograph of a lived-in
                         living room with natural light from a window and soft
                         furnishings.","compositional_deconstruction":{"backgrou
                         nd":"Wide windowsill framing a view of a soft blue sky
                         with light clouds, warm afternoon light flooding the
                         room, wooden floors with a subtle grain, beige walls
                         and a slightly cluttered ambiance from everyday
                         life.","elements":[{"type":"obj","desc":"A dark brown
                         leather armchair with a faded floral cushion resting on
                         a light oak rug, positioned near the window, angled to
                         face the center of the room, a small side table beside
                         it holding a brass lamp with a cream
                         shade."},{"type":"obj","desc":"A faded wool throw
                         blanket draped over the back of the armchair, with
                         visible frayed edges and a subtle checkered pattern, a
                         vintage stack of books on the side table with a small
                         potted fern beside them."},{"type":"obj","desc":"A
                         wooden dining table in the background with a set of
                         four mismatched ceramic mugs and a half-eaten slice of
                         bread, a window frame above the table showing the sky
                         and trees beyond."}]}}']" hijack=True
11:14:19-015707 TRACE    Server: alive=True requests=499 memory=77.3/125.32
                         status='running' task='TE Encode'
                         timestamp='20260611111401'
                         current='task(96pvrumdglbnxky)' id='03a3baff3107421'
                         job=0 jobs=1 total=21 step=0 steps=0 queued=0
                         uptime=432 elapsed=17.83 eta=None progress=0
Progress ?it/s                                              0% 0/12 00:00 ? Base
11:14:41-713008 ERROR    Processing: step=base args={'prompt': ['home'],
                         'generator':
                         [<modules.intel.ipex.hijacks.torch_Generator object at
                         0x7cbc11efd5d0>], 'callback_on_step_end': <function
                         diffusers_callback at 0x7cbcbd858400>,
                         'callback_on_step_end_tensor_inputs': ['latents'],
                         'num_inference_steps': 12, 'output_type': 'pil',
                         'width': 256, 'height': 256, 'prompt_upsampling': True}
                         Expected all tensors to be on the same device, but got
                         mat1 is on cpu, different from other tensors on xpu:0
                         (when checking argument in method wrapper_XPU_addmm)
11:14:41-714762 ERROR    Processing: RuntimeError
/app/modules/processing_diffusers.py:187 in process_base

  186             taskid = shared.state.begin('Inference')
❱ 187             output = shared.sd_model(**base_args)
  188             shared.state.end(taskid)

/mnt/python/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py:124 in decorate_context

  123         with ctx_factory():
❱ 124             return func(*args, **kwargs)
  125

/app/pipelines/ideogram/ideogram4.py:694 in __call__

  693                 pos_z = torch.cat([text_z_padding, latents], dim=1).to(self.transformer.dtype)
❱ 694                 pos_out = self.transformer(
  695                     hidden_states=pos_z,

/mnt/python/venv/lib/python3.12/site-packages/torch/nn/modules/module.py:1779 in _wrapped_call_impl

  1778         else:
❱ 1779             return self._call_impl(*args, **kwargs)
  1780

/mnt/python/venv/lib/python3.12/site-packages/torch/nn/modules/module.py:1790 in _call_impl

  1789                 or _global_forward_hooks or _global_forward_pre_hooks):
❱ 1790             return forward_call(*args, **kwargs)
  1791

/mnt/python/venv/lib/python3.12/site-packages/diffusers/models/transformers/transformer_ideogram4.py:411 in forward

  410         hidden_states = hidden_states * output_image_mask
❱ 411         hidden_states = self.input_proj(hidden_states) * output_image_mask
  412

/mnt/python/venv/lib/python3.12/site-packages/torch/nn/modules/module.py:1779 in _wrapped_call_impl

  1778         else:
❱ 1779             return self._call_impl(*args, **kwargs)
  1780

/mnt/python/venv/lib/python3.12/site-packages/torch/nn/modules/module.py:1790 in _call_impl

  1789                 or _global_forward_hooks or _global_forward_pre_hooks):
❱ 1790             return forward_call(*args, **kwargs)
  1791

/mnt/python/venv/lib/python3.12/site-packages/torch/nn/modules/linear.py:134 in forward

  133         """
❱ 134         return F.linear(input, self.weight, self.bias)
  135
RuntimeError: Expected all tensors to be on the same device, but got mat1 is on cpu, different from other tensors on xpu:0 (when checking argument in method wrapper_XPU_addmm)
11:14:42-209742 DEBUG    Search model:
                         name='Diffusers/CalamitousFelicitousness/Ideogram-4-bf1
                         6-Diffusers [1b97eeb07a]'
                         matched='/mnt/models/Diffusers/models--CalamitousFelici
                         tousness--Ideogram-4-bf16-Diffusers/snapshots/1b97eeb07
                         a23f6cd489de9dcaace08244f4291ec' type=alias
11:14:42-218112 DEBUG    Analyzed:
                         model='Diffusers/CalamitousFelicitousness/Ideogram-4-bf
                         16-Diffusers' type=ideogram4 class=Ideogram4Pipeline
                         size=0 mtime='2026-06-06 00:30:17'
                         modules=[name='scheduler'
                         cls=FlowMatchEulerDiscreteScheduler config=True,
                         name='vae' cls=AutoencoderKLFlux2 config=True
                         device=xpu:0 dtype=torch.bfloat16 params=84046115
                         modules=244, name='text_encoder' cls=Qwen3VLModel
                         config=True device=cpu dtype=torch.bfloat16
                         params=8144793840 modules=843, name='tokenizer'
                         cls=Qwen2Tokenizer config=False, name='transformer'
                         cls=Ideogram4Transformer2DModel config=True
                         device=xpu:0 dtype=torch.bfloat16 params=9279288448
                         modules=661, name='unconditional_transformer'
                         cls=Ideogram4Transformer2DModel config=True
                         device=xpu:0 dtype=torch.bfloat16 params=9279288448
                         modules=661, name='prompt_enhancer_head'
                         cls=Ideogram4PromptEnhancerHead config=True device=cpu
                         dtype=torch.bfloat16 params=622329856 modules=2]
11:14:42-219857 INFO     Processing modifiers: unapply
11:14:42-221846 DEBUG    Process: batch=1/1 interrupted
11:14:42-222409 DEBUG    History: len=3 add=Item(ts='2026-06-11 11:14:42'
                         ops=['txt2img'] images=0)

Operating system

Linux

Compute

Intel IPEX

Interface

Standard

Branch

Dev

Model

Unspecified

Acknowledgements

• I have read the above and searched for existing issues
• I confirm that this is classified correctly and its not an extension issue

[vladmandic]

prompt-enhancer is a weird beast, its a 1gb head + reuse of already loaded text-encoder
so its a virtually-assembled-model and that causes all kinds of offloading issues with transformers
i have a really nasty monkey-patch in place that solves that for cuda, but as you can see it doesn't solve it completely for xpu.
but given i dont have xpu environment where i can even attempt to run ideogram, not sure what i can do.

[liutyi]

not sure if this info will help but temporary workaround that is work for me is

diff --git a/pipelines/ideogram/ideogram4.py b/pipelines/ideogram/ideogram4.py
index 2efb45646..7721ef484 100644
--- a/pipelines/ideogram/ideogram4.py
+++ b/pipelines/ideogram/ideogram4.py
@@ -257,7 +257,7 @@ class Ideogram4Pipeline(DiffusionPipeline):
             )
         finally:
             restore_qwen3vl()
-            self.prompt_enhancer.to('cpu')
+            self.prompt_enhancer.to('xpu')
         debug_prompt(f'Prompt: input="{prompt}"')
         debug_prompt(f'Prompt: enhanced="{caption}"')
         return caption
@@ -421,7 +421,7 @@ class Ideogram4Pipeline(DiffusionPipeline):
             prompt = self._encode_prompt(prompt, grid_h, grid_w, max_sequence_length, device)
         finally:
             if shared.opts.diffusers_offload_mode != 'none':
-                self.text_encoder.to(devices.cpu)
+                self.text_encoder.to(devices.xpu)
         return prompt

     def prepare_latents(

[vladmandic]

that effectively disables offloading and leaves prompt-enhancer on gpu even after its complete, so that's definitely not a solution.
issue is that earlier line

    self.prompt_enhancer.to(device)

does not move it to gpu completely in case of a virtually assembled model.
hijack_qwen3vl is what is explicitly written to deal with that and it works for cuda, but appearently not enough for xpu.