How to use GLIGEN textbox model?

[Darkflo264]

Hello everyone!

One of the last additions is the feature to use a GLIGEN textbox model. This seems really interesting, but I can’t find any infos about how to use it or integrate in a workflow.

Has someone already used this and can provide some Info about how to use it, or, even better, provide a example workflow?

Thank you and best regards

[comfyanonymous]

model files are here: https://huggingface.co/comfyanonymous/GLIGEN_pruned_safetensors/tree/main
put them in: models/gligen

Here's an example workflow:

The textbox gligen model is to control the generation by giving hints where to place objects/etc...
You write your prompt with everything.
Then you make some GLIGENTextBoxApply with where you want certain objects to be.

I'll add it to the examples page soon.

[Darkflo264]

That seems fantastic!

Thank you for your fast response!

i will give it a try this evening / as soon as I am home.

[SuperJin9]

today I start use gligen and Follow the prompts to prepare the required configuration.
But I have a problem and I don't know how to solve it. maybe I need help .thanks!!
cpu mode under windows10 system，The prompt content is as follows：

Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 320 and using 8 heads.
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 320 and using 8 heads.
0%| | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\execution.py", line 182, in execute
recursive_execute(self.server, prompt, self.outputs, x, extra_data, executed)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\execution.py", line 58, in recursive_execute
recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\execution.py", line 58, in recursive_execute
recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\execution.py", line 67, in recursive_execute
outputs[unique_id] = getattr(obj, obj.FUNCTION)(**input_data_all)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\nodes.py", line 786, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\nodes.py", line 757, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 79, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 532, in sample
samples = uni_pc.sample_unipc(self.model_wrap, noise, latent_image, sigmas, sampling_function=sampling_function, max_denoise=max_denoise, extra_args=extra_args, noise_mask=denoise_mask, variant='bh2')
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 880, in sample_unipc
x = uni_pc.sample(img, timesteps=timesteps, skip_type="time_uniform", method="multistep", order=order, lower_order_final=True)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 731, in sample
model_prev_list = [self.model_fn(x, vec_t)]
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 422, in model_fn
return self.data_prediction_fn(x, t)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 404, in data_prediction_fn
noise = self.noise_prediction_fn(x, t)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 398, in noise_prediction_fn
return self.model(x, t)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 330, in model_fn
return noise_pred_fn(x, t_continuous)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\extra_samplers\uni_pc.py", line 298, in noise_pred_fn
output = sampling_function(model, x, t_input, **model_kwargs)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 235, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 140, in calc_cond_uncond_batch
p = get_area_and_mult(x, x_in, cond_concat_in, timestep)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 64, in get_area_and_mult
gligen_patch = gligen_model.set_position(input_x.shape, gligen[2], input_x.device)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\gligen.py", line 280, in set_position
return self._set_position(
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\gligen.py", line 247, in _set_position
objs = self.position_net(boxes, masks, positive_embeddings)
File "D:\goodjob\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\goodjob\ComfyUI_windows_portable\ComfyUI\comfy\gligen.py", line 229, in forward
masks + (1 - masks) * positive_null
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[ltdrdata]

Workflow is fine. Looks like a bug related to GPU handling.

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[SuperJin9]

Well, my GPU is rather old，NVIDIA GeForce GTX 1650,
just now i tried run ControlNet fail ：RuntimeError: GET was unable to find an engine to execute this computation.
As more and more features are added, older computers may not run.

[comfyanonymous]

Can you post the full ComfyUI log?

You can also try updating with: update\update_comfyui.bat

[SuperJin9]

Ok, I've run the .bat update script. The log of the background window is as follows：
####picture#####

###########log#########
ControlNetlog.txt

[SuperJin9]

Sorry, I did not update the graphics card driver to the latest version
Today I went to the official website to download the latest driver and it run ControlNet ok ！！