Qwen image incontext

[Artiprocher]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @Artiprocher, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces "in-context control" capabilities to the Qwen image generation pipeline, allowing users to guide image synthesis with additional contextual images. It also includes updates to documentation and examples for newly released Qwen-Image models, specifically EliGen-V2 and In-Context Control Union.

Highlights

• Enhanced Image Control: Implemented "in-context control" functionality in the QwenImagePipeline, enabling the use of a context_image to influence image generation.
• New Model Integration: Added support and documentation for two new Qwen-Image models: DiffSynth-Studio/Qwen-Image-EliGen-V2 (an improved EliGen model) and DiffSynth-Studio/Qwen-Image-In-Context-Control-Union (a structural control LoRA model).
• Pipeline Extension: Introduced QwenImageUnit_ContextImageEmbedder to the image processing pipeline to handle the embedding of context images.
• Documentation and Examples: Updated README.md files (English and Chinese) to reflect the new models and their usage, and added new example scripts for inference, low VRAM inference, training, and validation of these models.
• Attention Mask Correction: Corrected the image mask repetition logic within diffsynth/models/qwen_image_dit.py to ensure proper attention mask generation for single image sequences.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant new capabilities, including in-context control for Qwen-Image models and a new version of Qwen-Image-EliGen. The changes are extensive, with new pipeline units, example scripts, and documentation updates. Overall, the implementation is solid. I have identified one high-priority issue in diffsynth/models/qwen_image_dit.py that could lead to a runtime error under certain conditions and should be addressed. I've also included a few medium-severity suggestions to improve the robustness and code quality of the new example scripts.

[gemini-code-assist]

The integer division // used to calculate repeat_time assumes that single_image_seq is perfectly divisible by image_mask.shape[-1]. This holds true if all concatenated images (main, context, edit) have the same latent dimensions. However, if edit_image_auto_resize=False is used with an edit_image of a different aspect ratio, their sequence lengths might differ, leading to a non-integer division. This would cause image_mask.shape[-1] * repeat_time to be shorter than single_image_seq, resulting in a runtime error due to shape mismatch during the assignment to attention_mask.

To make the code more robust, I suggest adding an assertion to ensure divisibility, which makes the assumption explicit and prevents unexpected errors.

Suggested change

	repeat_time = single_image_seq // image_mask.shape[-1]
	image_mask = image_mask.repeat(1, 1, repeat_time)
	if image_mask.shape[-1] > 0 and single_image_seq % image_mask.shape[-1] != 0:
	raise ValueError(
	f"The total image sequence length ({single_image_seq}) is not divisible by "
	f"the mask sequence length ({image_mask.shape[-1]}). "
	"This can happen if concatenated images have different resolutions and `edit_image_auto_resize` is False."
	)
	repeat_time = single_image_seq // image_mask.shape[-1] if image_mask.shape[-1] > 0 else 1
	image_mask = image_mask.repeat(1, 1, repeat_time)

[gemini-code-assist]

The colors list contains duplicate entries. Lines 20-27 are identical to lines 12-19. You can remove these redundant lines to make the code cleaner.

[gemini-code-assist]

The script attempts to load the font wqy-zenhei.ttc directly. This will fail with an IOError if the font is not installed on the user's system. Although there's a fallback to the default font, it likely doesn't support Chinese characters, which are present in some of the example prompts. This will result in unreadable text (tofu characters) in the visualized mask image.

To improve usability, consider providing the font file with the examples or adding a note in the documentation about font installation.

[gemini-code-assist]

The colors list contains duplicate entries. Lines 21-28 are identical to lines 13-20. Removing these redundant lines would make the code cleaner.

[gemini-code-assist]

The script attempts to load the font wqy-zenhei.ttc directly. This will fail with an IOError if the font is not installed on the user's system. Although there's a fallback to the default font, it likely doesn't support Chinese characters, which are present in some of the example prompts. This will result in unreadable text (tofu characters) in the visualized mask image.

To improve usability, consider providing the font file with the examples or adding a note in the documentation about font installation.