support Qwen-Image #130

Glaceon-Hyy · 2025-08-04T10:58:39Z

No description provided.

gemini-code-assist

Summary of Changes

Hello @Glaceon-Hyy, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the capabilities of the diffsynth_engine by adding full support for the Qwen-Image model. This integration involves introducing new model architectures, configurations, and a dedicated tokenizer, allowing users to leverage Qwen-Image for advanced image generation. The changes also include general pipeline enhancements for better model loading, state dictionary management, and performance optimizations like feature-based caching and torch.compile integration.

Highlights

New Model Support: Introduced comprehensive support for the Qwen-Image model, enabling its use for image generation tasks within the diffsynth_engine.
Component Integration: Integrated key components of the Qwen-Image model, including its Diffusion Transformer (DiT), Variational Autoencoder (VAE), and a Qwen2.5-VL based text/vision encoder.
Configuration and Tokenizer Additions: Added new configuration classes (QwenImagePipelineConfig, QwenImageStateDicts) and a dedicated tokenizer (Qwen2TokenizerFast) to manage the Qwen-Image model's specific requirements.
Optimization Features: Implemented QwenImageDiTFBCache for optimized inference with feature-based caching and added use_torch_compile support to pipeline configurations for potential performance gains.
Pipeline Structure Refinement: Refactored the base pipeline and existing image pipelines (Flux, SD, SDXL) to standardize state dictionary loading and model initialization using new BaseStateDicts and model-specific StateDicts classes.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for Qwen-Image, including new models, configurations, and a dedicated pipeline.

I've identified a critical issue with a potentially incorrect configuration file that could affect existing models, along with several opportunities for refactoring to improve code clarity and maintainability.

gemini-code-assist · 2025-08-04T11:01:27Z

tests/common/utils.py

+    checkpoint_path: str | List[str], device: str = "cpu", dtype: torch.dtype = torch.float16
+) -> Dict[str, torch.Tensor]:
+    if isinstance(checkpoint_path, str):
+        checkpoint_path = [checkpoint_path]
+    state_dict = {}
+    for path in checkpoint_path:
+        if not os.path.isfile(path):
+            raise FileNotFoundError(f"{path} is not a file")
+        elif path.endswith(".safetensors"):
+            state_dict_ = load_file(path, device=device)
+            for key, value in state_dict_.items():
+                state_dict[key] = value.to(dtype)
+
+        elif path.endswith(".gguf"):
+            state_dict.update(**load_gguf_checkpoint(path, device=device, dtype=dtype))
+        else:
+            raise ValueError(f"{path} is not a .safetensors or .gguf file")
+    return state_dict


This load_model_checkpoint function is a duplicate of the static method with the same name in diffsynth_engine.pipelines.base.BasePipeline.

To avoid code duplication and improve maintainability, please consider removing this function and importing the one from BasePipeline in your test files where it's needed.

gemini-code-assist · 2025-08-04T11:01:27Z

diffsynth_engine/pipelines/flux_image.py

+        cls.convert(state_dicts.model, config.model_dtype)
+        cls.convert(state_dicts.vae, config.vae_dtype)
+        if config.load_text_encoder:
+            cls.convert(state_dicts.clip, config.clip_dtype)
+            cls.convert(state_dicts.t5, config.t5_dtype)


The cls.convert() calls here appear to be redundant. The cls.load_model_checkpoint() method already converts the loaded tensors to the specified dtype.

Calling convert again on the already-converted state dicts is unnecessary and could be removed to simplify the code.

Suggested change

cls.convert(state_dicts.model, config.model_dtype)

cls.convert(state_dicts.vae, config.vae_dtype)

if config.load_text_encoder:

cls.convert(state_dicts.clip, config.clip_dtype)

cls.convert(state_dicts.t5, config.t5_dtype)

# cls.convert(state_dicts.model, config.model_dtype)

# cls.convert(state_dicts.vae, config.vae_dtype)

# if config.load_text_encoder:

# cls.convert(state_dicts.clip, config.clip_dtype)

# # cls.convert(state_dicts.t5, config.t5_dtype)

gemini-code-assist · 2025-08-04T11:01:27Z

diffsynth_engine/models/qwen_image/qwen2_5_vl.py

+            self.num_attention_heads * self.head_dim, self.hidden_size, bias=False, device=device, dtype=dtype
+        )
+
+        self.rotary_emb = Qwen2_5_VLRotaryEmbedding(dim=self.head_dim, device=device, dtype=dtype)


The self.rotary_emb attribute is initialized in the __init__ method but it's never used in the forward method of this class. The forward method receives position_embeddings as an argument and uses that instead.

This appears to be dead code and should be removed to improve clarity and reduce memory usage.

Suggested change

self.rotary_emb = Qwen2_5_VLRotaryEmbedding(dim=self.head_dim, device=device, dtype=dtype)

# self.rotary_emb = Qwen2_5_VLRotaryEmbedding(dim=self.head_dim, device=device, dtype=dtype)

gemini-code-assist · 2025-08-04T11:01:27Z

diffsynth_engine/models/qwen_image/qwen2_5_vl.py

+                position_ids = torch.arange(seq_length, device=inputs_embeds.device)
+                position_ids = position_ids.view(1, -1).expand(batch_size, -1)
+                if cache_position is not None:  # otherwise `deltas` is an int `0`
+                    delta = delta.repeat_interleave(batch_size // delta.shape[0], dim=0)


The repeat_interleave call here is a no-op. self.rope_deltas has a shape of (batch_size, 1), so delta will also have a batch dimension equal to the batch size. This means batch_size // delta.shape[0] will always be 1.

While this doesn't cause a functional issue, it's confusing and suggests the code might have been copied from a different context. It would be clearer to remove this unnecessary operation.

Suggested change

delta = delta.repeat_interleave(batch_size // delta.shape[0], dim=0)

if cache_position is not None: # otherwise `deltas` is an int `0`

delta = delta

gemini-code-assist · 2025-08-04T11:01:28Z

diffsynth_engine/pipelines/sd_image.py

+    def from_state_dict(cls, state_dicts: SDStateDicts, pipeline_config: SDPipelineConfig) -> "SDImagePipeline":
        raise NotImplementedError()



The signature of from_state_dict has been updated to align with the base class, but the implementation is still raise NotImplementedError(). This indicates an incomplete refactoring.

To improve maintainability and ensure consistency, this method should either be fully implemented for this pipeline or removed if it's not intended to be supported.

gemini-code-assist · 2025-08-04T11:01:28Z

diffsynth_engine/pipelines/sdxl_image.py

+    def from_state_dict(cls, state_dicts: SDXLStateDicts, pipeline_config: SDXLPipelineConfig) -> "SDXLImagePipeline":
        raise NotImplementedError()



Similar to sd_image.py, the from_state_dict method signature has been updated, but the implementation is still raise NotImplementedError(). This should be either implemented or removed for consistency and to avoid confusion.

akaitsuki-ii · 2025-08-04T12:18:39Z

tests/test_tokenizers/test_qwen2.py

+    def setUpClass(cls):
+        cls.tokenizer = Qwen2TokenizerFast.from_pretrained(QWEN_IMAGE_TOKENIZER_CONF_PATH)
+
+    # TODO: fix


@qzzz95 这个test case有问题，mark一下之后帮忙看看

support Qwen-Image

4e074c1

gemini-code-assist bot reviewed Aug 4, 2025

View reviewed changes

modelscope deleted a comment from gemini-code-assist bot Aug 4, 2025

update docs

41d1434

akaitsuki-ii reviewed Aug 4, 2025

View reviewed changes

akaitsuki-ii approved these changes Aug 4, 2025

View reviewed changes

akaitsuki-ii merged commit 10bff22 into main Aug 4, 2025

akaitsuki-ii deleted the Qwen-Image branch August 4, 2025 12:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

support Qwen-Image #130

support Qwen-Image #130

Uh oh!

Glaceon-Hyy commented Aug 4, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Aug 4, 2025

Uh oh!

gemini-code-assist bot Aug 4, 2025

Uh oh!

gemini-code-assist bot Aug 4, 2025

Uh oh!

gemini-code-assist bot Aug 4, 2025

Uh oh!

gemini-code-assist bot Aug 4, 2025

Uh oh!

gemini-code-assist bot Aug 4, 2025

Uh oh!

akaitsuki-ii Aug 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	self.rotary_emb = Qwen2_5_VLRotaryEmbedding(dim=self.head_dim, device=device, dtype=dtype)
	# self.rotary_emb = Qwen2_5_VLRotaryEmbedding(dim=self.head_dim, device=device, dtype=dtype)

	delta = delta.repeat_interleave(batch_size // delta.shape[0], dim=0)
	if cache_position is not None: # otherwise `deltas` is an int `0`
	delta = delta

		def from_state_dict(cls, state_dicts: SDStateDicts, pipeline_config: SDPipelineConfig) -> "SDImagePipeline":
		raise NotImplementedError()

support Qwen-Image #130

support Qwen-Image #130

Uh oh!

Conversation

Glaceon-Hyy commented Aug 4, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

akaitsuki-ii Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants