Skip to content

Implement shard parallelism configuration and update documentation#1942

Merged
chensuyue merged 11 commits into
mainfrom
xinhe/6-23
Jun 24, 2026
Merged

Implement shard parallelism configuration and update documentation#1942
chensuyue merged 11 commits into
mainfrom
xinhe/6-23

Conversation

@xin3he

@xin3he xin3he commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Description

This pull request introduces and documents a new environment variable, AR_MODEL_FREE_SHARD_PARALLELISM, which controls the number of shards processed concurrently during model-free quantization. It also adds comprehensive tests for the new parallelism policy, updates documentation (both English and Chinese) to explain the variable and its effects (including benchmarks), and adds a new test for quantization config building. These changes improve configurability, transparency, and reliability of model-free quantization.

Environment variable and configuration support:

  • Added the AR_MODEL_FREE_SHARD_PARALLELISM environment variable to auto_round/envs.py to control model-free quantization parallelism, including logic for auto-policy and environment override. [1] [2]

Documentation updates:

  • Documented AR_MODEL_FREE_SHARD_PARALLELISM in both English and Chinese environment variable guides, including description, default policy, usage, and examples. [1] [2]
  • Added model-free quantization parallelism benchmarks (tables and analysis) to both English and Chinese step-by-step guides. [1] [2]

Testing and validation:

  • Added a new test class to verify the shard parallelism policy, including auto-policy, environment override, capping, invalid input fallback, and correct handling of non-divisible shard counts.
  • Added a test for the quantization config builder to ensure correct filtering of ignored layers.
  • Imported the new _build_quantization_config function in the test module.

Parallelism Benefit Table (Using Rounded Minutes)

Model Scheme Comparison Runtime Change (min) Speedup Time Saved Peak Memory Change
Qwen3-235B W4A16 Parallelism 1 -> 10 17 -> 5 3.40x 70.6% 2G -> 8G
GLM-5.2 MXFP4-Mixed Parallelism 1 -> 10 60 -> 16 3.75x 73.3% 2G -> 27G
GLM-5.2 W4A16 Parallelism 1 -> 10 30 -> 10 3.00x 66.7% 3G -> 16G
GLM-5.2 W4A16 Parallelism 1 -> 20 30 -> 8 3.75x 73.3% 3G -> 32G
MiniMax-M2.7 W4A16 Parallelism 1 -> 10 18 -> 5 3.60x 72.2% 2G -> 10G
DeepSeek-V4-Pro W4A16 Parallelism 1 -> 10 80 -> 13 6.15x 83.8% 6G -> 50G

Type of Change

New feature

Related Issues

Fixes or relates to #

Checklist Before Submitting

  • My code has been tested locally.
  • Documentation has been updated as needed.
  • New or updated tests are included where applicable.
  • The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

Signed-off-by: Xin He <xin3.he@intel.com>
@chensuyue

Copy link
Copy Markdown
Contributor

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@xin3he xin3he closed this Jun 23, 2026
@xin3he xin3he reopened this Jun 23, 2026
@xin3he xin3he marked this pull request as draft June 23, 2026 07:17
@xin3he xin3he marked this pull request as ready for review June 23, 2026 12:06
xin3he added 2 commits June 24, 2026 02:06
Signed-off-by: Xin He <xin3.he@intel.com>
Signed-off-by: Xin He <xin3.he@intel.com>
@xin3he xin3he added this to the 0.14.0 milestone Jun 24, 2026
…on config tests

Signed-off-by: Xin He <xin3.he@intel.com>
@chensuyue

Copy link
Copy Markdown
Contributor

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

xin3he added 3 commits June 24, 2026 03:45
Signed-off-by: Xin He <xin3.he@intel.com>

@lkk12014402 lkk12014402 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread auto_round/envs.py Outdated
Comment thread auto_round/compressors/model_free.py
Comment thread auto_round/compressors/model_free.py
Comment thread auto_round/compressors/model_free.py Outdated
@chensuyue

Copy link
Copy Markdown
Contributor

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Comment thread auto_round/compressors/model_free.py Outdated
Signed-off-by: Xin He <xin3.he@intel.com>
@chensuyue

Copy link
Copy Markdown
Contributor

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@chensuyue chensuyue merged commit 7e2b69d into main Jun 24, 2026
45 of 47 checks passed
@chensuyue chensuyue deleted the xinhe/6-23 branch June 24, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants