Support for immediate saving to reduce ram usage #965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

wenhuach21 merged 38 commits into main from kaihui/save_block

Nov 7, 2025

Contributor

Kaihui-intel commented Oct 30, 2025 •

edited

Loading

Accuracy

scheme /(opt-125m,)	format	RTN	iter>0
W4A16	auto_round	0.2882	0.3526
W2A16	auto_round		0.1657
W3A16	auto_round		0.3247
W8A16	auto_round		0.3784
bit s group_size 32	auto_round	0.3749	0.3679
bit s group_size 32	auto_gptq	0.3747	0.3658
bit s group_size 32	auto_awq	0.3749	0.3646

Memory

memory check
Qwen2.5-7B-Instruct-w4g32 RTN auto_round
mprof peak
16659.441MiB->9200.250MiB ~55%

Time

quantization and saving time
opt-125m

branch	RTN	iter>0
cur branch	67s	78s
main branch	50s	88s

Qwen2.5-7B-Instruct

branch	RTN	iter>0
cur branch	4min4s	18min10s
main branch	3min54s	17min53s

immediate pacing and saving now only support formats[0]

Kaihui-intel and others added 8 commits

October 16, 2025 01:41


          save per block

9da69a8

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          enable multi block save

dc24682

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          support export save

856ab06

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          update rtn support

97cbfdb

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          support

fb026aa

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          rebase main

bb6113f

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          del utils.py

148f085

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

afb6eef

for more information, see https://pre-commit.ci

Contributor

wenhuach21 commented Oct 30, 2025

Thanks for the great work! Could you check the maximum RAM usage to see whether it has been reduced significantly, as expected?

wenhuach21 requested review from n1ck-guo, xin3he and yiliu30

October 30, 2025 09:00

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved


          fix args

b2c14df

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>

xin3he modified the milestones: 1.0, 0.9.0

Kaihui-intel and others added 8 commits

October 31, 2025 01:07


          optimize memory and fix rtn module

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          revert max_shard_size

8f38839

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          move save_block_immediate into utils

70d873e

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

f4ace2c

for more information, see https://pre-commit.ci


          [pre-commit.ci] auto fixes from pre-commit.com hooks

c314231

for more information, see https://pre-commit.ci


          fix import

f64cfcf

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          update max_shard_size

e471148

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          Merge branch 'kaihui/save_block' of https://github.com/intel/auto-round…

b19db96

… into kaihui/save_block

Kaihui-intel requested a review from wenhuach21

October 31, 2025 07:56

wenhuach21 changed the title ~~Support for immediate saving~~ [High Risk]Support for immediate saving

wenhuach21 requested review from WeiweiZhang1 and hshen14

October 31, 2025 08:14

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Show resolved Hide resolved

Kaihui-intel and others added 6 commits

November 4, 2025 02:54


          set immediate packing saving to True

5aa3f1e

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

67e2591

for more information, see https://pre-commit.ci


          Merge branch 'main' into kaihui/save_block

ca55a90


          flow saving setting

7a21c5a

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          check gguf

e5fd4f2

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          pack layer_names immediately

5367af6

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved

Kaihui-intel and others added 3 commits

November 5, 2025 08:30


          wrapper _immediate_pack & rm uts & pop lm_head & rm expose args

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          revert ut

ceb2a92

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

5375be6

for more information, see https://pre-commit.ci

Contributor

wenhuach21 commented Nov 6, 2025

Since this PR is still in development and may not have undergone much verification, I suggest adding a low_cpu_mem_usage option to the API. If enabled, we can trigger save_immediately whenever possible.

Kaihui-intel and others added 4 commits

November 6, 2025 00:29


          add low_cpu_mem_usage

70f59ec

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          merge main

bd1532b

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

a551e13

for more information, see https://pre-commit.ci


          fix conflict

50a82e2

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/utils.py Show resolved Hide resolved

wenhuach21 reviewed

View reviewed changes

test/test_cpu/test_autoround.py Outdated Show resolved Hide resolved

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved


          update low_cpu_mem_usage

3c682c6

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>

wenhuach21 reviewed

View reviewed changes

auto_round/compressors/base.py Outdated Show resolved Hide resolved

wenhuach21 reviewed

View reviewed changes

auto_round/autoround.py Outdated Show resolved Hide resolved


          move to last arg

Signed-off-by: Kaihui-intel <kaihui.tang@intel.com>

wenhuach21 self-requested a review

November 7, 2025 05:05

wenhuach21 changed the title ~~[High Risk]Support for immediate saving~~ Support for immediate saving to reduce ram usage

wenhuach21 approved these changes

View reviewed changes

wenhuach21 merged commit daeb3bb into main

23 checks passed

wenhuach21 deleted the kaihui/save_block branch

November 7, 2025 05:06

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

wenhuach21 wenhuach21 approved these changes

n1ck-guo Awaiting requested review from n1ck-guo

yiliu30 Awaiting requested review from yiliu30

xin3he Awaiting requested review from xin3he

WeiweiZhang1 Awaiting requested review from WeiweiZhang1

hshen14 Awaiting requested review from hshen14

Labels

None yet