Skip to content

Comments

reduce memory usage during repack#168

Merged
wejoncy merged 1 commit intowejoncy:mainfrom
ReinForce-II:main
Mar 7, 2025
Merged

reduce memory usage during repack#168
wejoncy merged 1 commit intowejoncy:mainfrom
ReinForce-II:main

Conversation

@ReinForce-II
Copy link
Contributor

solve #166

@wejoncy
Copy link
Owner

wejoncy commented Mar 7, 2025

Thanks for your contribution.

One question: Do you verify it of reducing memory usage? Could you please paste the number here?

Much appreciate.

@ReinForce-II
Copy link
Contributor Author

python -m qllm --load Qwen/Qwen2.5-14B-Instruct-AWQ --eval --save /tmp/tmp0012 --pack_mode=GPTQ

w/o this change, peak memory usage is 66.8GiB.
w/ this change, peak memory usage is 23.0GiB.

I didn't do more tests.

@wejoncy wejoncy merged commit df20c15 into wejoncy:main Mar 7, 2025
2 checks passed
@wejoncy wejoncy linked an issue Mar 7, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory consumption about convert model format

2 participants