-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deduplication #7
Conversation
main.py
Outdated
) | ||
parser.add_argument("--load", type=str, default="", help="Load quantized model.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need this argument?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, is fine to me
datautils.py
Outdated
|
||
if "llama" in model_path: | ||
tokenizer = LlamaTokenizer.from_pretrained(model_path, use_fast=False) | ||
# addresses problem on inconsistent `LLaMATokenizer` capitalization |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addresses problem on inconsistent LLaMATokenizer
capitalization see ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed, moved to docstring
torch.nn.init.normal_, | ||
) # preserving | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check that low_cpu_mem_usage=True do the same thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1.Got this advice here https://stackoverflow.com/questions/76356591/how-to-skip-weights-init-when-loading-pretrained-transformers-model
2. see comments in HF code: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L2129
Tests for mem use were done, to be found.
datautils.py
Outdated
|
||
if "llama" in model_path: | ||
tokenizer = LlamaTokenizer.from_pretrained(model_path, use_fast=False) | ||
# addresses problem on inconsistent `LLaMATokenizer` capitalization |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed, moved to docstring
torch.nn.init.normal_, | ||
) # preserving | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1.Got this advice here https://stackoverflow.com/questions/76356591/how-to-skip-weights-init-when-loading-pretrained-transformers-model
2. see comments in HF code: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L2129
Tests for mem use were done, to be found.
import os | ||
import sys | ||
|
||
import_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../..") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
recheck imports
elif 'falcon' in pretrained.lower(): | ||
falcon_sequential(self.model, train_data, quantization_config, device) | ||
|
||
quantize_model(self.model, train_data, quantization_config, device) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
recheck is this still needed
model.model.embed_tokens = model.model.embed_tokens.cpu() | ||
model.model.norm = model.model.norm.cpu() | ||
layers[0] = layers[0].to(layer_dev) | ||
model.get_input_embeddings().to(emb_dev) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check that changes are equal
main.py
Outdated
stats_payload["layer_time"] = time.time() - start_time | ||
stats_payload["ol_share"] = round(normal_outlier_count / w_count, 6) | ||
stats_payload["out_loss"] = torch.mean(out_losses).item() | ||
stats_payload["layer_time"] = round(time.time() - start_time, 2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
get rid off round(c)`vahe1994 -- DONE
|
||
model.model.embed_tokens = model.model.embed_tokens.to(dev) | ||
layers[0] = layers[0].to(dev) | ||
def quantize_nearest(model, args, dev): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
run 1 exp
model.model.norm = model.model.norm.to(dev) | ||
model.lm_head = model.lm_head.to(dev) | ||
|
||
get_model_head(model).to(dev) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
проверил 22.06 - такой код работает, голова переносится.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
combining compression code between llama/falcon and perllexity/lm_eval
tested ppl with Llama65B - negligibly better than original