Deduplication #7

poedator · 2023-06-10T23:20:31Z

combining compression code between llama/falcon and perllexity/lm_eval

tested ppl with Llama65B - negligibly better than original

main.py black interim main, modelU upd modelutils upd modelutils_2 added einops to req main & hf & rest upd remove lm_eval/quant + some minors gitignore more black

Godofnothing · 2023-06-14T09:37:39Z

main.py

    )
+    parser.add_argument("--load", type=str, default="", help="Load quantized model.")


Do we need this argument?

Godofnothing

Overall, is fine to me

datautils.py

Vahe1994 · 2023-06-16T11:06:50Z

datautils.py

+
+    if "llama" in model_path:
+        tokenizer = LlamaTokenizer.from_pretrained(model_path, use_fast=False)
+        # addresses problem on inconsistent `LLaMATokenizer` capitalization


addresses problem on inconsistent LLaMATokenizer capitalization see ...

fixed, moved to docstring

Vahe1994 · 2023-06-16T11:17:17Z

lm-evaluation-harness/lm_eval/models/huggingface.py

-            torch.nn.init.normal_,
-        ) # preserving
-
+


Check that low_cpu_mem_usage=True do the same thing.

1.Got this advice here https://stackoverflow.com/questions/76356591/how-to-skip-weights-init-when-loading-pretrained-transformers-model
2. see comments in HF code: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L2129
Tests for mem use were done, to be found.

lm-evaluation-harness/lm_eval/models/huggingface.py

Merge

poedator · 2023-06-16T20:52:48Z

datautils.py

+
+    if "llama" in model_path:
+        tokenizer = LlamaTokenizer.from_pretrained(model_path, use_fast=False)
+        # addresses problem on inconsistent `LLaMATokenizer` capitalization


fixed, moved to docstring

datautils.py

poedator · 2023-06-16T21:08:35Z

lm-evaluation-harness/lm_eval/models/huggingface.py

-            torch.nn.init.normal_,
-        ) # preserving
-
+


1.Got this advice here https://stackoverflow.com/questions/76356591/how-to-skip-weights-init-when-loading-pretrained-transformers-model
2. see comments in HF code: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L2129
Tests for mem use were done, to be found.

poedator · 2023-06-21T13:39:08Z

lm-evaluation-harness/lm_eval/models/huggingface.py

+import os
+import sys
+
+import_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../..")


recheck imports

poedator · 2023-06-21T13:41:36Z

lm-evaluation-harness/lm_eval/models/huggingface.py

-            elif 'falcon' in pretrained.lower():
-                falcon_sequential(self.model, train_data, quantization_config, device)
-
+            quantize_model(self.model, train_data, quantization_config, device)


recheck is this still needed

poedator · 2023-06-21T14:11:04Z

main.py

-    model.model.embed_tokens = model.model.embed_tokens.cpu()
-    model.model.norm = model.model.norm.cpu()
+    layers[0] = layers[0].to(layer_dev)
+    model.get_input_embeddings().to(emb_dev)


check that changes are equal

poedator · 2023-06-21T14:13:35Z

main.py

-        stats_payload["layer_time"] = time.time() - start_time
-        stats_payload["ol_share"] = round(normal_outlier_count / w_count, 6)
-        stats_payload["out_loss"] = torch.mean(out_losses).item()
+        stats_payload["layer_time"] = round(time.time() - start_time, 2)


get rid off round(c)`vahe1994 -- DONE

poedator · 2023-06-21T14:15:16Z

main.py

-
-    model.model.embed_tokens = model.model.embed_tokens.to(dev)
-    layers[0] = layers[0].to(dev)
+def quantize_nearest(model, args, dev):


poedator · 2023-06-21T14:16:52Z

main.py

-        model.model.norm = model.model.norm.to(dev)
-    model.lm_head = model.lm_head.to(dev)
-
+    get_model_head(model).to(dev)


проверил 22.06 - такой код работает, голова переносится.

lmeval.py

Vahe1994

lgtm

poedator and others added 9 commits June 9, 2023 15:57

datautils.py updated, merged with rw

67e3fac

new modelutils.py

1c0b0e2

main upd

0bd661f

main.py black interim main, modelU upd modelutils upd modelutils_2 added einops to req main & hf & rest upd remove lm_eval/quant + some minors gitignore more black

llama condition upd

4dbee90

imports fix, config fix

ffb5495

params fix

b4fd568

compression return results fixed

5082050

Refactor and README edit

14b0cd0

subsequent minor updates

284feea

poedator force-pushed the dedup branch from 8b905e8 to 284feea Compare June 12, 2023 09:23

poedator added 2 commits June 12, 2023 16:10

get_loaders handles custom now

d5f3675

returning layer to original device, for compatibility with accelerate

abb7f46

poedator force-pushed the dedup branch from 340d53a to abb7f46 Compare June 12, 2023 13:44

poedator added 4 commits June 12, 2023 19:11

main.py improvements

53c8163

new lmeval.py

7b170ef

moved lmeval fns to utils as in newer versions

a533776

fix for non-str output from evaluator

484aed9

poedator force-pushed the dedup branch from ca67ec8 to 5765df2 Compare June 13, 2023 17:02

Vahe1994 requested review from Vahe1994 and Godofnothing June 14, 2023 08:30

Godofnothing reviewed Jun 14, 2023

View reviewed changes

Vahe1994 reviewed Jun 16, 2023

View reviewed changes

datautils.py Show resolved Hide resolved

Vahe1994 reviewed Jun 16, 2023

View reviewed changes

lm-evaluation-harness/lm_eval/models/huggingface.py Outdated Show resolved Hide resolved

poedator added 2 commits June 16, 2023 23:54

fixed calls to get data

ccfbd90

datautils upd

2c3f950

poedator force-pushed the dedup branch from b24b0be to 2c3f950 Compare June 16, 2023 20:55

rolled back some edits in lm_eval

1f12a86

poedator and others added 9 commits June 17, 2023 00:16

removed args.load

011a163

lmeval fixes

a66e62f

upd device refs

4b5d71a

device refs fixed

a0504d0

rm debug print

b1d39c6

req transformers ver upd

b7ac3ab

8bit param pass_thru

5282050

Minorn fixes

9f9d66c

Merge branch 'dedup' of https://github.com/Vahe1994/SpQR into dedup

b283d07

Merge

Vahe1994 mentioned this pull request Jun 21, 2023

Evaluation code for Falcon models #17

Open

poedator commented Jun 22, 2023

View reviewed changes

poedator added 5 commits June 22, 2023 13:14

pr_improvements

728cff5

dataU undo black

871d9cf

upd readmes

d874157

dataU seed fixed

bc6742b

added accelerate to req

db414e2

poedator force-pushed the dedup branch from bd27075 to db414e2 Compare June 27, 2023 12:03

tokenizer patch for 4.28.dev0 compat.

0d40872

Vahe1994 approved these changes Jun 28, 2023

View reviewed changes

Vahe1994 and others added 2 commits June 28, 2023 22:21

Merge branch 'main' into dedup

d702bab

Updated README.md

0b14fc4

Vahe1994 merged commit 1c27ed6 into main Jun 29, 2023

Vahe1994 deleted the dedup branch June 29, 2023 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deduplication #7

Deduplication #7

poedator commented Jun 10, 2023 •

edited

Loading

Godofnothing Jun 14, 2023

Godofnothing left a comment

Vahe1994 Jun 16, 2023 •

edited by poedator

Loading

poedator Jun 16, 2023

Vahe1994 Jun 16, 2023

poedator Jun 16, 2023

poedator Jun 16, 2023

poedator Jun 16, 2023

poedator Jun 21, 2023

poedator Jun 21, 2023

poedator Jun 21, 2023

poedator Jun 21, 2023

poedator Jun 21, 2023

poedator Jun 21, 2023

Vahe1994 left a comment

		)
		parser.add_argument("--load", type=str, default="", help="Load quantized model.")

Deduplication #7

Deduplication #7

Conversation

poedator commented Jun 10, 2023 • edited Loading

Choose a reason for hiding this comment

Godofnothing left a comment

Choose a reason for hiding this comment

Vahe1994 Jun 16, 2023 • edited by poedator Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vahe1994 left a comment

Choose a reason for hiding this comment

poedator commented Jun 10, 2023 •

edited

Loading

Vahe1994 Jun 16, 2023 •

edited by poedator

Loading