Skip to content
This repository has been archived by the owner on Oct 25, 2024. It is now read-only.

Fix missing documentation in LLM runtime, and a broken link + some typos #1128

Merged
merged 3 commits into from
Jan 11, 2024
Merged

Fix missing documentation in LLM runtime, and a broken link + some typos #1128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
85634c3
Fix broken link
aahouzi Jan 10, 2024
dac13ae
Add requirements installation for Python script
aahouzi Jan 10, 2024
d042922
Fix script typos
aahouzi Jan 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion intel_extension_for_transformers/llm/runtime/graph/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,10 +408,11 @@ while True:
outputs = model.generate(inputs, streamer=streamer, interactive=True, ignore_prompt=True, do_sample=True)
```

## How to use: Python script
## How to use: Straightforward Python script
Install from binary
```shell
pip install intel-extension-for-transformers
pip install -r requirements.txt # under graph folder
```

Build from source
Expand Down
6 changes: 3 additions & 3 deletions intel_extension_for_transformers/llm/runtime/graph/scripts/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ def main(args_in: Optional[List[str]] = None) -> None:
convert_cmd.extend(["--outfile", Path(work_path, "ne_{}_f32.bin".format(model_type))])
convert_cmd.extend(["--outtype", "f32"])
convert_cmd.append(args.model)
print("convert model ...")
print("Convert model ...")
subprocess.run(convert_cmd)

# 2. quantize
Expand All @@ -185,7 +185,7 @@ def main(args_in: Optional[List[str]] = None) -> None:
if args.use_ggml:
quant_cmd.extend(["--use_ggml"])
quant_cmd.extend(["--build_dir", args.build_dir])
print("quantize model ...")
print("Quantize model ...")
subprocess.run(quant_cmd)

# 3. inference
Expand All @@ -204,7 +204,7 @@ def main(args_in: Optional[List[str]] = None) -> None:
infer_cmd.extend(["--build_dir", args.build_dir])
if args.shift_roped_k:
infer_cmd.extend(["--shift-roped-k"])
print("inferce model ...")
print("Inference model ...")
subprocess.run(infer_cmd)


Expand Down