llama2-infer

学习 https://github.com/zjhellofss/KuiperInfer，从零实现一个深度学习推理框架的练手实验。

编译

mkdir build && cd build
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
cmake ..
make

读取huggingface格式模型，直接生成tokenizer, param和bin文件，使用了ncnn的param格式，便于netron等工具查看。

python export_model.py models/stories42M stories42M

以转换时最后一个参数为stories42M为例，转换后会在当前文件夹下生成stories42M.ncnn.param, stories42M.ncnn.bin, stories42M_tokenizer.bin这3个文件。推理命令如下

./inference stories42M "Once upon a time" 256 0.8

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
README.md		README.md
binaryop.cpp		binaryop.cpp
binaryop.h		binaryop.h
concat.cpp		concat.cpp
concat.h		concat.h
convert.py		convert.py
embed.cpp		embed.cpp
embed.h		embed.h
export_model.py		export_model.py
gguf.py		gguf.py
graph.cpp		graph.cpp
graph.h		graph.h
import_hf.py		import_hf.py
inference.cpp		inference.cpp
input.cpp		input.cpp
input.h		input.h
layer.cpp		layer.cpp
layer.h		layer.h
layer_register.cpp		layer_register.cpp
layer_register.h		layer_register.h
matmul.cpp		matmul.cpp
matmul.h		matmul.h
memorydata.cpp		memorydata.cpp
memorydata.h		memorydata.h
model.py		model.py
node.cpp		node.cpp
node.h		node.h
posenc.cpp		posenc.cpp
posenc.h		posenc.h
reduction.cpp		reduction.cpp
reduction.h		reduction.h
reshape.cpp		reshape.cpp
reshape.h		reshape.h
sample.py		sample.py
softmax.cpp		softmax.cpp
softmax.h		softmax.h
swish.cpp		swish.cpp
swish.h		swish.h
tensor.cpp		tensor.cpp
tensor.h		tensor.h
tokenizer.py		tokenizer.py
unaryop.cpp		unaryop.cpp
unaryop.h		unaryop.h
utils.cpp		utils.cpp
utils.h		utils.h