微调后的模型怎么放入web_demo中进行使用啊？

[JACKzhuz]

No description provided.

[JACKzhuz]

我已经微调完成了，然后推理也成功了，我想放入web_demo中进行使用，但是这个应该怎么做啊？在哪里更改路径呢？

[1049451037]

这里：https://github.com/THUDM/VisualGLM-6B/blob/main/model/infer_util.py#L21

同时还要在web_demo.py的main函数之前from finetune_visualglm import FineTuneVisualGLMModel

[JACKzhuz]

好的 感谢大佬

[631068264]

这里：https://github.com/THUDM/VisualGLM-6B/blob/main/model/infer_util.py#L21

同时还要在web_demo.py的main函数之前from finetune_visualglm import FineTuneVisualGLMModel

改了下还是不行

For torch.distributed users or loading model parallel models, set environment variables RANK, WORLD_SIZE and LOCAL_RANK.
/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
[2023-09-01 15:40:28,284] [INFO] [RANK 0]  > number of parameters on model parallel rank 0: 7802193408
[2023-09-01 15:40:37,467] [INFO] [RANK 0] global rank 0 is loading checkpoint /xxxxxx/VisualGLM-6B/checkpoints/finetune-visualglm-6b-09-01-14-44/300/mp_rank_00_model_states.pt
[2023-09-01 15:41:01,711] [INFO] [RANK 0] Will continue but found unexpected_keys! Check whether you are loading correct checkpoints: ['transformer.layers.0.attention.query_key_value.original.weight', 'transformer.layers.0.attention.query_key_value.original.bias', 'transformer.layers.0.attention.query_key_value.matrix_A.0', 'transformer.layers.0.attention.query_key_value.matrix_A.1', 'transformer.layers.0.attention.query_key_value.matrix_A.2', 'transformer.layers.0.attention.query_key_value.matrix_B.0', 'transformer.layers.0.attention.query_key_value.matrix_B.1', 'transformer.layers.0.attention.query_key_value.matrix_B.2', 'transformer.layers.0.attention.dense.original.weight', 'transformer.layers.0.attention.dense.original.bias', 'transformer.layers.0.attention.dense.matrix_A.0', 'transformer.layers.0.attention.dense.matrix_B.0', 'transformer.layers.14.attention.query_key_value.original.weight', 'transformer.layers.14.attention.query_key_value.original.bias', 'transformer.layers.14.attention.query_key_value.matrix_A.0', 'transformer.layers.14.attention.query_key_value.matrix_A.1', 'transformer.layers.14.attention.query_key_value.matrix_A.2', 'transformer.layers.14.attention.query_key_value.matrix_B.0', 'transformer.layers.14.attention.query_key_value.matrix_B.1', 'transformer.layers.14.attention.query_key_value.matrix_B.2', 'transformer.layers.14.attention.dense.original.weight', 'transformer.layers.14.attention.dense.original.bias', 'transformer.layers.14.attention.dense.matrix_A.0', 'transformer.layers.14.attention.dense.matrix_B.0'].
Traceback (most recent call last):
  File "/data/home/yaokj5/dl/apps/VisualGLM-6B/web_demo.py", line 130, in <module>
    main(args)
  File "/data/home/yaokj5/dl/apps/VisualGLM-6B/web_demo.py", line 84, in main
    model, tokenizer = get_infer_setting(gpu_device=0, quant=args.quant)
  File "/data/home/yaokj5/dl/apps/VisualGLM-6B/model/infer_util.py", line 21, in get_infer_setting
    model, args = VisualGLMModel.from_pretrained('/data/home/yaokj5/dl/apps/VisualGLM-6B/checkpoints/finetune-visualglm-6b-09-01-14-44', args)
  File "/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/sat/model/base_model.py", line 215, in from_pretrained
    return cls.from_pretrained_base(name, args=args, home_path=home_path, url=url, prefix=prefix, build_only=build_only, overwrite_args=overwrite_args, **kwargs)
  File "/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/sat/model/base_model.py", line 209, in from_pretrained_base
    load_checkpoint(model, args, load_path=model_path, prefix=prefix)
  File "/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/sat/training/model_io.py", line 247, in load_checkpoint
    raise ValueError(f'Missing keys for inference: {missing_keys}.\nIf you still want to inference anyway, pass --force_inference to args.')
ValueError: Missing keys for inference: ['transformer.layers.0.attention.query_key_value.weight', 'transformer.layers.0.attention.query_key_value.bias', 'transformer.layers.0.attention.dense.weight', 'transformer.layers.0.attention.dense.bias', 'transformer.layers.14.attention.query_key_value.weight', 'transformer.layers.14.attention.query_key_value.bias', 'transformer.layers.14.attention.dense.weight', 'transformer.layers.14.attention.dense.bias'].
If you still want to inference anyway, pass --force_inference to args.

[hnsywangxin]

这里：https://github.com/THUDM/VisualGLM-6B/blob/main/model/infer_util.py#L21
同时还要在web_demo.py的main函数之前from finetune_visualglm import FineTuneVisualGLMModel

改了下还是不行

For torch.distributed users or loading model parallel models, set environment variables RANK, WORLD_SIZE and LOCAL_RANK.
/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
[2023-09-01 15:40:28,284] [INFO] [RANK 0]  > number of parameters on model parallel rank 0: 7802193408
[2023-09-01 15:40:37,467] [INFO] [RANK 0] global rank 0 is loading checkpoint /xxxxxx/VisualGLM-6B/checkpoints/finetune-visualglm-6b-09-01-14-44/300/mp_rank_00_model_states.pt
[2023-09-01 15:41:01,711] [INFO] [RANK 0] Will continue but found unexpected_keys! Check whether you are loading correct checkpoints: ['transformer.layers.0.attention.query_key_value.original.weight', 'transformer.layers.0.attention.query_key_value.original.bias', 'transformer.layers.0.attention.query_key_value.matrix_A.0', 'transformer.layers.0.attention.query_key_value.matrix_A.1', 'transformer.layers.0.attention.query_key_value.matrix_A.2', 'transformer.layers.0.attention.query_key_value.matrix_B.0', 'transformer.layers.0.attention.query_key_value.matrix_B.1', 'transformer.layers.0.attention.query_key_value.matrix_B.2', 'transformer.layers.0.attention.dense.original.weight', 'transformer.layers.0.attention.dense.original.bias', 'transformer.layers.0.attention.dense.matrix_A.0', 'transformer.layers.0.attention.dense.matrix_B.0', 'transformer.layers.14.attention.query_key_value.original.weight', 'transformer.layers.14.attention.query_key_value.original.bias', 'transformer.layers.14.attention.query_key_value.matrix_A.0', 'transformer.layers.14.attention.query_key_value.matrix_A.1', 'transformer.layers.14.attention.query_key_value.matrix_A.2', 'transformer.layers.14.attention.query_key_value.matrix_B.0', 'transformer.layers.14.attention.query_key_value.matrix_B.1', 'transformer.layers.14.attention.query_key_value.matrix_B.2', 'transformer.layers.14.attention.dense.original.weight', 'transformer.layers.14.attention.dense.original.bias', 'transformer.layers.14.attention.dense.matrix_A.0', 'transformer.layers.14.attention.dense.matrix_B.0'].
Traceback (most recent call last):
  File "/data/home/yaokj5/dl/apps/VisualGLM-6B/web_demo.py", line 130, in <module>
    main(args)
  File "/data/home/yaokj5/dl/apps/VisualGLM-6B/web_demo.py", line 84, in main
    model, tokenizer = get_infer_setting(gpu_device=0, quant=args.quant)
  File "/data/home/yaokj5/dl/apps/VisualGLM-6B/model/infer_util.py", line 21, in get_infer_setting
    model, args = VisualGLMModel.from_pretrained('/data/home/yaokj5/dl/apps/VisualGLM-6B/checkpoints/finetune-visualglm-6b-09-01-14-44', args)
  File "/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/sat/model/base_model.py", line 215, in from_pretrained
    return cls.from_pretrained_base(name, args=args, home_path=home_path, url=url, prefix=prefix, build_only=build_only, overwrite_args=overwrite_args, **kwargs)
  File "/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/sat/model/base_model.py", line 209, in from_pretrained_base
    load_checkpoint(model, args, load_path=model_path, prefix=prefix)
  File "/data/home/yaokj5/anaconda3/envs/vglm/lib/python3.10/site-packages/sat/training/model_io.py", line 247, in load_checkpoint
    raise ValueError(f'Missing keys for inference: {missing_keys}.\nIf you still want to inference anyway, pass --force_inference to args.')
ValueError: Missing keys for inference: ['transformer.layers.0.attention.query_key_value.weight', 'transformer.layers.0.attention.query_key_value.bias', 'transformer.layers.0.attention.dense.weight', 'transformer.layers.0.attention.dense.bias', 'transformer.layers.14.attention.query_key_value.weight', 'transformer.layers.14.attention.query_key_value.bias', 'transformer.layers.14.attention.dense.weight', 'transformer.layers.14.attention.dense.bias'].
If you still want to inference anyway, pass --force_inference to args.

model, args = VisualGLMModel.from_pretrained('visualglm-6b', args)
to

finetuned_model = "your finetuned model path"
from finetune_visualglm import FineTuneVisualGLMModel
model, args = FineTuneVisualGLMModel.from_pretrained(finetuned_model, args)