How to use 1.3B OPT weights in metaseq-api-local API?

[ParadoxZW]

I'm trying to use 1.3B OPT weights in metaseq-api-local API and test some inference examples. But I got stucked.

There are two reshard files of 1.3B OPT. But CHECKPOINT_LOCAL in constants.py can only point to a reshard.pt file. What is the correct way to load these two reshard files? (P.S. the comments in constants.py says MODEL_SHARED_FOLDER = "/example/175B/reshard_no_os" but there is CHECKPOINT_FOLDER = os.path.join(MODEL_SHARED_FOLDER, "175B", "reshard_no_os") in line 49, is this also a mistake?

[soloice]

You can modify the paths manually to load different models. Also, I think this script is related.

[ParadoxZW]

@soloice I know that we can modify path to load different models. My question is CHECKPOINT_LOCAL can only point to a ckpt file but not a list of reshard files.

[soloice]

I have the same question. I assume this is not a bug: the only thing missing here is a script to merge all sharded checkpoints.
This should be the one needed. So we can just wait for good news.

[ParadoxZW]

I guess there is a way to directly load two reshard file (and infer under the strategy of model paralleling) rather than merging them.

[suchenzang]

Consolidating all asks for model loading into #88, which will be released soon from HuggingFace.

We also have #78 and #77 open to fix some of these usability issues on our end.