How to define infer/eval configs to accelerate when nginx provides multiple API/url model inference calls?

[hanjr92]

hello! I use nginx to provide multiple remote model inference interfaces. How can I set inference configs to accelerate the inference process?
Now，i set infer cfg like this：
`infer = dict(
partitioner=dict(type=SizePartitioner, max_task_size=500),
runner=dict(
type=LocalRunner,
max_num_workers=4,
task=dict(type=OpenICLInferTask)),
)

eval = dict(
partitioner=dict(type=NaivePartitioner),
runner=dict(
type=LocalRunner,
max_num_workers=32,
task=dict(type=OpenICLEvalTask)),
)Does this parametermax_num_workers` mean that four urls can be used for parallel inference through nginx?

[Leymore]

Yes, max_num_workers can be used for parallel inference. But I suggest a round-robin over the urls in the Model class, which is easier to implement and more intuitive in concept.
You may find this document helpful

[hanjr92]

thanks! another question for your help!
If I submit an API for testing my LLM. Which set of datasets will be used for testing? Base_ Medium or chat_ Medium or a secret for tester?

[hanjr92]

So the ratings displayed on the official website are evaluated through chat_medium?

[gaotongxiao]

For API-based models like ChatGPT, yes. For other open-source models which allow us to access the raw logits, we use base_medium to test, as discriminative evaluation comes with less noise.

[hanjr92]

i see. thanks for your reply.

[hanjr92]

There are a total of 38 datasets on the official website LLM Leaderboard, but there are 52 datasets in chat_medium. Is there any difference between them?

[gaotongxiao]

While OpenCompass is continuously supporting more datasets, the leaderboard may lie behind since the evaluation takes time. We will post updates to the leaderboard recently.