Q: about models load

[kawhiiiileo]

(main_task pid=65914) Retrying...
(main_task pid=65914) Error code: 404 - {'error': {'message': 'The model meta/llama-3.1-70b-instruct does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}, 'request_id': 'cca43f64-79fe-4880-b95b-32e7b402b526'}
(main_task pid=65914) Retrying...
(main_task pid=65914) Error code: 404 - {'error': {'message': 'The model meta/llama-3.1-70b-instruct does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}, 'request_id': 'fe064925-6437-4775-9fa2-cbc35c212724'}
(main_task pid=65914) Retrying...
(main_task pid=65914) Error code: 404 - {'error': {'message': 'The model meta/llama-3.1-8b-instruct does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}, 'request_id': 'ca818b1d-a238-40cc-9413-e11288931c9b'}
(main_task pid=65914) Retrying...
(main_task pid=65914) Error code: 404 - {'error': {'message': 'The model google/gemma-2-27b-it does not exist or you do not have access to it.', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}, 'request_id': 'b236781e-57e0-4177-84f9-1300ad23a51a'}
(main_task pid=65914) Retrying...

Thank you for your excellent work. I encountered the issue mentioned above during reproduction, where it seems that Hugging Face cannot connect properly. I would like to ask, if a normal connection cannot be established, do I need to download all the models locally? Looking forward to your reply.

[ViktorAxelsen]

Thank you for your interest in our work.

From the error messages, it is very likely due to an API configuration issue.

In this work, we access candidate LLMs via LLM API services, rather than loading Hugging Face models locally (although you can do so if you have sufficient GPU resources). You can use any supported API platform (e.g., OpenAI-compatible services, Together AI, etc.), as long as the API base URL and API key are configured correctly.

Please also ensure that the model names are specified correctly. Different API providers often use different model identifiers (even for the same underlying model), and mismatched names can trigger the model_not_found error you observed. This also affects the LLM name parser configuration in the code.

[kawhiiiileo]

Yes, even after downloading all the models, I still encounter this error. Could you kindly provide the relevant URL or information regarding the api_key that you use during experiments? I would really appreciate it!

[ViktorAxelsen]

If you are loading the candidate models locally via Hugging Face, then based on the error messages you shared, I suspect this may be due to a model name mismatch (especially case sensitivity).

For example, your error shows:

meta/llama-3.1-70b-instruct

while the actual Hugging Face repository name is:

meta-llama/Llama-3.1-70B-Instruct

I’ve run into similar issues before where the model identifier (including capitalization and namespace) did not exactly match the HF repo name, which led to model_not_found errors.

I’m not entirely sure whether this is the root cause in your case, but it may be worth double-checking.

[kawhiiiileo]

(WorkerDict pid=55819) NCCL version 2.20.5+cuda12.4 [repeated 6x across cluster]
(WorkerDict pid=55815) kwargs: {'n': 1, 'logprobs': 1, 'max_tokens': 1024, 'detokenize': False, 'temperature': 1, 'top_k': -1, 'top_p': 0.95, 'ignore_eos': False} [repeated 7x across cluster]
Processing: 0%| | 0/54 [00:00<?, ?it/s]
(WorkerDict pid=55815) /home/chenshuai/miniconda3/envs/router-r1/lib/python3.9/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=55815) warnings.warn( [repeated 7x across cluster]

(main_task pid=53692) Connection error.
(main_task pid=53692) Retrying...
(main_task pid=53692) Connection error.
(main_task pid=53692) Retrying...
(main_task pid=53692) Connection error.
(main_task pid=53692) Retrying...

thanks! I try to use api. But maybe final errors. I want to ask if I use together.ai ,below is right?

"+api_base="https://api.together.xyz/v1" "

Thans for your reply!

[kawhiiiileo]

I get it. thx!!

[kawhiiiileo]

maybe another question~
If I train the model on 4× A100 (80GB), how long will it take?

[ViktorAxelsen]

It mainly depends on the dataset size and training hyperparameters (e.g., number of training steps). Under the default configuration, training typically takes around half a day on 4 x A6000