Run prediction is so slow.

[JiexingQi]

Does the prediction_output only process one example every time？This speed is too slow, and I wonder to know if you run on codalab, the time it will spend? I run on my own machine and it shows that it will use 6 hours to prediction. Is there a good way to speed it up, especially on codalab platform?

Thanks

[tscholak]

Hi @JiexingQi. Codalab has tight computational resources. I was not able to run the T5-3B model on the GPUs they provide. Instead I had to use CPU. 6 hours sounds about right.

[JiexingQi]

Thank you for your reply. @tscholak . I notice that the spider leadboard submission instruction(in https://worksheets.codalab.org/worksheets/0x82150f426cb94c17b861ef4162817399/) is

Attention

Please restrict your predict script's runtime to under two hours and up to two models per submission.

So, if your prediction in dev spend 6 hours and consider that the test set is twice as large as the dev set. It will spend more than 12 hours rather under two hours in the above instruction. Can you share some tricks for this spider submission on codalab. Thank you very much.

[tscholak]

Hi, I don't think you can make it much faster. I suggest you contact the organizers of the leaderboard and ask if they accept your submission despite the long runtime.

[JiexingQi]

Thank you very much @tscholak . By the way, I wonder the time spend for your T5-3B model with PICARD when you submit to codalab leadboard. Could you tell me about it? And the time for evaluate is much faster than run prediction_output. If you use evaluate rather than prediction_output for submission and why? Thank you for your kindly reply.

[JiexingQi]

Hi, @i just find that I run prediction_output on GPU, and it shows that this run will spend more than 6 hours,

and the config I use as follows:

{
    "model_path": "tscholak/cxmefzzi",
    "source_prefix": "",
    "schema_serialization_type": "peteshaw",
    "schema_serialization_randomized": false,
    "schema_serialization_with_db_id": true,
    "schema_serialization_with_db_content": true,
    "normalize_query": true,
    "target_with_db_id": true,
    "db_path": "/app/dataset_files/ori_dataset/spider/database",
    "cache_dir": "/transformers_cache",
    "num_beams": 4,
    "use_picard": true,
    "launch_picard": true,
    "picard_mode": "parse_with_guards",
    "picard_schedule": "incremental",
    "picard_max_tokens_to_check": 2,
    "device": 3,
    "conversational": false,
    "inputs_path":"/app/dataset_files/ori_dataset/spider/dev.json",
    "output_path": "/eval/predicted_sql.txt"
}

Is that normal? Looking forward to your reply. Thank you.

[tscholak]

Hi, No, that's not normal. What kind of device are you running this on?

[JiexingQi]

Thank you for reply, I run it on one 40G A100 GPU. And it get more and more slow than the start.

[tscholak]

Hm, that should be fast. If you are worried about the absence of batching in this script (which doesn't seem to matter for CPU), use run_seq2seq in eval mode instead.

[JiexingQi]

Thanks a lot, I just try eval mode. Thank you. @tscholak

[JiexingQi]

Hi, haoyang

I use eval mode to accelerate this process. @lihaoyang-ruc

[lihaoyang-ruc]

Thanks for your quick reply @JiexingQi .

I have tried the eval mode and still have this problem, can you tell me which version of PICARD (i.e., commit id) you are using? Next, I will try to use the same hardware and software environment as yours.

I am looking forward to your reply. Thank you.

[JiexingQi]

Hi, haoyang @lihaoyang-ruc

I release my code on https://github.com/LUMIA-Group/rasat and I use picard repo in this hash be27a96,

I hope this could help you