Can I train question-answering on TPU using Huggingface

[thak123]

❓ Questions & Help

I am trying to run run_squad from question answering on TPU but I get the following error.

Details

export SQUAD_DIR=/path/to/SQUAD
python examples/xla_spawn.py --num_cores 8
run_squad.py
--model_type bert
--model_name_or_path bert-base-uncased
--do_train
--do_eval
--do_lower_case
--train_file $SQUAD_DIR/train-v1.1.json
--predict_file $SQUAD_DIR/dev-v1.1.json
--per_gpu_train_batch_size 12
--learning_rate 3e-5
--num_train_epochs 2.0
--max_seq_length 384
--doc_stride 128
--output_dir /tmp/debug_squad/

run_squad.py: error: unrecognized arguments: --tpu_num_cores 1

[julien-c]

as you can see in https://github.com/huggingface/transformers/tree/master/examples#the-big-table-of-tasks question-answering is not implemented using Trainer, yet – so doesn't have TPU support.

This is on our todo-list but might take a few weeks. Feel free to open a PR though – or use TFTrainer is TF is an option

[thak123]

I think I can live with TFTrainer ... Thanks for the reply

[thak123]

how do i use TFTrainer with TPU

[LysandreJik]

Hi! The running on TPU section of the examples covers that.

Basically, if on a correctly setup TPU, the TF trainer will automatically launch on TPU.