5. Advanced Techniques

We used Huggingface pertained transformer models (Roberta-base-cased) and added customized question-answer head layers using TensorFlow to reach a better question answering result. For more details, please refer to this notebook.

Why would we use a TensorFlow approach over a SimpleTransformer alternative?

• TensorFlow supports more customized functions including a loss function, more question-answering structures, number of inputs, and k-fold techniques.
• TensorFlow can help you understand deeper how transformers work by creating an attention mask, input ids, and padding your train set.
• Training time takes longer for this approach as we are using the k-fold method to avoid overfitting