Added example for fine tuning BERT on Text Extraction task (SQuAD)

[apoorvnandan]

• Uses pretrained bert-base-uncased from HuggingFace and their tokenizers.
• Gets an exact match score of 79.4 after 2 epoch on TPU. (Paper shows 80.8, most likely due to a bunch of extra post-processing steps that I saw in the original implementation while converting the predicted token indexes back to the answer text)
• We will have to limit the data before training begins because this will take hours on CPU.

[fchollet]

Thank you for the PR. It looks great! Very useful example.

I fixed various minor nits. Other than that my comments deal with documentation improvements.

[fchollet]

Let's go with "BERT (from HuggingFace Transformers) for Text Extraction" -- in the future we will have BERT examples that don't use HuggingFace.

[apoorvnandan]

Oh okay. By the way I am very interested in writing examples of BERT and GPT with just tf.keras. Maybe just the pertaining part for example. We can discuss over over a separate issue if you think it would be a good addition.

[fchollet]

"HuggingFace Transformers"

[fchollet]

Please add a paragraph of text explaining that this example should preferably be run on the Colab TPU runtime.

[fchollet]

Rather than using x_eval from the outer scope, pass it to __init__ and set it as callback attribute

In general: data should be passed as argument, functions / classes can be fetched from the outer scope.

[fchollet]

Please add a docstring with a quick explanation of how the callback works

[fchollet]

Question-Answering

[fchollet]

Nit: please capitalize comments

[apoorvnandan]

Thanks for the comments. Making the required changes.

[fchollet]

LGTM, thank you. Please add the general files (or let me know if you want me to generate them instead).

[apoorvnandan]

I'll be running the add_example command on my laptop. So, I'll change

use_tpu = False

and add some lines like

# Remove these lines to train on the entire data
x_train  = [_[:10,:] for _ in x_train]
y_train  = [_[:10,:] for _ in y_train]
model.fit(x_train, y_train, ...)

Or is there a better way?

[fchollet]

How long does it take to run the example on a V100? (as an approximative estimate)

If less than ~20 min on a V100, I can run it on my side and generate the files

[apoorvnandan]

1 epoch took 1 hour 12 min on Colab GPU (K80 I think?).

Anyhow, I copied everything over to Colab, and ran python autogen.py add_example ... with TPU runtime. Generated files have been added.
I set epochs=1 for this.

[fchollet]

Please add a comment here that the recommend number of epochs is 3, not 1. You don't need to regenerate the files, you can juste edit the ipynb and md files directly to add the comment.

[fchollet]

LGTM, thank you! This is a very nice script, it will be valuable.