BERT examples with/without HuggingFace?

[apoorvnandan]

Regarding examples around BERT, do we use HuggingFace transformers and tokenizers? Because then finetuning and downstream tasks can be shown in small examples like this.

# create model : Input > BERT > Dense
bert = TFBertModel.from_pretrained(config.MODEL_PATH, config=BertConfig())
input_ids = layers.Input(shape=(config.maxlen,), dtype=tf.int32)
sequence_output = bert(input_ids)[0][:, 0, :]
out = layers.Dense(1, activation="sigmoid")(sequence_output)
classifier = models.Model(inputs=input_ids, outputs=out)

classifier.compile(
    optimizers.Adam(lr=3e-5), loss="binary_crossentropy", metrics=[metrics.AUC()]
)
train_history = classifier.fit(train_dataset, steps_per_epoch=150, epochs=30)

And then we can show that doing the same thing on TPU is as easy as adding

tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)
strategy = tf.distribute.experimental.TPUStrategy(tpu)

and creating model under strategy.scope().

Does this feel like a good example?

[fchollet]

Yes, I think demonstrating the use of the HuggingFace Transformers library would be useful. What task/dataset were you thinking of applying it to?

[apoorvnandan]

I was thinking about the Question Answering task. SQuAD for data.

[apoorvnandan]

Created a PR. Closing this issue.
#46