Skip to content

Add attribution for the BERT modeling code #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 28, 2022
Merged

Add attribution for the BERT modeling code #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b8f8ce6
Add attribution for the BERT modeling code
mattdangerw Apr 28, 2022
357550e
typo fix
mattdangerw Apr 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion examples/bert/bert_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@
# limitations under the License.
"""Bert model and layer implementations.

We should work to replace this with components from the keras-nlp library.
These components come from the tensorflow official model repository for BERT:
https://github.com/tensorflow/models/tree/master/official/nlp/modeling

This is to get us into a testable state. We should work to replace all of these
components with components from the keras-nlp library.
"""

import tensorflow as tf
Expand Down
3 changes: 3 additions & 0 deletions examples/bert/create_pretraining_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@
sentence pairs, as will the original paper. So a 20gb source of wikipedia and
bookscorpus will result in a 400gb dataset.

This script is adapted from the original BERT respository:
https://github.com/google-research/bert/blob/master/create_pretraining_data.py

Usage:
python create_pretraining_data.py \
--input_files ~/datasets/bert-sentence-split-data/shard_0.txt \
Expand Down