Reproducibility of results

[jurader]

Hi.

I have several questions about reproducibility of the code.

1 Pretraining part1

python run_pretraining.py \
  --input_file=../preprocessed_data/tf_examples_128.tfrecord \
  --output_dir=../output_pretrained/pretraining_output_128 \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=bert_config.json \
  --init_checkpoint=bert_model.ckpt \
  --train_batch_size=64 \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5

This command produced the following result (../output_pretrained/pretraining_output_128/eval_results.txt).
Is this result acceptable?

global_step = 100000
loss = 1.0482247
masked_lm_accuracy = 0.79015565
masked_lm_loss = 0.90652496
next_sentence_accuracy = 0.9425
next_sentence_loss = 0.14315148

2 Pretraining part2

python run_pretraining.py \
  --input_file=../preprocessed_data/tf_examples_512.tfrecord \
  --output_dir=../output_pretrained/pretraining_output_512 \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=bert_config.json \
  --init_checkpoint=../output_pretrained//pretraining_output_128/model.ckpt-100000 \
  --train_batch_size=16 \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5

This command produced the following result (../output_pretrained/pretraining_output_512/eval_results.txt).
Is this result acceptable?

global_step = 100000
loss = 0.64307
masked_lm_accuracy = 0.85223603
masked_lm_loss = 0.6104435
next_sentence_accuracy = 0.99
next_sentence_loss = 0.029400254

3 Early Notes Prediction

python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_eval \
  --data_dir ./data/3days/ \
  --bert_model ./model/early_readmission/ \
  --max_seq_length 512 \
  --output_dir ./result_early

"./data/3days/" was obtained by [preprocess.py] (https://github.com/kexinhuang12345/clinicalBERT/blob/master/preprocess.py).
"./model/early_readmission/" was obtained from the link of ClinicalBERT Weights.

This command produced the following result.
Is this result acceptable?

08/29/2022 08:17:52 - INFO - main - ***** Eval results *****
08/29/2022 08:17:52 - INFO - main - RP80 = 0.596958174904943
08/29/2022 08:17:52 - INFO - main - eval_accuracy = 0.7395699319977945
08/29/2022 08:17:52 - INFO - main - eval_loss = 0.4970936025972028
08/29/2022 08:17:52 - INFO - main - global_step = 0
08/29/2022 08:17:52 - INFO - main - training loss = 100000.0
Recall at Precision of 80 is {} 0.596958174904943

4 Training readmission prediction model from pretraining ClinicalBERT

python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_train \
  --do_eval \
  --data_dir ./data/3days/ \
  --bert_model ./model/pretraining \
  --max_seq_length 512 \
  --train_batch_size 16 \
  --learning_rate 2e-5 \
  --num_train_epochs 1 \
  --output_dir ./result_finetuning

"./data/3days/" was obtained by preprocess.py.
"./model/pretraining" was obtained from the link of ClinicalBERT Weights.

This command produced the following result.
For me, this result is not acceptable.
If there is something wrong, please let me know the way to finetune the model.

08/29/2022 08:43:39 - INFO - main - ***** Eval results *****
08/29/2022 08:43:39 - INFO - main - RP80 = 0.12547528517110265
08/29/2022 08:43:39 - INFO - main - eval_accuracy = 0.6173497518838449
08/29/2022 08:43:39 - INFO - main - eval_loss = 0.6674913585799357
08/29/2022 08:43:39 - INFO - main - global_step = 2988
08/29/2022 08:43:39 - INFO - main - training loss = 0.6257770110864237
Recall at Precision of 80 is {} 0.12547528517110265