[ALBERT]Has anyone reproduced ALBERT a scores on GLUE dataset?

[lonePatient]

I convert tf weight to pytorch weight ,and on QQP dataset, I only get 87% accuracy.

model: albert-base
epochs: 3
learning_rate; 2e-5
batch size: 24
max sequence length: 128
warmup_proportion: 0.1

[kamalkraj]

https://github.com/kamalkraj/ALBERT-TF2.0 [WIP]
got better accuracy on dev set CoLA.

[wxp16]

I convert tf weight to pytorch weight ,and on QQP dataset, I only get 87% accuracy.

model: albert-base
epochs: 3
learning_rate; 2e-5
batch size: 24
max sequence length: 128
warmup_proportion: 0.1

On the MNLI dataset, using the 'ALBERT' base v1, I got the following results. Clearly, the accuracy is very low.

eval_accuracy = 0.77962303
eval_loss = 0.5517804
global_step = 24543
loss = 0.5517709

[kamalkraj]

https://github.com/kamalkraj/ALBERT-TF2.0 [WIP]
got better accuracy on dev set CoLA.

Dataset: MNLI
Model: ALBERT large v1
Dev accuracy : 0.8089
epochs : 3
max_seq_length : 128
batch_size: 128
learning_rate : 3e-5

[lonePatient]

https://github.com/lonePatient/albert_pytorch

Dataset: MNLI
Model: ALBERT_BASE_V2
Dev accuracy : 0.8418

[kamalkraj]

@lonePatient
Could your share the Hyperparameters?
Max seq length ?

[lonePatient]

@kamalkraj
--max_seq_length=128
--per_gpu_train_batch_size=16
--per_gpu_eval_batch_size=16
--spm_model_file=${BERT_BASE_DIR}/30k-clean.model
--learning_rate=1e-5
--num_train_epochs=3.0
--logging_steps=24544
--save_steps=24544 \

[kamalkraj]

@lonePatient
Dropouts ? All 0 ?

[lonePatient]

@kamalkraj 。fine-tuning， dropout rate=0.1