Different results from allennlp tune and allennlp retrain with transformers

[MagiaSN]

When I am tuning a transformer model, I get different results from allennlp tune and allennlp retrain with the same hyperparameters.

I found this is caused by allennlp.common.cached_transformers module, which only constructs the model in the first trial (which would consume some random numbers), and uses the cached model in trials afterwards (which would not consume random numbers), leading to inconsistent results between tune and retrain.

[MagiaSN]

#46 fixes this for me, but it has to access allennlp private interfaces, is this acceptable?

[himkt]

@MagiaSN Thank you so much for the investigation! I think we have to fix it.
However, it would be better to clear the cache in the Optuna's AllenNLPExecutor, as allennlp-optuna is a simple wrapper of Optuna and AllenNLPExecutor (in Optuna) invokes AllenNLP functionalities.

Would you mind sending the PR to Optuna? I would review your PR if you could send it.
And could you please some small reproducible configuration? It would be really helpful.

[MagiaSN]

@himkt Fine, I have opened an issue in Optuna with reproducible scripts optuna/optuna#2716 and a PR optuna/optuna#2717

[MagiaSN]

Since this is fixed in optuna, I am closing this now :)