Using external validation set to optimize TPOT?

[fcoppey]

I was wondering if there is a possibility to apply scoring an external X_test, y_test dataset instead of cross-validation in order to optimise TPOT Pipeline?

I'm thinking the way hypopt (https://pypi.org/project/hypopt/) does...

Would that be useful to anyone?

[weixuanfu]

This issue is related to issue #919. Please check a possible solution via a custom cv setting there.

TPOT does not provide this option by default since we think CV can help to avoid overfitting issue.

[Lamiane]

I have run into the same problem some time ago. The proposed solution with using custom cv works for me but I'd like to leave an argument for why this feature could be helpful.

Using cross-validation is a default choice when using small datasets. However:

• with big datasets it might be a better option to just have a train-validation-test split,
• some benchmark datasets define train-validation-test split and this has to be followed.

No ready way to use an external validation set made me decide not to use TPOT at all and I've spend quite some time looking for alternatives. I've found nothing more suitable than TPOT and ended up writing custom cv.

To wrap up: I believe this feature is useful and it's worth considering adding it to TPOT.