How is the default validation split done?

[vamp-ire-tap]

Can someone answer whether the validation dataset loader (base loader) samples randomly from the training dataset that is it does: in-sample validation or does it do out-of-sample validation? In the DeepAR paper it is written: "we fit our model on the first partition of the training set containing 90% of the data and we pick the one that has the minimal negative log-likelihood on the remaining 10%." that means in-sample validation.

[lostella]

Disclaimer: the code currently in GluonTS is not the same that was used for the DeepAR paper. How data was sampled for training/validation purposes in the paper might differ from what is done by default in the DeepAREstimator.

By default, the validation data loader in the DeepAREstimator is defined so that it slices the final windows from each time series in the validation set (i.e., with the prediction interval aligned with the trailing prediction_length observations of such time series). This can be seen by looking at the default "sampler" object

https://github.com/awslabs/gluon-ts/blob/94b916e5daa0bfe5ed0feaf6e8987cfe3f6e4974/src/gluonts/model/deepar/_estimator.py#L278-L282

that is later used to configure the data loader

https://github.com/awslabs/gluon-ts/blob/94b916e5daa0bfe5ed0feaf6e8987cfe3f6e4974/src/gluonts/model/deepar/_estimator.py#L373-L395

What is provided as "validation dataset" is up to the user: one can provide completely different time series other than the ones used for training, or the same at a later point in time (i.e., with additional data, so that validation scores are computed "after" the training data).

[vamp-ire-tap]

thanks alot!