Should validation_data contain the full training_data into .train_model() just like how test_data is handled in GluonTS?

[e-alizadeh]

I see in GluonTS implementation that we have to pass the full dataset as test_data and the GluonTS will automatically split the test based on the prediction_length internally. Is the same thing applies when passing validation_data to .train() functions (e.g. in DeepAREstimator)?

Currently I'm doing the following:
e.g. We have 100 samples and with the prediction_length of 5.

The following is not a working example, but I'm using the list slicing to convey my thought here!

test_data = full_data
train_data = full_data[:90]
validation_data = full_data[90:95]

estimator = DeepAREstimator(prediction_length=5)
predictor_model = estimator.train_model(training_data=train_data, validation_data=validation_data)
forecast_gen, ts_actual_gen = make_evaluation_predictions(dataset=test_data, predictor=predictor_model.predictor)

Does above implementation makes sense?

[lostella]

Yes, that’s how validation datasets are used by default, currently: validation instances are sliced out of each series, where loss is computed on the last prediction_length observations. This behavior is controlled via “samplers”, whose default eg for DeepAR is defined at https://github.com/awslabs/gluon-ts/blob/49a5186475ad5d9ca8ee83935f32b564118f1908/src/gluonts/mx/model/deepar/_estimator.py#L278

[e-alizadeh]

@lostella Just to re-confirm, based on my implementation above, GluonTS expects the test data to include the whole dataset (train+validation+test) but for validation data it doesn't expect both training and validation? After posting my question, I thought my data slicing is actually wrong. I plotted my learning curve and I'm getting the training and validation losses on different order of magnitude. Below is a learning curve in case Validation dataset doesn't include the training data (train_data = full_data[:90] and validation_data = full_data[90:95])

So, I tried to include the training data in the validation dataset as well and below is the learning curve when validation dataset includes the training data ((train_data = full_data[:90] and
validation_data = full_data[:95])

Beside the different orders of magnitude for losses, I thought GluonTS API handles input data in a similar fashion both for test and validation. And just the fact that I'm getting the losses on the same level, I'm thinking this may be the right way!!

So to recap which of the following data slicing is correct?

1. Validation data excluding the training (my original question above but loss curve with different order of magnitude)

train_data = full_data[:90]
validation_data = full_data[90:95]
test_data = full_data

or
2. Validation data including the training

train_data = full_data[:90]
validation_data = full_data[:95]
test_data = full_data

[lostella]

Just to re-confirm, based on my implementation above, GluonTS expects the test data to include the whole dataset (train+validation+test) but for validation data it doesn't expect both training and validation?

That’s not the case. As you verified, you should providing the whole series, with both training and validation time range. Validation-only wouldn’t make sense, since the model wouldn’t be able to see the “input” (ie data prior to the validation time range)

[e-alizadeh]

Thanks a lot 🙏🏻

[ShirleyMgit]

@lostella can we give a different validation set that does not contain the training set, but its length >= context_length + prediction_length? and how should it be done in deepVAR? I have managed to do that in deepAR but not with deepVAR. thanks