Simple Feed Forward Network predicts better than DeepAR

[MarleneBs]

I have got different Data and the SImple Feed Forward Network predicts better than DeepAR. Is this possible ? The timeseries I predict are Partly also very volatile.
Please help me, I am very confused.

[mbohlkeschneider]

Hi @MarleneBs,

Thank you for your question. This can for sure happen, some models perform better on some datasets, other models on other ones. Overall, SimpleFeedForward is not too shabby (see this paper).

[MarleneBs]

Could you maybe also check my Code? I am so afraid I have done something wrong, I need this for a Thesis and in some cases I got an NRMSE of 0.55 with DeepAR

[MarleneBs]

to_pandasdataset = lambda data: PandasDataset(
    data,
    target="value1",
    feat_dynamic_real=[feat1,feat2,feat3]
)
train = to_pandasdataset([ts.iloc[:-prediction_length, :] for ts in multiple_ts])
test = to_pandasdataset(multiple_ts)

In multiple_ts there is my data.
(I gave the same pandasdataset to the Feed Forward, is this correct?)

trainer = Trainer(epochs=e, num_batches_per_epoch= batch_per_ep)
from gluonts.transform.sampler import ValidationSplitSampler

estimator_with_features = DeepAREstimator(
        freq= '60min',
        prediction_length= prediction_length,
        context_length= 2*prediction_length, #hat vielleicht ein bisschen was gebracht ?
        dropoutcell_type= 'RNNZoneoutCell', # hat vielleicht ein bisschen was gebracht?
        trainer= trainer,
        use_feat_dynamic_real= True,
        num_layers= nl,
        num_cells= nc,
        cell_type= 'lstm',
        #dropoutcell_type='ZoneoutCell',
        use_feat_static_cat= False ,
        use_feat_static_real= False,
        impute_missing_values= True, #was passiert dann?
        cardinality= None,
        batch_size= batch,
        validation_sampler= ValidationSplitSampler(),

    )

predictor = estimator_with_features.train(train)

forecast_it, ts_it = make_evaluation_predictions(dataset=test, predictor=predictor,num_samples= 100)
forecasts = list(forecast_it)
tests = list(ts_it)
evaluator = Evaluator(quantiles=[0.1, 0.5, 0.8, 0.95])
agg_metrics, item_metrics = evaluator(tests, forecasts, num_series=len(test))

[MarleneBs]

And for the Feed Forward I used the exact same data and:

from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
feedfoward = SimpleFeedForwardEstimator(
  prediction_length=prediction_length,
  context_length=prediction_length,
 trainer=Trainer(epochs=100, num_batches_per_epoch= batch_per_ep),
  batch_size= batch
)
predictor= feedfoward.train(train)
forecast_it, ts_it = make_evaluation_predictions(dataset=test, predictor=predictor)
forecasts = list(forecast_it)
tests = list(ts_it)
evaluator = Evaluator(quantiles=[0.1, 0.5, 0.8, 0.95])
agg_metrics, item_metrics = evaluator(tests, forecasts, num_series=len(test))
Is this correct?

[MarleneBs]

And with every run there is a very different NRMSE with DeepAR

[Saifbenhmida]

you need to fix the seed and you will find the same NRMSE for every run