Transformation result keys names conflict

[Hari-pyt]

Hi,

I am having some issues with transformations

train_tf = transformation(iter(train_ds), is_train=True)

train = next(iter(train_tf))
[temp for temp in train.keys()]
**output**:
['start',
 'feat_static_cat',
 'source',
 'past_feat_dynamic_age',
 'future_feat_dynamic_age',
 'past_feat_dynamic_real',
 'future_feat_dynamic_real',
 'past_observed_values',
 'future_observed_values',
 'past_target',
 'future_target',
 'past_is_pad',
 'forecast_start']

After applying transformation, i sent the train_tf to estimator.train(train_tf)
but it is expecting the key 'target'

`KeyError                                  Traceback (most recent call last)
<ipython-input-92-f57d69ea49bf> in <module>
      1 estimator = DeepAREstimator(freq="4H",context_length = 48, prediction_length=24, distr_output = gluonts.distribution.GaussianOutput(), trainer=Trainer(epochs=25))
----> 2 predictor = estimator.train(training_data=train_tf)

~/.local/lib/python3.7/site-packages/gluonts/model/estimator.py in train(self, training_data)
    187     def train(self, training_data: Dataset) -> Predictor:
    188 
--> 189         training_transformation, trained_net = self.train_model(training_data)
    190 
    191         # ensure that the prediction network is created within the same MXNet

~/.local/lib/python3.7/site-packages/gluonts/model/estimator.py in train_model(self, training_data)
    180             net=trained_net,
    181             input_names=get_hybrid_forward_input_names(trained_net),
--> 182             train_iter=training_data_loader,
    183         )
    184 

~/.local/lib/python3.7/site-packages/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
    249 
    250                     with tqdm(train_iter) as it:
--> 251                         for batch_no, data_entry in enumerate(it, start=1):
    252                             if self.halt:
    253                                 break

/opt/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py in __iter__(self)
    977 """, fp_write=getattr(self.fp, 'write', sys.stderr.write))
    978 
--> 979             for obj in iterable:
    980                 yield obj
    981                 # Update and possibly print the progressbar.

~/.local/lib/python3.7/site-packages/gluonts/dataset/loader.py in __iter__(self)
    186         assert self._cur_iter is not None
    187         while True:
--> 188             data_entry = next(self._cur_iter)
    189             self._buffer.add(data_entry)
    190             if (

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    343         self, data_it: Iterator[DataEntry], is_train: bool
    344     ) -> Iterator:
--> 345         for data_entry in data_it:
    346             try:
    347                 for result in self.flatmap_transform(

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    299                 yield self.map_transform(data_entry.copy(), is_train)
    300             except Exception as e:
--> 301                 raise e
    302 
    303     @abc.abstractmethod

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    297         for data_entry in data_it:
    298             try:
--> 299                 yield self.map_transform(data_entry.copy(), is_train)
    300             except Exception as e:
    301                 raise e

~/.local/lib/python3.7/site-packages/gluonts/transform.py in map_transform(self, data, is_train)
    312 
    313     def map_transform(self, data: DataEntry, is_train: bool) -> DataEntry:
--> 314         return self.transform(data)
    315 
    316     @abc.abstractmethod

~/.local/lib/python3.7/site-packages/gluonts/transform.py in transform(self, data)
    437 
    438     def transform(self, data: DataEntry) -> DataEntry:
--> 439         value = data[self.field]
    440         if not isinstance(value, float):
    441             # this lines produces "ValueError: setting an array element with a

KeyError: 'target'

[jaheba]

It appears that your dataset has no column with the name target, which is required.

[Hari-pyt]

Thank you for reply jaheba,
It's obvious that my data don't have the name target.

Before applying the transformation my data have target but after applying the transformation it's getting splitted into past_target and future target which corresponds to context and prediction length

[lostella]

@hari7696 when you call estimator.train, the estimator-specific transformation is applied internally: there is no need to apply the transformation by hand.

What happens in your example is that you apply a transformation outside, the "target" field gets split into "past_target"/"future_target" and disappears, and this transformed data is once again passed through a transformation inside the call to estimator.train, which expects "target" to be there.

[Hari-pyt]

@lostella
Hi Lostella
Thanks for the reply
This is the estimator I am using
estimator = DeepAREstimator(freq="4H",context_length = 48, prediction_length=24, distr_output = gluonts.distribution.GaussianOutput(), trainer=Trainer(epochs=50))
As you said the estimator specific tranformation will be applied to the data.
But I cant find any arguments except time_features related to transformer in DeepAREstimator
refered doc link : http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-93/1/api/gluonts/gluonts.model.deepar.html

can you give me some info on customizing the estimator specific transformer, I would like to add observed_features, age, is_pad and time_features to my data.

Thanks

[lostella]

@hari7696 those features are automatically added by the transformation. In your example above, just do

predictor = estimator.train(train_ds)

without applying any transformation by hand to your data.

We will look into improving the documentation on how to correctly use estimators.

[Hari-pyt]

Got a picture after looking into source code of DeepAREstimator().

Thank you