should the ProcessDataEntry and corresponding transformations be proper transformations?

[kashif]

Currently, in the dataset/common.py we have a bunch of transformations and I feel it might be better if they are proper transformations and moved to the corresponding location.

What do you think?

Thanks!

[jaheba]

My plan is to replace ProcessDataEntry with schemas, see #1000.

The idea would be to let each model decide what data it requires by providing a schema, which then is just mapped to the input data. We have started some work in #2001, but there a lot of details still to be figured out.

[kashif]

I love that each model has the same data format allowing me to train and compare different models without worrying about the data... I suppose that bit will remain via the schema?

[jaheba]

Yes and no.

Even now, models use different schemas and even a given model can use different schemas depending on the settings. E.g., when features are used the schema is different from a no-feature scenario. What the schema provides is to ensure that e.g. feat_dynamic_real must be present in the data if required, but would ignore it otherwise.

Currently, we handle these fields in ProcessDataEntry even if they are actually not needed.

We also have models which use different data formats, such as point processes. They are just not compatible but are currently not really well supported.

We still want to encourage commonality across models, such as a given dataset can be used with many models, but there is nothing truly enforcing this.

There was some discussion in #384 about whether we can have an intermediate layer which would translate a dataset into the required format on the fly.