[Time-Series Prediction Platform 1.1 for PyTorch] Saving Inferenced predictions to CSV

[SephirothFFKH]

Related to Time-Series Prediction Platform 1.1 for PyTorch

Is your feature request related to a problem? Please describe.
Currently, the inference seems to only output Performance and Accuracy metrics from the model.

Describe the solution you'd like
It would be great to have a function to inference predictions, post-process them and save these actual prediction to a CSV file.

Describe alternatives you've considered
Alternatively, it could also be beneficial to have two functions, one to save the inference prediction without post-processing to CSV and another to optionally post-process the inferred predictions and save these to CSV.

Additional context
N/A

[IzzyPutterman]

Hey,
Thanks for the feature request. I am happy to tell you that the post-process prediction saving already exists inside the TSPP: https://github.com/NVIDIA/DeepLearningExamples/blob/master/Tools/PyTorch/TimeSeriesPredictionPlatform/evaluators/evaluator.py#L36

To activate you can set +evaluator.config.save_predictions=True in your launch_training.py or launch_inference.py commands. It should save in the hydra current working directory as a csv. Let us know if you run into any issues.

[SephirothFFKH]

Thank you so much! This is exactly what I was looking for and it works perfectly. I am currently experimenting with the "electricity" dataset and I am getting predictions.csv but, I am not sure how the formatting of these results should be interpreted in order to properly read these predictions. For example what row number vs "id" vs "t-x" vs "EstimatorY_t+x" represents. Once again, thank you

[IzzyPutterman]

Great to hear!

ID refers to the time-series ID that the data is from. The individual row numbers are the various sliding window samples that are passed through the model.
The "t-x" gives the history of that sliding window, so it basically includes the "encoder length of the target". For the electricity dataset you should see t-167 to t-0, since we have an encoder length of 168. So these values are not predicted, instead given.
The EstimatorY_t+x gives the predictions from the model. The t+x here refers to the xth prediction into the future. For electricity, we have an example length of 192 and encoder of 168, meaning the prediction horizon is 192-168=24. This means that you should see EstimatorY_t+0 to EstimatorY_t+23. The Estimators refer to the various outputs of a model. With our XGBoost model there will only be 1 estimator, however using the default configuration of the TFT for electricity yields 3 different quantile outputs for each time step, [0.1, 0.5, 0.9]. This means we have 3 estimators, where the Estimator0 refers to the 10th quantile predictions, Estimator1 refers to the 50th quantile predictions, and Estimator2 refers to the 90th quantile predictions.

Let us know about any questions or issues you run into.

[SephirothFFKH]

Thanks! I fully understand and it's now crystal clear. Using your explanation, I was able to see that prediction.csv row 53504, column t-167 to t0 matches electricity.csv column power_usage, row 2213808 to 2213975 as inputs therefore Estimator0,1,2 of row 53504 are the three quantile predictions inferencing what electricity.csv column 2213976 to 2213999 should be.

Since the inferenced values are predictions to test against existing electricity.csv values, what would be the way to go if I wanted to inference future values beyond 2213999 meaning prediction of values after the last know value in electricity.csv? Thank you very much

[IzzyPutterman]

Great to hear again!
To predict future values that are not known in any capacity is a little tricky. I think the best way to go about this would be to use our +inference.config.dataset_path=/path/to/data. https://github.com/NVIDIA/DeepLearningExamples/blob/master/Tools/PyTorch/TimeSeriesPredictionPlatform/inference/inference.py#L42

This config option allows you to input raw data (so before preprocessing) in the form of a csv. After inputting the data, it will preprocess it, treating it like a test set, and run inference on the data.

So in order to predict beyond the last time step, you would need to create/pad a dataset with values that you want to predict. You would probably need to look at the electricity.csv and see how the raw data is structured.

But for instance, for electricity, if we are talking about ID 0, we can take the last 168 values of it (in terms of time), then add 24 more rows with the relevant values. By this I mean:

1. Keep the static variables the same in these 24 steps.
2. Add the known values, like hour, and date in the correct way.
3. You can fill in the target/observed values with 0 or null depending on dtype.

It can be a little bit of a tricky process, but the main idea is to pad horizon length into the future and fill in the static/known values with their correct values and fill in the target/observed so they are not blank (but are not used).

Static/known/target/date/observed features are all defined in conf/dataset/electricity.yaml

Let me know if thats useful and if you have any questions. We are happy to help.
Inference will still output the metrics, which would probably be nonsense in this case (if we set the target to 0), but if you set the save_predictions option to True you should get the model predictions.

[SephirothFFKH]

Thank you very much once again! I would not have been able to get this far without your guidance. I was able to successfully obtain future predictions using the steps you provided. The results seem to be very good but, I do get a warning saying Warning: Please make sure the dataframe is preprocessed but I think preprocessing it still happening in the previous step, I have provided a snippet of the entire output from the inference below. Thank you very much.

root@docker-desktop:/workspace# python launch_inference.py checkpoint=/workspace/outputs/2022-09-08/19-02-19 +inference.config.dataset_path=/workspace/datasets/electricity_pad/electricity_inf.csv +evaluator.config.save_predictions=True

{'inference': {'_target_': 'inference.inference.run_inference', 'config': {'checkpoint': '???', 'batch_size': 64, 'precision': 'fp32', 'device': 'cuda', 'dataset_path': '/workspace/datasets/electricity_pad/electricity_inf.csv'}}, 'checkpoint': '/workspace/outputs/2022-09-08/19-02-19', 'evaluator': {'config': {'save_predictions': True}}}

Reading preprocessor state binary file: /workspace/datasets/electricity/tspp_preprocess.bin

Reading in data from CSV File: /workspace/datasets/electricity_pad/electricity_inf.csv

Sorting on time feature

Mapping nodes

Mapping categoricals to bounded range

Applying scalers

Fixing any nans in continuous features

Using backend: pytorch

Input DataFrame provided to create_datasets functions

Warning: Please make sure the dataframe is preprocessed

Evaluation Metrics: {'MAE': 21063.84961567209, 'RMSE': 21078.757344093774, 'SMAPE': 199.99999238215713, 'ND': 52580234.11841875}

[IzzyPutterman]

Great that you were able to get the inference going on this padded dataset. The warning you mention is from these lines here: https://github.com/NVIDIA/DeepLearningExamples/blob/master/Tools/PyTorch/TimeSeriesPredictionPlatform/data/datasets.py#L529 which is printed by default when a dataframe is used to create the dataloader instead of a path to a file.

As you state correctly, the preprocessing is being done in the previous steps, so you can ignore that print out. However do let us know if the saved outputs do look funky or incorrect.

Let us know about any other issues or questions you have. We will also gladly accept any feedback/criticism, we are continually working on improvements.

[SephirothFFKH]

Glad to know that this is the expected output in the console log. The output actually looks really good and and is very much what I would expect. I think you have completely solved all the issues I had with this. Thank you so much! I will now look into running my own dataset by following the steps in the README and I will definitely let you know if I need any help. Thank you