MFND³R: Multivariate Forecasting Network with Deep Data-Driven Reconstruction for Engineering Application

This is the origin Pytorch implementation of MFND³R in the following paper: [MFND³R: Multivariate Forecasting Network with Deep Data-Driven Reconstruction for Engineering Application](Manuscript submitted to EAAI).

The Entire Architecture of MFND³R

Figure 1. The entire architecture of MFND³R.

Requirements

• Python 3.8.8
• matplotlib == 3.3.4
• numpy == 1.20.1
• pandas == 1.2.4
• scipy == 1.9.0
• scikit_learn == 0.24.1
• torch == 1.11.0

Dependencies can be installed using the following command:

pip install -r requirements.txt

Data

ETT, ECL and weather dataset were acquired at: here. Solar and PEMS dataset were acquired at: here. M4 dataset was acquired at: M4. Ship dataset was in ./data/Ship/ and its raw data was in https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2022/index.html.

Data Preparation

After you acquire raw data of all datasets, please separately place them in corresponding folders at ./MFND3R/data.

We place ETT in the folder ./ETT-data, ECL in the folder ./electricity and weather in the folder ./weather of here (the folder tree in the link is shown as below) into folder ./data and rename them from ./ETT-data,./electricity and ./weather to ./ETT, ./ECL and./weather respectively. We rename the file of ECL from electricity.csv to ECL.csv and rename its last variable from OT to original MT_321.

|-autoformer
| |-ETT-data
| | |-ETTh1.csv
| | |-ETTh2.csv
| | |-ETTm1.csv
| | |-ETTm2.csv
| |
| |-electricity
| | |-electricity.csv
| |
| |-weather
| | |-weather.csv

We place Solar in the folder ./financial and PEMS in the folder ./PEMS of here (the folder tree in the link is shown as below) into the folder ./data and rename them as ./Solar and ./PEMS respectively.

|-dataset
| |-financial
| | |-solar_AL.txt
| |
| |-PEMS
| | |-PEMS08.npz

As for M4 dataset, we place the folders ./Dataset and ./Point Forecasts of M4 (the folder tree in the link is shown as below) into the folder ./data/M4. Moreover, we unzip the file ./Point Forecasts/submission-Naive2.rar to the current directory.

|-M4-methods
| |-Dataset
| | |-Test
| | | |-Daily-test.csv
| | | |-Hourly-test.csv
| | | |-Monthly-test.csv
| | | |-Quarterly-test.csv
| | | |-Weekly-test.csv
| | | |-Yearly-test.csv
| | |-Train
| | | |-Daily-train.csv
| | | |-Hourly-train.csv
| | | |-Monthly-train.csv
| | | |-Quarterly-train.csv
| | | |-Weekly-train.csv
| | | |-Yearly-train.csv
| | |-M4-info.csv
| |-Point Forecasts
| | |-submission-Naive2.rar

Then you can get the folder tree shown as below:

|-data
| |-ECL
| | |-ECL.csv
| |
| |-ETT
| | |-ETTh1.csv
| | |-ETTh2.csv
| | |-ETTm1.csv
| | |-ETTm2.csv
| |
| |-M4
| | |-Dataset
| | | |-Test
| | | | |-Daily-test.csv
| | | | |-Hourly-test.csv
| | | | |-Monthly-test.csv
| | | | |-Quarterly-test.csv
| | | | |-Weekly-test.csv
| | | | |-Yearly-test.csv
| | | |-Train
| | | | |-Daily-train.csv
| | | | |-Hourly-train.csv
| | | | |-Monthly-train.csv
| | | | |-Quarterly-train.csv
| | | | |-Weekly-train.csv
| | | | |-Yearly-train.csv
| | | |-M4-info.csv
| | |-Point Forecasts
| | | |-submission-Naive2.csv
| |
| |-PEMS
| | |-PEMS08.npz
| |
| |-Ship
| | |-Ship1.csv
| | |-Ship2.csv
| |
| |-Solar
| | |-solar_AL.txt
| |
| |-weather
| | |-weather.csv

Baseline

We select eight typical deep time series forecasting models based on CNN (SCINet), RNN (LSTNet), GNN (GTA), Transformer (Non-stationary Transformer, PatchTST, Crossformer) and Perceptron/MLP (DLinear, FiLM) as baselines in multivariate forecasting experiments. Two traditional time series forecasting models {ARIMA, Simple Exponential Smooth (SES)} and two deep forecasting model (N-HiTS, discover_PLF) are chosen as additional baselines only when handling the univariate dataset M4. Their source codes origins are given below:

Baseline	Source Code
Non-stationary Transformer	https://github.com/thuml/Nonstationary_Transformers
PatchTST	https://github.com/yuqinie98/PatchTST
Crossformer	https://github.com/Thinklab-SJTU/Crossformer
SCINet	https://github.com/cure-lab/SCINet
LSTNet	https://github.com/laiguokun/LSTNet
discover_PLF	https://github.com/houjingyi-ustb/discover_PLF
DLinear	https://github.com/cure-lab/LTSF-Linear
N-HiTS	https://github.com/cchallu/n-hits
FiLM	https://github.com/tianzhou2011/FiLM
GTA	https://github.com/ZEKAICHEN/GTA

Moreover, the default experiment settings/parameters of aforementioned ten baselines are given below respectively:

Baselines	Settings/Parameters name	Descriptions	Default mechanisms/values
Non-stationary Transformer	d_model	The number of hidden dimensions	512
	d_ff	Dimension of fcn	2048
	n_heads	The number of heads in multi-head attention mechanism	8
	e_layers	The number of encoder layers	2
	d_layers	The number of decoder layers	1
	p_hidden_dims	Hidden layer dimensions of projector (List)	[128, 128]
	p_hidden_layers	The number of hidden layers in projector	2
PatchTST	patch_len	Patch length	16
	stride	The stride length	8
discover_PLF	hidden_size	The number of hidden dimensions	128
	num_layers	The number of decoder layers	1
Crossformer	seq_len	Segment length (L_seq)	6
	d_model	The number of hidden dimensions	64
	d_ff	Dimension of fcn	128
	n_heads	The number of heads in multi-head attention mechanism	2
	e_layers	The number of encoder layers	2
SCINet	hidden-size	The number of hidden dimensions	8
	levels	SCINet block levels	3
	stacks	The number of SCINet blocks	1
LSTNet	hidCNN	The number of CNN hidden units	100
	hidRNN	The number of RNN hidden units	100
	window	Window size	168
	CNN_kernel	The kernel size of the CNN layers	6
	hidSkip	The skip-length Recurrent-skip layer	24
DLinear	moving_avg	The window size of moving average	25
N-HiTS	n_pool_kernel_size	Pooling kernel size	[4, 4, 4]
	n_blocks	The number of blocks in stacks	[1, 1, 1]
	n_x_hidden	Coefficients hidden dimensions	512
	n_freq_downsample	The number of stacks' coefficients	[64, 8, 1]
FiLM	d_model	The number of hidden dimensions	512
	d_ff	Dimension of fcn	2048
	n_heads	The number of heads in multi-head attention mechanism	8
	e_layers	The number of encoder layers	2
	d_layers	The number of decoder layers	1
	modes1	The number of Fourier modes to multiply	32
GTA	d_model	The number of hidden dimensions	512
	d_ff	Dimension of fcn	2048
	n_heads	The number of heads in multi-head attention mechanism	8
	e_layers	The number of encoder layers	2
	d_layers	The number of decoder layers	1

Usage

Commands for training and testing MFND³R of all datasets are in ./scripts/MFND3R.sh.

More parameter information please refer to main.py.

We provide a complete command for training and testing MFND³R:

python -u main.py --model <model> --mode <mode> --data <data> --root_path <root_path> --features <features> --input_len <input_len> --pred_len <pred_len> --ODA_layers <ODA_layers> --VRCA_layers <VRCA_layers> --d_model <d_model> --learning_rate <learning_rate> --dropout <dropout> --batch_size <batch_size> --use_RevIN --train_epochs <train_epochs> --patience <patience> --itr <itr>

Here we provide a more detailed and complete command description for training and testing the model:

Parameter name	Description of parameter
model	The model of experiment. This can be set to MFND3R
mode	Forecasting format
data	The dataset name
root_path	The root path of the data file
data_path	The data file name
features	The forecasting task. This can be set to M,S (M : multivariate forecasting, S : univariate forecasting
target	Target feature in S task
checkpoints	Location of model checkpoints
input_len	Input sequence length
pred_len	Prediction sequence length
enc_in	Input size
c_out	Output size
d_model	Dimension of model
representation	Representation dims in the end of the intra-reconstruction phase
dropout	Dropout
ODA_layers	The number of ODA layers
VRCA_layers	The number of VRCA layers
alpha	The significant level of Cucconi test
itr	Experiments times
train_epochs	Train epochs of the second stage
batch_size	The batch size of training input data in the second stage
patience	Early stopping patience
learning_rate	Optimizer learning rate
loss	Loss function
use_RevIN	Whether to use RevIN

Results

The experiment parameters of each data set are formated in the MFND3R.sh files in the directory ./scripts/. You can refer to these parameters for experiments, and you can also adjust the parameters to obtain better mse and mae results or draw better prediction figures.

Figure 2. Multivariate forecasting results

Figure 3. Univariate forecasting results

Contact

If you have any questions, feel free to contact Li Shen through Email (shenli@buaa.edu.cn) or Github issues. Pull requests are highly welcomed!