Time Series Classification and Extrinsic Regression Papers

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

✨ News: This work has been accepted for publication in ACM Computing Survey .

Citation

If you find This Survey useful for your research, please consider citing this paper using the following information:

```
@article{Survey24TS,
author = {Foumani, Navid Mohammadi and Miller, Lynn and Tan, Chang Wei and Webb, Geoffrey I. and Forestier, Germain and Salehi, Mahsa},
title = {Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey},
year = {2024},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {0360-0300},
url = {https://doi.org/10.1145/3649448},
doi = {10.1145/3649448},
note = {Just Accepted},
journal = {ACM Comput. Surv.},
}
```

Table of Contents

• CNN Models TSC/TSER
• Attention-based Models TSC/TSER
• Graph Neural Network Models TSC/TSER
• Self_supervised Model

CNN Models

Abbreviation	Title	Venue	Year	Code
MC-DCNN	Time series classification using multi-channels deep convolutional neural networks	WAIM	2014	-
MC-CNN	Deep convolutional neural networks on multichannel time series for human activity recognition	IJCAI	2015	-
-	Convolutional neural networks for time series classification	J. Syst. Eng. Electron	2017	-
FCN	Time series classification from scratch with deep neural networks: A strong baseline (FCN, ResNet, Disjoint-CNN)	IJCNN	2017	code
Res-CNN	Integration of residual network and convolutional neural network along with various activation functions and global pooling for time series classification	Neurocomputing	2019	-
DCNNs	Multivariate Time Series Classification using Dilated Convolutional Neural Network	arXiv	2019	-
Disjoint-CNN	Disjoint-cnn for multivariate time series classification	ICDMW	2021	code
-	Encoding time series as images for visual inspection and classification using tiled convolutional neural networks	AAAI	2015	-
-	Classification of time-series images using deep convolutional neural networks	ICMV	2017	-
-	Scalable classification of univariate and multivariate time series	IEEE Big Data	2018	-
-	Classify multivariate time series by deep neural network image classification	CCHI	2019	-
RPMCNN	A deep learning framework for time series classification using relative position matrix and convolutional neural network	Neurocomputing	2019	-
-	Sensor classification using convolutional neural network by encoding multivariate time series as two-dimensional colored images	Sensors	2019	-
MCNN	Multi-scale convolutional neural networks for time series classification	arXiv	2016	-
t-LetNet	Data augmentation for time series classification using convolutional neural networks	AALTD	2016	-
MVCNN	Time series classification with multivariate convolutional neural network	IEEE Trans. Ind. Electron	2018	-
-	A cnn adapted to time series for the classification of supernovae	Electronic Imaging	2019	code
InceptionTime	Inceptiontime: Finding alexnet for time series classification	DMKD	2020	code
EEG-inception	Prototypical inception network with cross branch attention for time series classification	IJCNN	2021	-
Inception-FCN	Time Series Classification with InceptionFCN	Sensors	2021	-
KDCTime	KDCTime: Knowledge distillation with calibration on InceptionTime for time-series classification	J. Inf. Sci	2022	-
LITE	LITE: Light Inception with boosTing tEchniques for Time Series Classification	DSAA	2023	code

Attention-based Models

Abbreviation	Title	Venue	Year	Code
MuVAN	MuVAN: A Multi-view Attention Network for Multivariate Temporal Data	ICDM	2018	-
ChannelAtt	A novel channel-aware attention framework for multi-channel eeg seizure detection via multi-view deep learning	IEEE EMBS- BHI	2018	-
GeoMAN	Geoman: Multi-level attention networks for geo-sensory time series prediction	IJCAI	2018	code
Multi-Stage-Att	Multistage attention network for multivariate time series prediction	Neurocomputing	2020	-
CT_CAM	A novel channel and temporal-wise attention in convolutional networks for multivariate time series classification	IEEE Acesss	2020	-
CA-SFCN	A new attention mechanism to classify multivariate time series	IJCAI	2020	code
RTFN	RTFN: A Robust Temporal Feature Network for Time Series Classification	Information Sciences	2021	-
LAXCAT	Explainable Multivariate Time Series Classification: A Deep Neural NetworkWhich Learns To Attend To Important Variables AsWell As Informative Time Intervals	ACM- WSDM	2021	-
MACNN	Multi-scale attention convolutional neural network for time series classification	Neural Networks	2021	code
WHEN	WHEN: A Wavelet-DTW Hybrid Attention Network for Heterogeneous Time Series Analysis	ACM KDD	2023	-
SAnD	Attend and Diagnose: Clinical Time Series Analysis Using Attention Models	AAAI	2018	-
T2	Paying Attention to Astronomical Transients: Introducing the Time-series Transformer for Photometric Classification	arXiv	2021	code
GTN	Gated Transformer Networks for Multivariate Time Series Classification	arXiv	2021	code
TRANS_tf	An end-to-end framework combining time–frequency expert knowledge and modified transformer networks for vibration signal classification	Expert Syst. Appl.	2021	-
FMLA	Rethinking Attention Mechanism in Time Series Classification	arXiv	2022	-
AutoTransformer	AutoTransformer: Automatic Transformer Architecture Design for Time Series Classification	PAKDD	2022	-
ConvTran	Improving position encoding of transformers for multivariate time series classification	Data Min. Knowl. Discov	2023	code

Graph Neural Network

Abbreviation	Title	Venue	Year	Code
TGCN	Temporal Graph Convolutional Networks for Automatic Seizure Detection	PMLR	2019	-
DGCNN	EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks	IEEE Trans. Affect. Comput	2020	-
GraphSleepNet	GraphSleepNet: Adaptive Spatial-Temporal Graph Convolutional Networks for Sleep Stage Classification	IJCAI	2020	code
T-GCN	A deep learning approach using graph convolutional networks for slope deformation prediction based on time-series displacement data	Neural Computing and Applications	2021	-
MRF-GCN	Multireceptive Field Graph Convolutional Networks for Machine Fault Diagnosis	IEEE Trans. Ind. Electron	2021	-
-	Graph Convolutional Network For Generalized Epileptiform Abnormality Detection On EEG	SPMB	2021	-
DCRNN	Self-Supervised Graph Neural Networks for Improved Electroencephalographic Seizure Analysi	ICLR	2022	code
Time2Graph+	Time2Graph+: Bridging Time Series and Graph Representation Learning via Multiple Attentions	IEEE Trans. Knowl	2023	code
RAINDROP	Graph-Guided Network for Irregularly Sampled Multivariate Time Series	ICLR	2022	code
STEGON	Attentive Spatial Temporal Graph CNN for Land Cover Mapping From Multi Temporal Remote Sensing Data	IEEE Access	2021	-
-	A deep graph neural network architecture for modelling spatio-temporal dynamics in resting-state functional MRI data	Medical Image Analysis	2022	code
MTPool	Multivariate time-series classification with hierarchical variational graph pooling	Neural Networks	2022	code
SimTSC	Towards Similarity-Aware Time-Series Classification	SIAM- SDM	2022	code
-	Graph Neural Networks Extract High-Resolution Cultivated Land Maps From Sentinel-2 Image Series	IEEE Geosci	2022	-
C-DGAM	Class-driven Graph Attention Network for Multi-label Time Series Classification in Mobile Health Digital Twins	IEEE J. Sel. Areas Commun	2023	-
-	Graph Dynamic Earth Net: Spatio-Temporal Graph Benchmark for Satellite Image Time Series	IGARSS	2023	code
TISER-GCN	Graph neural networks for multivariate time series regression with application to seismic data	Int. j. data sci. anal	2023	code
TodyNet	TodyNet: Temporal Dynamic Graph Neural Network for Multivariate Time Series Classification	arXiv	2023	code
LB-SimTSC	LB-SimTSC: An Efficient Similarity-Aware Graph Neural Network for Semi-Supervised Time Series Classification	arXiv	2023	-

Self-supervised

Abbreviation	Title	Venue	Year	Code
TCL	Unsupervised feature extraction by time-contrastive learning and nonlinear ica	NIPS	2016	-
T-Loss/SRL	***Unsupervised Scalable Representation Learning for Multivariate Time Series	NeurIPS	2019	code
TNC	Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding	arXiv	2021	code
TS-TCC	Time-Series Representation Learning via Temporal and Contextual Contrasting	IJCAI	2021	code
MCL	Mixing up contrastive learning: Self-supervised representation learning for time series	Pattern Recognit Lett	2022	-
TimeCLR	TimeCLR: A self-supervised contrastive learning framework for univariate time series representation	Knowl.-Based Syst	2022	-
TS2Vec	TS2Vec: Towards Universal Representation of Time Series	AAAI	2022	code
BTSF	Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion	ICML	2022	-
TF-C	Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency	NeurIPS	2022	code
MHCCL	MHCCL: Masked Hierarchical Cluster-Wise Contrastive Learning for Multivariate Time Series	AAAI	2023	code
BENDR	BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data	Front Hum Neurosci	2021	code
Voice2Series	Voice2Series: Reprogramming Acoustic Models for Time Series Classification	PMLR	2021	code
TST	A Transformer-based Framework for Multivariate Time Series Representation Learning	ACM KDD	2021	-
TARNet	TARNet: Task-Aware Reconstruction for Time-Series Transformer	ACM KDD	2022	code
TimeMAE	TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders	arXiv	2023	code
CRT	Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer	IEEE Trans. Neural Netw. Learn. Syst	2023	-
PHIT	Finding Foundation Models for Time Series Classification with a PreText Task	arXiv	2023	code
Series2Vec	Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification	arXiv	2023	code

Application

Summary of Human Activity Recognition

Abbreviation	Title	Venue	Year	Code
-	Convolutional Neural Networks for human activity recognition using mobile sensors	ICST	2014	-
DCNN	Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks	ACM	2015	-
-	Deep convolutional neural networks on multichannel time series for human activity recognition	IJCAI	2015	-
DeepConvLSTM	Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition	Sensors	2016	-
DeepHAR	Deep, convolutional, and recurrent models for human activity recognition using wearables	IJCAI	2016	code
-	Human activity recognition with smartphone sensors using deep learning neural networks	Expert Syst. Appl.	2016	-
-	Ensembles of Deep LSTM Learners for Activity Recognition using Wearables	ACM Interact. Mob. Wearable Ubiquitous Technol.	2017	-
-	Human activity recognition from accelerometer data using Convolutional Neural Network	IEEE BigComp	2017	-
-	Deep Recurrent Neural Networks for Human Activity Recognition	Sensors	2017	-
-	Real-time human activity recognition from accelerometer data using Convolutional Neural Networks	Appl. Soft Comput.	2018	-
-	Convolutional Neural Networks for Human Activity Recognition Using Body-Worn Sensors	Informatics	2018	-
-	Efficient dense labelling of human activity sequences from wearables using fully convolutional networks	Pattern Recognit.	2018	-
-	Understanding and improving recurrent networks for human activity recognition by continuous attention	ACM-ISWC	2018	-
AttnSense	AttnSense: multi-level attention mechanism for multimodal human activity recognition	IJCAI	2019	-
InnoHAR	InnoHAR: A Deep Neural Network for Complex Human Activity Recognition	IEEE Access	2019	-
-	A Novel IoT-Perceptive Human Activity Recognition (HAR) Approach Using Multihead Convolutional Attention	IEEE Internet Things J.	2020	-
-	A multibranch CNN-BiLSTM model for human activity recognition using wearable sensor data	The Visual Computer	2021	-
CNN-biGRU	Deep Convolutional Neural Network with RNNs for Complex Activity Recognition Using Wrist-Worn Wearable Sensor Data	Electronics	2021	-
DEBONAIR	Deep learning based multimodal complex human activity recognition using wearable devices	Appl. Intell.	2020	-
-	LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes	Sensors	2021	-
-	Biometric User Identification Based on Human Activity Recognition Using Wearable Sensors: An Experiment Using Deep Learning Models	Electronics	2021	-
-	Sensor-Based Human Activity Recognition with Spatio-Temporal Deep Learning	Sensors	2021	-
-	Deep ConvLSTM With Self-Attention for Human Activity Decoding Using Wearable Sensors	IEEE Sens. J.	2021	-
-	Deep Convolutional Networks With Tunable Speed–Accuracy Tradeoff for Human Activity Recognition Using Wearables	IEEE Trans. Instrum. Meas.	2021	-
-	Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable Sensors	IEEE Trans. Instrum. Meas.	2022	code

Summary of Satellite Earth Observation

Abbreviation	Title	Venue	Year	Code
TAN	Temporal Attention Networks for Multitemporal Multisensor Crop Classification	IEEE Acess	2019	-
TGA	A temporal group attention approach for multitemporal multisensor crop classification	Infrared Phys. Technol.	2020	-
3D-CNN	3D Convolutional Neural Networks for Crop Classification with Multi-Temporal Remote Sensing Images	Remote Sensing	2018	-
DCM	DeepCropMapping: A multi-temporal deep learning approach with improved spatial generalizability for dynamic corn and soybean mapping	Remote Sens. Environ.	2020	code
HierbiLSTM	Multimodal Crop Type Classification Fusing Multi-Spectral Satellite Time Series with Farmers Crop Rotations and Local Crop Distribution	arXiv	2022	-
L-TAE	Lightweight Temporal Self-attention for Classifying Satellite Images Time Series	AALTD	2020	code
PSE-TAE	Satellite Image Time Series Classification With Pixel-Set Encoders and Temporal Self-Attention	CVPR	2020	code
SITS-BERT	Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification	IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.	2020	-
1D-CNN	End-to-end learning of deep spatio-temporal representations for satellite image time series classification	PKDD/ECML	2017	-
1D & 2D-CNNs	Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data	IEEE GRSL	2017	-
TempCNN	Temporal convolutional neural network for the classification of satellite image time series	Remote Sensing	2019	code
TASSEL	Attentive Weakly Supervised land cover mapping for object-based satellite image time series data with spatial interpretation	arXiv	2020	-
TSI	Time series remote sensing image classification framework using combination of deep learning and multiple classifiers system	Int. J. Appl. Earth Obs. Geoinf.	2021	-
TWINNS	Combining Sentinel-1 and Sentinel-2 Satellite Image Time Series for land cover mapping via a multi-source deep learning architecture	ISPRS J. Photogramm. Remote Sens	2019	-
DuPLO	DuPLO: A DUal view Point deep Learning architecture for time series classificatiOn	ISPRS J. Photogramm. Remote Sens	2019	-
Sequential RNN	Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders	ISPRS Int. J. Geo-Inf.	2018	code
FG-UNET	Land Cover Maps Production with High Resolution Satellite Image Time Series and Convolutional Neural Networks: Adaptations and Limits for Operational Systems	Remote Sensing	2019	-
LSTM	Land Cover Classification via Multitemporal Spatial Data by Deep Recurrent Neural Networks	IEEE GRSL	2017	-
HOb2sRNN	Object-Based Multi-Temporal and Multi-Source Land Cover Mapping Leveraging Hierarchical Class Relationships	Remote Sensing	2020	-
OD2RNN	Combining Sentinel-1 and Sentinel-2 Time Series via RNN for Object-Based Land Cover Classification	IEEE IGARSS	2019	-
SITS-Former	SITS-Former: A pre-trained spatio-spectral-temporal representation model for Sentinel-2 time series classification	Int. J. Appl. Earth Obs. Geoinf.	2022	code
-	Mapping Deforestation in Cerrado Based on Hybrid Deep Learning Architecture and Medium Spatial Resolution Satellite Time Series	Remote Sensing	2022	-
-	FLOOD DETECTION IN TIME SERIES OF OPTICAL AND SAR IMAGES	Int. Archives Photogrammetry, Remote Sens. & Spatial Inf. Sci.	2020	-
-	Classifying surface fuel types based on forest stand photographs and satellite time series using deep learning	Int. J. Appl. Earth Obs. Geoinf.	2022	-
-	Deep Neural Networks for automatic extraction of features in time series satellite images	Int. Archives Photogrammetry, Remote Sens. & Spatial Inf.	2020	-
-	Deep Recurrent Neural Networks for Winter Vegetation Quality Mapping via Multitemporal SAR Sentinel-1	IEEE GRSL	2018	-
TempCNN-LFMC	Live fuel moisture content estimation from MODIS: A deep learning approach	ISPRS J. Photogramm. Remote Sens.	2021	code
Multi-tempCNN	Multi-modal temporal CNNs for live fuel moisture content estimation	Environ. Model. Softw.	2022	code
LFMC estimation	SAR-enhanced mapping of live fuel moisture content	RSE	2020	code
LFMC estimation	Retrieval of Live Fuel Moisture Content Based on Multi-Source Remote Sensing Data and Ensemble Deep Learning Model	Remote Sensing	2022	-
MLDL-net	Multilevel Deep Learning Network for County-Level Corn Yield Estimation in the U.S. Corn Belt	IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.	2020	-
SSTNN	Crop yield prediction from multi-spectral, multi-temporal remotely sensed imagery using recurrent 3D convolutional neural networks	Int. J. Appl. Earth Obs. Geoinf.	2021	-
MMFVE	Combining LiDAR Metrics and Sentinel-2 Imagery to Estimate Basal Area and Wood Volume in Complex Forest Environment via Neural Networks	IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.	2022	-

Commonly used human activity recognition datasets

Dataset	Paper	Venue	Year	Repository
DAPHNet FoG	Potentials of Enhanced Context Awareness in Wearable Assistants for Parkinson's Disease Patients with the Freezing of Gait Syndrome	ISWC	2009	repo
Opportunity challenge	The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition	Pattern Recognit. Lett.	2013	repo
PAMAP2	Introducing a New Benchmarked Dataset for Activity Monitoring	ISWC	2012	repo
Skoda	Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection	EWSN	2008	repo
UCI-HAR	APublic Domain Dataset for Human Activity Recognition Using Smartphones	ESANN	2013	repo
UniMiB SHAR	UniMiB SHAR: A Dataset for Human Activity Recognition Using Acceleration Data from Smartphones	Applied Science	2017	repo
USC-HAD	USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensor	ACM UbiComp	2012	repo
WISDM	Activity recognition using cell phone accelerometers	ACM SIGKDD	2011	repo

Earth Observation satellites and instruments collecting the data used in the studies reviewed.

Agency	Satellite & Instruments	Title	Type	Code
NASA	Landsat-7 ETM	Landsat Satellite Missions	Optical	-
NASA	Landsat-8 OLI	Landsat Satellite Missions	Optical	-
NASA	Terra/Aqua MODIS	MODIS Moderate Resolution Imaging Spectrometer	Optical	-
ESA	Sentinel-1A/1B SAR-C	Sentinel Online	Microwave SAR	-
ESA	Sentinel-2A/2B MSI	Sentinel Online	Optical	-
CNES (France)	Pléiades-1A/1B HiRI	Earth Online	Optical	-
NSPO (Taiwan)	Formosat-2 RSI	FORMOSAT-2	Optical	-
CRESDA (China)	Gaofen-1/2 MUX, PAN, WFV	Gaofen2	Optical	-