Human-computer Interaction for Brain-inspired Computing Based on Machine Learning And Deep Learning: A Review

This is the repository of HCI for brain-inspired-computing which offers a thorough review of the current state of research concerning the application of brain-to-text(speech) models.

Human-computer Interaction for Brain-inspired Computing Based on Machine Learning And Deep Learning: A Review
Paper

Feel free to contact us or pull requests if you find any related papers that are not included here.

Abstract

The continuous development of artificial intelligence has a profound impact on biomedical research and other fields.Brain-inspired computing is an important intersection of multimodal technology and biomedical field. This paper presents a comprehensive review of machine learning (ML) and deep learning (DL) models applied in human-computer interaction for brain-inspired computing, tracking their evolution, application value, challenges, and potential research trajectories. First, the basic concepts and development history are reviewed, and their evolution is divided into two stages: recent machine learning and current deep learning, emphasizing the importance of each stage in the research state of human-computer interaction for brain-inspired computing. In addition, the latest progress and key techniques of deep learning in different tasks of human-computer interaction for brain-inspired computing are introduced from six perspectives. Despite significant progress, challenges remain in making full use of its capabilities. This paper aims to provide a comprehensive review of human-computer interaction for brain-inspired computing models based on machine learning and deep learning, highlighting their potential in various applications and providing a valuable reference for future academic research.

Citation

If you find our work useful in your research, please consider citing:

@misc{yu2024humancomputer,
      title={Human-computer Interaction for Brain-inspired Computing Based on Machine Learning And Deep Learning:A Review}, 
      author={Bihui Yu and Sibo Zhang and Lili Zhou and Jingxuan Wei and Linzhuang Sun and Liping Bu},
      year={2024},
      eprint={2312.07213},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Menu

• Dataset
• EEG-To-Text
• fMRI-To-Text
• MEG-To-Text
• ECoG-To-Text

Dataset

Paper	Published in
ZuCo, A Simultaneous EEG and Eye-tracking Resource for Natural Sentence Reading	Scientific Data 2018
ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation	LREC 2020
Predicting Human Brain Activity Associated with The Meanings of Nouns	Science 2008
Toward A Universal Decoder of Linguistic Meaning from Brain Activation	Nature Communications 2018

EEG-To-Text

Paper	Published in
State-of-the-art speech recognition using eeg and towards decoding of speech spectrum from eeg	Arxiv 2019
Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech EEG	Sensors 2020
EEG-transformer: Self-attention from transformer architecture for decoding EEG of imagined speech	IEEE BCI 2022
Open Vocabulary Electroencephalography-to-text Decoding and Zero-shot Sentiment Classification	AAAI 2022
Aligning Semantic in Brain and Language: A Curriculum Contrastive Method for Electroencephalography-to-Text Generation	IEEE TNSRE 2023
DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation	NeurIPS 2023

fMRI-To-Text

Paper	Published in
Incorporating context into language encoding models for fMRI	NeurIPS 2018
Brain2Word: Improving Brain Decoding Methods and Evaluation	NeurIPS 2020
Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech	NeurIPS 2020
Neural decoding of speech with semantic-based classification	Cortex 2022
Toward a realistic model of speech processing in the brain with self-supervised learning	NeurIPS 2022
Cross-Modal Cloze Task: A New Task to Brain-to-Word Decoding	ACL 2022
UniCoRN: Unified Cognitive Signal Reconstruction Bridging Cognitive Signals and Human Language	ACL 2023
Semantic Reconstruction of Continuous Language from Non-invasive Brain Recordings	Nature Neuroscience 2023

MEG-To-Text

Paper	Published in
Automatic speech activity recognition from MEG signals using seq2seq learning	IEEE NER 2019
Decoding Speech from Single Trial MEG Signals Using Convolutional Neural Networks and Transfer Learning	EMBC 2019
Decoding imagined and spoken phrases from non-invasive neural (MEG) signals	Frontiers in Neuroscience 2020
MEG sensor selection for neural speech decoding	IEEE Access 2020
Decoding Speech Perception from Non-invasive Brain Recordings	Nature Machine Intelligence 2023

ECoG-To-Text

Paper	Published in
Machine Translation of Corticalcactivity to Text with An Encoder-decoder Framework	Nature Neuroscience 2020
Brain2char: a deep architecture for decoding text from brain recordings	Neural Engineering 2020
A Neural Speech Decoding Framework Leveraging Deep Learning and Speech Synthesis	BioRxiv 2023
Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models	Neural Engineering 2023
Synthesizing Speech from ECoG with A Combination of Transformer-based Encoder and Neural Vocoder	ICASSP 2023