Deep Learning for Audio (DLA)

• Lecture and seminar materials for each week are in ./week* folders, see README.md for materials and instructions
• Any technical issues, ideas, bugs in course materials, contribution ideas - add an issue
• The current version of the course is conducted in autumn 2020 at the CS Faculty of HSE

Syllabus

• week01 Introduction to Digital Signal Processing

  • Lecture: Signals, Fourier transform, Spectrograms, MFCC and etc
  • Seminar: Intro in PyTorch, DevOps, R&D in Deep Learning

• week02 Automatic Speech Recognition I

  • Lecture: Metrics, Attention, LAS, CTC, BeamSearch
  • Seminar: Docker, W&B, Augmentations for Audio

• week03 Automatic Speech Recognition II

  • Lecture: LM Fusing, RNN Transducer, Schedule Sampling, BPE
  • Seminar: Jasper, QurtzNet, Mixed Precision Training, DDP/DP

• week04 Key-word spottind (KWS) and Voice Activity Detection (VAD)

  • Lecture: (DNN, CNN, RNN+Attention) based KWS, SVDF, Orthogonality Regularization and other Tricks
  • Seminar: Speeding Up NNs: Tensor Decomposition, Quantization, Pruning, Distilation and Architecture Design

• week05 Speaker verification and identification

  • Lecture: Metric Learning: Cosine, Contrastive, Triplet Losses. Angular Softmax. ArcFace
  • Seminar: Generalized End2End Loss for Speaker Verification

• week06 Text to Speech

  • Lecture: Tacotron, DeepVoice, GST, FastSpeech, Attention Tricks
  • Seminar: Location-Sensitive Attention

• week07 Neural Vocoders

  • Lecture: Introduction into generative models: AR, GAN, NF. WaveNet, ParallelWaveNet, WaveGlow, WaveFlow, MelGAN, PWG.

• week08 Voice Conversion

  • Lecture: AutoVC, ConVoice, TTS Skins, StarGAN-VC-1-2, CycleGAN-1-2-3, Blow

• week09 Music Generation

  • Lecture: VQVAE, Sparse Transformer, MuseNet, JukeBox

• week10 Speech Enhancement, Denoising and Speaker Diarization

  • Lecture: SEGAN, TF Masking, HiFi Denoising, Speaker Diarization, VAD

• week11 Self-supervision in Audio and Speech

  • Lecture: Intro to SS Learning. InfoNCE, CPC

Homeworks

• DSP Implementation of basic ops like FFT, Spectrogram and MelScale

• ASR Implementation of small ASR model, beam search and LM fusing

• KWS Implementation of attention based KWS model, streaming scoring and model distillation

• TTS Implementation of TTS model with different tricks

• NV Implementation of Neural Vocoder Model

Contributors & course staff

Course materials and teaching performed by

• Alexander Markovich
• Daniil Ivanov