SSL

Patterns - Dimitris

review Chowdhury et al.

Vision

SimCLR — A Simple Framework for Contrastive Learning of Visual Representations ICML 2020 | code

• contrastive learning with cropping and color as augmentations
• maximizes agreement between differently transformed views of the same sample via a contrastive cosine similarity loss in the latent space

BYOL - Bootstrap Your Own Latent A New Approach to Self-Supervised Learning NeurIPS 2020

• not contrastive, done without negative pairs
• iteratively update the representations, use exponential history mean network as the target

Barlow twins - Barlow twins: Self-supervised learning via redundancy reduction PMLR | code *

Audio

survey - Audio Self-supervised Learning: A Survey Patterns Review 2022 *

COLA - Contrastive learning of general-purpose audio representations ICASSP 2021 | Google | code

• same audio as positive pair, others in batch as negative

BYOL-Audio - BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation | code

• employed normalization and augmentations to modify BYOL for audio SSL
• might be useful: lots of parameters mentioned in the paper

CLAR - Contrastive Learning of Auditory Representations AISTATS 2021 code

• time-frequency audio features is better than 1D; no contrast between them
• semi-supervised: using supervised CE and contrastive learning CL simultaneously while training
• baseline: CE, SupCon, SimCLR

Health domain

Time series

Neighborhood-based

NCL - Neighborhood Contrastive Learning Applied to Online Patient Monitoring ICML 2021 | code

TNC - Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding ICLR 2021 | code

Self-Supervised Pretraining and Transfer Learning Enable Flu and COVID-19 Predictions in Small Mobile Sensing Datasets Conference on Health, Inference, and Learning (CHIL) 2023 | code data on request

• dataset: Homekit Flu Monitoring Study: 591k user-days of Fitbit data, 5196 participants, 6 months
• pretext task: same user, reconstruct data, predict Domain Inspired Features (best!)
• transfer task: infer positive / symptoms

Step2Heart

SelfHAR

CLOCS

Using additional information

Supervised contrastive learning NIPS2020 code google *

Contrastive learning of heart and lung sounds for label-efficient diagnosis Patterns code Stanford + Harvard

• 用metadata (sex, age, recording location)来进行positive和negative sample 的选择, supervised contrastive loss
• diagnosing heart and lung diseases through heart and lung sound recordings
• linear(representation evaluation) 和finetuning (initialization evaluation) 都evaluate了

Weakly Supervised Contrastive Learning ICCV2021 code

• first head: instance discrimination, infoNCE (NT-Xent) loss
• second head: weak label based on the connected component labeling process, supervised contrastive loss

combining language models / LLM

CLIP - Learning Transferable Visual Models From Natural Language Supervision ICLR2021 openAI open implementation

• 同时train text和vision encoder， 让同一对pair的相似 其他的不相似
• 问题：有false negative pairs

CLAP - Learning audio concepts from natural language supervision Microsoft ICASSP2023 code

• Zero-Shot, Frozen, Finetuning evaluation都做了
• downstream: sound event classification, speech emotion recognition