This is an extensive and continuously updated compilation of graph SSL literature categorized by the knowledge-based taxonomy, proposed by our paper 📄A Survey on Self-Supervised Pre-Training of Graph Foundation Models: A Knowledge-Based Perspective. Here every pretext of each paper is listed and briefly explained. You can find all pretexts and their corresponding papers with detailed metadata below, including additional pretexts and literature not listed in our paper.
A kind reminder: to search for a certain paper, type the title or the abbreviation of the proposed method (recommended) into the browser search bar (Ctrl + F). Some papers fall under multiple sections.
- Relevant surveys and empirical studies
- Microscopic pretexts
- Macroscopic pretexts
- Task generalization strategies
Note: 🕸️ graph-related; 🤖 LLM-related; 🔬 empirical study
Node features
- Feature prediction: to predict the original node features by decoding low-dimensional representations
- Feature denoising: to add (generally continuous, e.g. isotropic Gaussian) noises to the original features and try to reconstruct them
- Masked feature prediction: a special, discrete case of feature denoising, which predicts the original features of masked nodes by representations of unmasked ones. It is "autoregressive" if the predicted nodes are generated one-by-one
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
MGAE: Marginalized Graph Autoencoder for Graph Clustering | CIKM'17 | Feature prediction | Graph partitioning | link |
Symmetric Graph Convolutional Autoencoder for Unsupervised Graph Representation Learning (GALA) | ICCV'19 | Feature prediction | Node clustering; link prediction; image clustering | link |
Strategies for Pre-training Graph Neural Networks (AttrMask) | ICLR'20 | Masked feature prediction | Graph classification; biological function prediction | link |
Graph Representation Learning via Graphical Mutual Information Maximization (GMI) | WWW'20 | Feature prediction (JS) | Node classification; link prediction | link |
When Does Self-Supervision Help Graph Convolutional Networks? (GraphComp) | ICML'20 | Masked feature prediction | Node classification | link |
GPT-GNN: Generative Pre-Training of Graph Neural Networks | KDD'20 | Masked feature prediction (autoregressive) | Node classification; edge regression (recommendation score); meta-path prediction | link |
Graph Attention Auto-Encoders (GATE) | ICTAI'20 | Feature prediction | Node classification | link |
Graph-Bert: Only Attention is Needed for Learning Graph Representations | arXiv:2001 | Feature prediction | Node classification; node clustering | link |
Self-supervised Learning on Graphs: Deep Insights and New Direction (AttributeMask) | arXiv:2006 | Masked feature prediction | Node classification | link |
SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks | NeurIPS'21 | Masked feature prediction | Node classification; image classification | link |
Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (MGSSL) | NeurIPS'21 | Masked feature prediction | Graph classification | link |
Multi-Scale Variational Graph AutoEncoder for Link Prediction (MSVGAE) | WSDM'22 | Feature prediction | Link prediction | -- |
Self-Supervised Representation Learning via Latent Graph Prediction (LaGraph) | ICML'22 | Masked feature prediction | Node classification; graph classification | link |
GraphMAE: Self-Supervised Masked Graph Autoencoders | KDD'22 | Masked feature prediction | Node classification; graph classification | link |
Interpretable Node Representation with Attribute Decoding (NORAD) | TMLR'22 | Feature prediction | Node classification; node clustering; link prediction | -- |
Graph Masked Autoencoders with Transformers (GMAE) | arXiv:2202 | Masked feature prediction | Node classification; graph classification | link |
Wiener Graph Deconvolutional Network Improves Graph Self-Supervised Learning (WGDN) | AAAI'23 | Feature prediction | Node classification; graph classification | link |
Heterogeneous Graph Masked Autoencoders (HGMAE) | AAAI'23 | Feature prediction; masked feature prediction | (Heterogeneous) node classification; node clustering | link |
Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules | ICLR'23 | Masked feature prediction | Graph classification; graph regression | link |
GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner | WWW'23 | Masked feature prediction | Node classification | link |
SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking | WWW'23 | Masked feature prediction | Node classification; link prediction; attribute prediction | link |
Directional Diffusion Models for Graph Representation Learning (DDM) | NeurIPS'23 | Feature denoising | Node classification; graph classification | link |
DiP-GNN: Discriminative Pre-Training of Graph Neural Networks | NeurIPS Workshop (GLFrontiers)'23 | Masked feature prediction | Node classification; link prediction | -- |
Towards Effective and Robust Graph Contrastive Learning With Graph Autoencoding (AEGCL) | TKDE'23 | Feature prediction | Node classification; node clustering; link prediction | link |
RARE: Robust Masked Graph Autoencoder | TKDE'23 | Masked feature prediction | Node classification; graph classification; image classification | link |
Homophily-Enhanced Self-Supervision for Graph Structure Learning: Insights and Directions (HES-GSL) | TNNLS'23 | Feature denoising | Node classification | link |
Incomplete Graph Learning via Attribute-Structure Decoupled Variational Auto-Encoder (ASD-VAE) | WSDM'24 | Feature prediction | Node classification; node attribute completion | link |
Deep Contrastive Graph Learning with Clustering-Oriented Guidance (DCGL) | AAAI'24 | Feature prediction | Node clustering | link |
Rethinking Graph Masked Autoencoders through Alignment and Uniformity (AUG-MAE) | AAAI'24 | Masked feature prediction | Node classification; graph classification | link |
Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery (DGPM) | AAAI'24 | Masked feature prediction | Graph classification | link |
Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE1) | ICDE'24 | Masked feature prediction | Node classification; node clustering; graph classification; link prediction | link |
Masked Graph Modeling with Multi-View Contrast (GCMAE2) | ICDE'24 | Masked feature prediction | Node classification; graph classification; link prediction | link |
Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders (StructMAE) | IJCAI'24 | Masked feature prediction | Graph classification | link |
Redundancy Is Not What You Need: An Embedding Fusion Graph Auto-Encoder for Self-Supervised Graph Representation Learning (EFGAE) | TNNLS'24 | Feature prediction | Node classification | -- |
- Latent feature matching: to minimize the (Euclidean) distance between pairs of positive representation vectors
- Instance discrimination: to minimize/maximize the distance between pairs of positive/negative representation samples. Jenson-Shannon (JS), InfoNCE (incl. NT-Xent), Triplet margin, and Bootstrapping are all estimators of mutual information (MI) between nodes. Other contrastive losses:
- SP stands for the population spectral contrastive loss
- BPR stands for Bayesian Personalized Ranking loss, mostly used in recommendation
- Dimension discrimination: to minimize/maximize the mutual information (MI) between pairs of positive/negative representation dimensions. Could be either intra-sample or inter-sample
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Deep Graph Contrastive Representation Learning (GRACE) | ICML Workshop (GRL+)'20 | Instance discrimination (InfoNCE) | Node classification | link |
GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-wise Transformations | CVPR'20 | Latent feature matching | Graph (point cloud) classification; node classification (point cloud segmentation) | link |
Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning (CG3) | AAAI'21 | Instance discrimination (InfoNCE) | Node classification | link |
Graph Contrastive Learning with Adaptive Augmentation (GCA) | WWW'21 | Instance discrimination (InfoNCE) | Node classification | link |
SelfGNN: Self-supervised Graph Neural Networks without Explicit Negative Sampling | WWW Workshop (SSL)'21 | Instance discrimination (Bootstrapping) | Node classification | link |
Self-supervised Graph Learning for Recommendation (SGL) | SIGIR'21 | Instance discrimination (InfoNCE, BPR) | Recommendation | link |
Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning (MERIT) | IJCAI'21 | Instance discrimination (InfoNCE) | Node classification | link |
Pre-training on Large-Scale Heterogeneous Graph (PT-HGNN) | KDD'21 | Instance discrimination (InfoNCE) | (Heterogeneous) node classification; link prediction | link |
Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning (HeCo); Hierarchical Contrastive Learning Enhanced Heterogeneous Graph Neural Network (HeCo++) | KDD'21; TKDE'23 | Instance discrimination (InfoNCE) | (Heterogeneous) node classification; node clustering | link |
InfoGCL: Information-Aware Graph Contrastive Learning | NeurIPS'21 | Instance discrimination (Bootstrapping) | Node classification; graph classification | -- |
From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (CCA-SSG) | NeurIPS'21 | Dimension discrimination; latent feature matching | Node classification | link |
Self-Supervised GNN that Jointly Learns to Augment (GraphSurgeon) | NeurIPS Workshop (SSL)'21 | Latent feature matching; dimension discrimination | Node classification | link |
Simple Unsupervised Graph Representation Learning (SUGRL) | AAAI'22 | Instance discrimination (Triplet margin) | Node classification | link |
Large-Scale Representation Learning on Graphs via Bootstrapping (BGRL) | ICLR'22 | Instance discrimination (Bootstrapping) | Node classification | link |
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (VICReg) | ICLR'22 | Dimension discrimination; latent feature matching | Node classification | link |
Adversarial Graph Contrastive Learning with Information Regularization (ARIEL) | WWW'22 | Instance discrimination (InfoNCE) | Node classification; graph classification | link |
Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendation (SimGCL); XSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation | SIGIR'22; TKDE'23 | Instance discrimination (InfoNCE, BPR) | Recommendation | link |
Self-Supervised Representation Learning via Latent Graph Prediction (LaGraph) | ICML'22 | Latent feature matching | Node classification; graph classification | link |
ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning | ICML'22 | Instance discrimination (InfoNCE) | Node classification | link |
COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning | KDD'22 | Instance discrimination (InfoNCE) | Node classification | link |
Relational Self-Supervised Learning on Graphs (RGRL) | CIKM'22 | Instance discrimination (Bootstrapping) | Node classification; link prediction | link |
Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrum (SpCo) | NeurIPS'22 | Instance discrimination (InfoNCE) | Node classification | link |
Contrastive Graph Structure Learning via Information Bottleneck for Recommendation (CGI) | NeurIPS'22 | Instance discrimination (InfoNCE) | Recommendation | link |
Uncovering the Structural Fairness in Graph Contrastive Learning (GRADE) | NeurIPS'22 | Instance discrimination (InfoNCE) | Node classification | link |
Co-Modality Graph Contrastive Learning for Imbalanced Node Classification (CM-GCL) | NeurIPS'22 | Instance discrimination (InfoNCE) | Node classification (imbalanced) | link |
Graph Barlow Twins: A Self-supervised Representation Learning Framework for Graphs (G-BT) | Knowledge-Based Systems'22 | Dimension discrimination | Node classification | link |
Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming (G-Zoom) | TNNLS'22 | Instance discrimination (InfoNCE) | Node classification | -- |
GRLC: Graph Representation Learning With Constraints | TNNLS'22 | Instance discrimination (Triplet margin) | Node classification; node clustering; link prediction | link |
Neural Eigenfunctions Are Structured Representation Learners (NeuralEF) | arXiv:2210 | Dimension discrimination | Node classification; image retrieval; object detection; instance segmentation | link |
MA-GCL: Model Augmentation Tricks for Graph Contrastive Learning | AAAI'23 | Instance discrimination (InfoNCE) | Node classification | link |
ImGCL: Revisiting Graph Contrastive Learning on Imbalanced Node Classification | AAAI'23 | Instance discrimination (InfoNCE) | Node classification (imbalanced) | -- |
Link Prediction with Non-Contrastive Learning (T-BGRL) | ICLR'23 | Instance discrimination (Bootstrapping) | Link prediction | link |
LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation | ICLR'23 | Instance discrimination (InfoNCE) | Recommendation | link |
GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner | WWW'23 | Latent feature matching | Node classification | link |
Graph Self-supervised Learning with Augmentation-aware Contrastive Learning (ABGML) | WWW'23 | Instance discrimination (Bootstrapping) | Node classification; node clustering; similarity search | link |
Randomized Schur Complement Views for Graph Contrastive Learning (rLap) | ICML'23 | Instance discrimination (InfoNCE, Bootstrapping) | Node classification | link |
Graph Contrastive Learning with Generative Adversarial Network (GACN) | KDD'23 | Instance discrimination (InfoNCE, BPR) | Node classification; link prediction | -- |
GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent Space Reconstruction | CIKM'23 | Instance discrimination (InfoNCE) | Node classification; node clustering; link prediction | link |
Provable Training for Graph Contrastive Learning (POT) | NeurIPS'23 | Instance discrimination (InfoNCE) | Node classification | link |
Graph Contrastive Learning with Stable and Scalable Spectral Encoding (Sp2GCL) | NeurIPS'23 | Instance discrimination (InfoNCE) | Node classification; graph classification; graph regression | link |
RARE: Robust Masked Graph Autoencoder | TKDE'23 | Latent feature matching | Node classification; graph classification; image classification | link |
Multi-Scale Self-Supervised Graph Contrastive Learning With Injective Node Augmentation (MS-CIA) | TKDE'23 | Instance discrimination (InfoNCE) | Node classification | -- |
Boosting Graph Contrastive Learning via Adaptive Sampling (AdaS) | TNNLS'23 | Instance discrimination (InfoNCE) | Node classification | -- |
Affinity Uncertainty-Based Hard Negative Mining in Graph Contrastive Learning (AUGCL) | TNNLS'23 | Instance discrimination (InfoNCE) | Node classification | link |
Unsupervised Structure-Adaptive Graph Contrastive Learning | TNNLS'23 | Instance discrimination (InfoNCE) | Node classification; node clustering; graph classification | -- |
Hierarchically Contrastive Hard Sample Mining for Graph Self-Supervised Pretraining (HCHSM) | TNNLS'23 | Instance discrimination (JS) | Node classification; node clustering | link |
Dual Contrastive Learning Network for Graph Clustering (DCLN) | TNNLS'23 | Dimension discrimination | Node classification; node clustering | link |
Graph Contrastive Learning With Adaptive Proximity-Based Graph Augmentation (PA-GCL) | TNNLS'23 | Dimension discrimination | Node classification; link prediction | link |
Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations (iGCL) | TNNLS'23 | Dimension discrimination; latent feature matching | Node classification | link |
Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph (SP-GCL) | TMLR'23 | Instance discrimination (SP) | Node classification | link |
Calibrating and Improving Graph Contrastive Learning (Contrast-Reg) | TMLR'23 | Instance discrimination (InfoNCE) | Node classification; node clustering; link prediction | link |
Oversmoothing: A Nightmare for Graph Contrastive Learning? (BlockGCL) | arXiv:2306 | Dimension discrimination | Node classification | link |
Rethinking and Simplifying Bootstrapped Graph Latents (SGCL2) | WSDM'24 | Instance discrimination (Bootstrapping) | Node classification | link |
Towards Alignment-Uniformity Aware Representation in Graph Contrastive Learning (AUAR) | WSDM'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | -- |
ReGCL: Rethinking Message Passing in Graph Contrastive Learning | AAAI'24 | Instance discrimination (InfoNCE) | Node classification | link |
A New Mechanism for Eliminating Implicit Conflict in Graph Contrastive Learning (PiGCL) | AAAI'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | link |
ASWT-SGNN: Adaptive Spectral Wavelet Transform-Based Self-Supervised Graph Neural Network | AAAI'24 | Instance discrimination (InfoNCE) | Node classification; graph classification | -- |
Graph Contrastive Invariant Learning from the Causal Perspective (GCIL) | AAAI'24 | Dimension discrimination | Node classification | link |
A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks (SpikeGCL) | ICLR'24 | Instance discrimination (Triplet margin) | Node classification | link |
Self-supervised Heterogeneous Graph Learning: a Homophily and Heterogeneity View (HERO) | ICLR'24 | Latent feature matching | (Heterogeneous) node classification; similarity search | link |
Graph Augmentation for Recommendation (GraphAug) | ICDE'24 | Instance discrimination (InfoNCE, BPR) | Recommendation | link |
Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE1) | ICDE'24 | Instance discrimination (InfoNCE) | Node classification; node clustering; graph classification; link prediction | link |
Graph Contrastive Learning with Cohesive Subgraph Awareness (CTAug) | WWW'24 | Instance discrimination (InfoNCE) | Node classification | link |
Towards Expansive and Adaptive Hard Negative Mining: Graph Contrastive Learning via Subspace Preserving (GRAPE) | WWW'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | link (Unavailable) |
MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning | WWW'24 | Instance discrimination (InfoNCE) | Node classification; graph classification | link |
Graph Contrastive Learning via Interventional View Generation (GCL-IVG) | WWW'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | -- |
Graph Contrastive Learning with Kernel Dependence Maximization for Social Recommendation (CL-KDM) | WWW'24 | Instance discrimination (InfoNCE, BPR) | Recommendation | -- |
High-Frequency-aware Hierarchical Contrastive Selective Coding for Representation Learning on Text-attributed Graphs (HASH-CODE) | WWW'24 | Instance discrimination (SP) | Node classification; link prediction | -- |
Redundancy Is Not What You Need: An Embedding Fusion Graph Auto-Encoder for Self-Supervised Graph Representation Learning (EFGAE) | TNNLS'24 | Dimension discrimination | Node classification | -- |
Multilevel Contrastive Graph Masked Autoencoders for Unsupervised Graph-Structure Learning (MCGMAE) | TNNLS'24 | Instance discrimination (InfoNCE) | Node classification | -- |
UniGLM: Training One Unified Language Model for Text-Attributed Graphs | arXiv:2406 | Instance discrimination (InfoNCE) | Node classification; link prediction | link |
Node properties
- Property prediction: a regression task to predict the property of a node (e.g. degree)
- Centrality ranking: to estimate whether the centrality score of a node is greater/lower than that of another node
- Node order matching: to match the output node order with the input order
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Unsupervised Pre-training of Graph Convolutional Networks (ScoreRank) | ICLR Workshop (RLGM)'19 | Centrality ranking | Node classification | -- |
Self-supervised Learning on Graphs: Deep Insights and New Direction (NodeProperty) | arXiv:2006 | Property prediction (degree, clustering coefficient, etc.) | Node classification | link |
Permutation-Invariant Variational Autoencoder for Graph-Level Representation Learning (PIGAE) | NeurIPS'21 | Node order matching | Graph classification | link |
Graph Auto-Encoder Via Neighborhood Wasserstein Reconstruction (NWR-GAE) | ICLR'22 | Property prediction (degree) | Node classification; structural role identification | link |
What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders (MaskGAE) | KDD'23 | Property prediction (degree) | Node classification; link prediction | link |
Links
- Link prediction: a generally binary classification task that predicts if two nodes are connected by a link.
- Hyperedge prediction: link prediction in hypergraphs. To search for the missing node given other nodes in a hyperedge
- Link denoising: to add (generally continuous) noises to the original edge set and try to reconstruct it
- Masked link prediction: to predict the masked links by node representations propagated on the unmasked graph. It is "autoregressive" if the predicted links are generated one-by-one
- Meta-path prediction: link prediction on heterogeneous graphs, to predict if two nodes are connected by a meta-path
- (Masked) edge feature prediction: to predict the original (masked) edge features by node representations
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Variational Graph Auto-Encoders (GAE, VGAE) | NIPS Workshop (BDL)'16 | Link prediction | Link prediction | link |
Adversarially Regularized Graph Autoencoder for Graph Embedding (ARGA, ARVGA) | IJCAI'18 | Link prediction | Link prediction; node clustering | link |
Unsupervised Pre-training of Graph Convolutional Networks (DenoisingRecon) | ICLR Workshop (RLGM)'19 | Masked link prediction | Node classification | -- |
Graphite: Iterative Generative Modeling of Graphs | ICML'19 | Link prediction | Node classification; link prediction | link |
Semi-Implicit Graph Variational Auto-Encoders (SIG-VAE) | NeurIPS'19 | Link prediction | Node classification; link prediction; node clustering; graph generation | link |
Strategies for Pre-training Graph Neural Networks (AttrMask) | ICLR'20 | Masked edge feature prediction | Graph classification; biological function prediction | link |
GPT-GNN: Generative Pre-Training of Graph Neural Networks | KDD'20 | Masked link prediction (autoregressive) | Node classification; edge classification; meta-path prediction | link |
Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs (SELAR) | NeurIPS'20 | Meta-path prediction | (Heterogeneous) node classification; link prediction | link |
Self-supervised Learning on Graphs: Deep Insights and New Direction (EdgeMask) | arXiv:2006 | Masked link prediction | Node classification | link |
Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning (CG3) | AAAI'21 | Link prediction | Node classification | link |
How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision (SuperGAT) | ICLR'21 | Link prediction | Node classification; link prediction | link |
Permutation-Invariant Variational Autoencoder for Graph-Level Representation Learning (PIGAE) | NeurIPS'21 | Link prediction; edge feature prediction | Graph classification | link |
Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (MGSSL) | NeurIPS'21 | Masked edge feature prediction | Graph classification | link |
Self-Supervised Graph Representation Learning via Topology Transformations (TopoTER) | TKDE'21 | Masked link prediction | Node classification; graph classification; link prediction | link |
Directed Graph Auto-Encoders (DiGAE) | AAAI'22 | Link prediction | (Directed) link prediction | link |
GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks | KDD'22 | Masked link prediction | Node classification | link |
Link Prediction with Contextualized Self-Supervision (CSSL2) | TKDE'22 | Link prediction | Link prediction | link |
Interpretable Node Representation with Attribute Decoding (NORAD) | TMLR'22 | Link prediction | Node classification; node clustering; link prediction | -- |
MGAE: Masked Autoencoders for Self-Supervised Learning on Graphs; S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking | WSDM'23 | Masked link prediction | Node classification; graph classification; link prediction | link |
Dual Low-Rank Graph Autoencoder for Semantic and Topological Networks (DLR-GAE) | AAAI'23 | Link prediction | Node classification | link |
Heterogeneous Graph Masked Autoencoders (HGMAE) | AAAI'23 | Masked meta-path prediction | (Heterogeneous) node classification; node clustering | link |
Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding (DMGAE, DMVGAE) | ICASSP'23 | Link prediction | Node clustering; link prediction | -- |
Multi-head Variational Graph Autoencoder Constrained by Sum-product Networks (SPN-MVGAE) | WWW'23 | Link prediction | Node classification; link prediction | link (Unavailable) |
SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking | WWW'23 | Masked link prediction | Node classification; link prediction; attribute prediction | link |
GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks; Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs (GraphPrompt+) | WWW'23 | Link prediction | Node classification; graph classification | link |
DiP-GNN: Discriminative Pre-Training of Graph Neural Networks | NeurIPS Workshop (GLFrontiers)'23 | Masked link prediction | Node classification; link prediction | -- |
Maximizing Mutual Information Across Feature and Topology Views for Representing Graphs (MVMI-FT) | TKDE'23 | Link prediction | Node classification; node clustering | link |
Towards Effective and Robust Graph Contrastive Learning With Graph Autoencoding (AEGCL) | TKDE'23 | Link prediction | Node classification; node clustering; link prediction | link |
ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt | arXiv:2310 | Link prediction | Node classification; link prediction | link |
Incomplete Graph Learning via Attribute-Structure Decoupled Variational Auto-Encoder (ASD-VAE) | WSDM'24 | Edge feature prediction | Node classification; node attribute completion | link |
HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs | ICLR'24 | Hyperedge prediction | Node classification; Hyperedge prediction | -- |
Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE1) | ICDE'24 | Link prediction | Node classification; node clustering; graph classification; link prediction | link |
Decoupled Variational Graph Autoencoder for Link Prediction (D-VGAE) | WWW'24 | Link prediction | Node classification; node clustering; link prediction | link |
Masked Graph Autoencoder with Non-discrete Bandwidths (Bandana) | WWW'24 | Link denoising | Node classification; link prediction | link |
Redundancy Is Not What You Need: An Embedding Fusion Graph Auto-Encoder for Self-Supervised Graph Representation Learning (EFGAE) | TNNLS'24 | Link prediction | Node classification | -- |
Context
- Context discrimination: to distinguish between contextual nodes and non-contextual nodes. LE stands for Laplacian Eigenmaps objective
- Factorized context discrimination: to maximize the log-likelihood of context representations given the corresponding central node conditioned on multiple disentangled latent factors
- Contextual subgraph discrimination: to distinguish between representations aggregated from different contextual subgraphs (maybe from different receptive fields). CE stands for cross-entropy
- Neighbor feature prediction: node feature prediction but to reconstruct the features of k-hop neighbors instead (BPR stands for Bayesian Personalized Ranking loss)
- Contextual property prediction: to predict the properties of contextual subgraphs (e.g. node / edge types contained, total node / edge counts, structural coefficient)
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Inductive Representation Learning on Large Graphs (GraphSAGE) | NIPS'17 | Context discrimination (JS) | Node classification | link |
Strategies for Pre-training Graph Neural Networks (ContextPred) | ICLR'20 | Contextual subgraph discrimination (CE) | Graph classification; biological function prediction | link |
GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding | ICLR'20 | Contextual subgraph discrimination (CE) | Node classification; link prediction | link |
GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training | KDD'20 | Contextual subgraph discrimination (InfoNCE) | Node classification; graph classification; similarity search | link |
Graph Attention Auto-Encoders (GATE) | ICTAI'20 | Context discrimination (JS) | Node classification | link |
Sub-Graph Contrast for Scalable Self-Supervised Graph Representation Learning (Subg-Con) | ICDM'20 | Context discrimination (Triplet margin) | Node classification | link |
Self-Supervised Graph Transformer on Large-Scale Molecular Data (GROVER) | NeurIPS'20 | Contextual property prediction | Graph classification; graph regression | link |
Pre-training on Large-Scale Heterogeneous Graph (PT-HGNN) | KDD'21 | Context discrimination (InfoNCE) | (Heterogeneous) node classification; link prediction | link |
Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization (EGI) | NeurIPS'21 | Context discrimination (JS) | Role identification; relation prediction | link |
Contrastive Laplacian Eigenmaps (COLES) | NeurIPS'21 | Context discrimination (LE) | Node classification; node clustering | link |
Graph-MLP: Node Classification without Message Passing in Graph | arXiv:2106 | Context discrimination (InfoNCE) | Node classification | link |
Augmentation-Free Self-Supervised Learning on Graphs (AFGRL) | AAAI'22 | Context discrimination (Bootstrapping) | Node classification; node clustering; similarity search | link |
Simple Unsupervised Graph Representation Learning (SUGRL) | AAAI'22 | Context discrimination (Triplet margin) | Node classification | link |
SAIL: Self-Augmented Graph Contrastive Learning | AAAI'22 | Neighbor feature prediction (BPR) | Node classification; node clustering; link prediction | -- |
Robust Self-Supervised Structural Graph Neural Network for Social Network Prediction | WWW'22 | Contextual subgraph discrimination (InfoNCE) | Node classification; graph classification; similarity search | -- |
Node Representation Learning in Graph via Node-to-Neighbourhood Mutual Information Maximization (N2N) | CVPR'22 | Context discrimination (InfoNCE) | Node classification | link |
RoSA: A Robust Self-Aligned Framework for Node-Node Graph Contrastive Learning | IJCAI'22 | Contextual subgraph discrimination (InfoNCE) | Node classification | link |
Graph Auto-Encoder Via Neighborhood Wasserstein Reconstruction (NWR-GAE) | ICLR'22 | Neighbor feature prediction (Wasserstein distance) | Node classification; structural role identification | link |
Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction (GIANT) | ICLR'22 | Neighbor matching1 | Node classification | link |
Towards Self-supervised Learning on Graphs with Heterophily (HGRL) | CIKM'22 | Context discrimination (InfoNCE) | Node classification; node clustering | link |
Unifying Graph Contrastive Learning with Flexible Contextual Scopes (UGCL) | ICDM'22 | Context discrimination (InfoNCE) | Node classification | link |
Generalized Laplacian Eigenmaps (GLEN) | NeurIPS'22 | Context discrimination (LE) | Node classification; node clustering | link |
Decoupled Self-supervised Learning for Graphs (DSSL) | NeurIPS'22 | Factorized context discrimination | Node classification | link |
Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming (G-Zoom) | TNNLS'22 | Context discrimination (JS) | Node classification | -- |
Link Prediction with Contextualized Self-Supervision (CSSL2) | TKDE'22 | Context discrimination (CE) | Link prediction | link |
Localized Graph Contrastive Learning (Local-GCL) | arXiv:2212 | Context discrimination (InfoNCE) | Node classification | link |
Deep Graph Structural Infomax (DGSI) | AAAI'23 | Context discrimination (JS) | Node classification | link |
Neighbor Contrastive Learning on Learnable Graph Augmentation (NCLA) | AAAI'23 | Context discrimination (InfoNCE) | Node classification | link |
Eliciting Structural and Semantic Global Knowledge in Unsupervised Graph Contrastive Learning (S3-CL) | AAAI'23 | Contextual subgraph discrimination (InfoNCE) | Node classification; node clustering | link |
GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks; Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs (GraphPrompt+) | WWW'23; TKDE'24 | Contextual subgraph discrimination (InfoNCE) | Node classification; graph classification | link |
Contrastive Learning Meets Homophily: Two Birds with One Stone (NeCo) | ICML'23 | Context discrimination (InfoNCE) | Node classification | -- |
Contrastive Cross-scale Graph Knowledge Synergy (CGKS) | KDD'23 | Context discrimination (LE); contextual subgraph discrimination (InfoNCE) | Node classification; graph classification | -- |
Simple and Asymmetric Graph Contrastive Learning without Augmentations (GraphACL) | NeurIPS'23 | Context discrimination (InfoNCE) | Node classification | link |
Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks (APT) | NeurIPS'23 | Context discrimination (InfoNCE) | Node classification; graph classification | link |
Dual Contrastive Learning Network for Graph Clustering (DCLN) | TNNLS'23 | Context discrimination (InfoNCE) | Node classification; node clustering | link |
Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning (HTML) | AAAI'24 | Contextual property prediction (structural coefficient) | Graph classification | link |
Graph Contrastive Learning Reimagined: Exploring Universality (ROSEN) | WWW'24 | Context discrimination (InfoNCE) | Node classification; node clustering | -- |
High-Frequency-aware Hierarchical Contrastive Selective Coding for Representation Learning on Text-attributed Graphs (HASH-CODE) | WWW'24 | Context discrimination (SP); contextual subgraph discrimination (SP) | Node classification; link prediction | -- |
[Efficient Contrastive Learning for Fast and Accurate Inference on Graphs] (GraphECL) | ICML'24 | Context discrimination (InfoNCE) | Node classification | link (Unavailable) |
1GIANT fine-tunes a language model to extract structural node features for GNN input
Long-range similarities
- Similarity prediction: to predict a similarity matrix between nodes. The pairwise similarity can be defined by shortest path distance, PageRank similarity, Katz index, Jaccard coefficient,
$\ell_2$ distance & cosine similarity between output representations / input-output, etc - Similarity-based discrimination: instance discrimination that is node similarity-aware
- Similarity graph alignment: to construct an additional similarity graph based on pairwise similarities of node features or graph topology, and minimize the distance of representation distributions between them (the original and similarity graph, or two different similarity graphs)
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Adaptive Graph Encoder for Attributed Graph Embedding (AGE) | KDD'20 | Similarity prediction (cosine similarity) | Node clustering; link prediction | link |
AM-GCN: Adaptive Multi-channel Graph Convolutional Networks | KDD'20 | Similarity graph alignment (MSE) | Node classification | link |
Graph-Bert: Only Attention is Needed for Learning Graph Representations | arXiv:2001 | Similarity prediction (PageRank, etc.) | Node classification; node clustering | link |
Self-supervised Learning on Graphs: Deep Insights and New Direction (PairwiseDistance, PairwiseAttrSim) | arXiv:2006 | Similarity prediction (shortest path distance; cosine similarity) | Node classification | link |
SAIL: Self-Augmented Graph Contrastive Learning | AAAI'22 | Similarity prediction (cosine similarity) | Node classification; node clustering; link prediction | -- |
Self-Supervised Graph Representation Learning via Global Context Prediction; A New Self-supervised Task on Graphs: Geodesic Distance Prediction (S2GRL) | Information Sciences'22 | Similarity prediction (shortest path distance) | Node classification; node clustering; link prediction | -- |
Dual Low-Rank Graph Autoencoder for Semantic and Topological Networks (DLR-GAE) | AAAI'23 | Similarity graph alignment (CE) | Node classification | link |
Attribute and Structure Preserving Graph Contrastive Learning (ASP) | AAAI'23 | Similarity graph alignment (InfoNCE) | Node classification | link |
Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding (DMGAE, DMVGAE) | ICASSP'23 | Similarity prediction ( |
Node clustering; link prediction | -- |
Self-Supervised Teaching and Learning of Representations on Graphs (GraphTL) | WWW'23 | Similarity-based discrimination (cosine similarity) | Node classification | -- |
Graph Self-supervised Learning via Proximity Divergence Minimization (PDM) | UAI'23 | Similarity prediction (heat kernel, personalized PageRank, SimRank) | Node classification | link |
Maximizing Mutual Information Across Feature and Topology Views for Representing Graphs (MVMI-FT) | TKDE'23 | Similarity graph alignment (JS) | Node classification; node clustering | link |
Towards Effective and Robust Graph Contrastive Learning With Graph Autoencoding (AEGCL) | TKDE'23 | Similarity graph alignment (InfoNCE) | Node classification; node clustering; link prediction | link |
ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt | arXiv:2310 | Similarity prediction (cosine similarity) | Node classification; link prediction | link |
Deep Contrastive Graph Learning with Clustering-Oriented Guidance (DCGL) | AAAI'24 | Similarity graph alignment (InfoNCE) | Node clustering | link |
Select Your Own Counterparts: Self-Supervised Graph Contrastive Learning With Positive Sampling (GPS) | TNNLS'24 | Similarity-based discrimination (cosine similarity, personalized PageRank, etc; InfoNCE, Bootstrapping) | Node classification | -- |
Motifs
- Motif prediction: to assign each node (or supernode in the fragment graph) a motif pseudo-label given by unsupervised motif discovery algorithms (e.g. RDKit) and learn to predict them. It is "autoregressive" if the predicted supernodes are generated one-by-one
- Motif-based masked feature prediction: similar to masked feature prediction, but the features are masked in motifs
- Motif-based discrimination: to perform contrast between the original graph view and the fragment graph view
Papers | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Self-Supervised Graph Transformer on Large-Scale Molecular Data (GROVER) | NeurIPS'20 | Motif prediction | Graph classification; graph regression | link |
Motif-Driven Contrastive Learning of Graph Representations (MICRO-Graph) | WWW Workshop (SSL)'21 | Motif-based discrimination (InfoNCE) | Graph classification | link |
Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (MGSSL) | NeurIPS'21 | Motif prediction (autoregressive) | Graph classification | link |
Fragment-based Pretraining and Finetuning on Molecular Graphs (GraphFP) | NeurIPS'23 | Motif prediction; motif-based discrimination (InfoNCE) | Graph classification; graph regression | link |
Motif-aware Attribute Masking for Molecular Graph Pre-training (MoAMa) | NeurIPS Workshop (GLFrontiers)'23 | Motif-based masked feature prediction | Graph classification | link |
Motif-aware Riemannian Graph Neural Network with Generative-Contrastive Learning (MotifRGC) | AAAI'24 | Motif-based discrimination (InfoNCE) | Node classification; link prediction | link |
Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery (DGPM) | AAAI'24 | Motif prediction | Graph classification | link |
Graph Contrastive Learning with Cohesive Subgraph Awareness (CTAug) | WWW'24 | Motif-based discrimination (InfoNCE) | Graph classification | link |
Clusters
- Synthetic graph discrimination: binary classification between two synthetic graphs with different synthesizers (Erdős-Rényi generator / SBM generator)
- Node clustering: to assign each node a cluster centroid (prototype) and - i) minimize the distance between nodes and their corresponding centroids in the latent space; or ii) minimize the distance between the learned centroids and the ground-truth centroids given by unsupervised feature clustering algorithms (e.g. K-means, DeepCluster)
- Graph partitioning: to assign each node a cluster centroid (prototype) and - i) predict the quality of the learned partitions evaluated by some metrics, e.g. maximizing modularity or minimizing the normalized edge weights of a graph cut (spectral clustering); or ii) predict the cluster membership of each node given by unsupervised graph partitioning algorithms (structure-based, e.g. METIS, Louvain)
- Cluster/partition-based instance discrimination: instance discrimination that is aware of graph clustering/partitioning memberships
- Cluster/partition-conditioned link prediction: to maximize the log-likelihood of existing links, but conditioned by the graph cluster/partition distributions
- Partition-conditioned masked link prediction: similar to masked link prediction, but the links are masked in clusters
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
SGR: Self-Supervised Spectral Graph Representation Learning | KDD Workshop (DLD)'18 | Synthetic graph discrimination | Graph classification | -- |
Unsupervised Pre-training of Graph Convolutional Networks (ClusterDetect) | ICLR Workshop (RLGM)'19 | Graph partitioning | Node classification | -- |
Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes (M3S) | AAAI'20 | Node clustering | Node classification | link |
Collaborative Graph Convolutional Networks: Unsupervised Learning Meets Semi-Supervised Learning (CGCN) | AAAI'20 | Partition-conditioned link prediction | Node classification; node clustering | link (Deleted) |
When Does Self-Supervision Help Graph Convolutional Networks? (NodeCluster, GraphPar) | ICML'20 | Node clustering; graph partitioning | Node classification | link |
CommDGI: Community Detection Oriented Deep Graph Infomax | CIKM'20 | Cluster-based instance discrimination (JS); graph partitioning | Node clustering | link |
Dirichlet Graph Variational Autoencoder (DGVAE) | NeurIPS'20 | Partition-conditioned link prediction | Graph generation; node clustering | link |
Self-supervised Learning on Graphs: Deep Insights and New Direction (Distance2Clusters) | arXiv:2006 | Graph partitioning | Node classification | link |
Mask-GVAE: Blind Denoising Graphs via Partition | WWW'21 | Graph partitioning; partition-conditioned masked link prediction | Node clustering; graph denoising | link |
Motif-Driven Contrastive Learning of Graph Representations (MICRO-Graph) | WWW Workshop (SSL)'21 | Graph partitioning | Graph classification | link |
Self-supervised Graph-level Representation Learning with Local and Global Structure (GraphLoG) | ICML'21 | Node clustering | Graph classification; biological function prediction | link |
Graph Communal Contrastive Learning (gCooL) | WWW'22 | Partition-based instance discrimination (InfoNCE) | Node classification; node clustering | link |
Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering (SHGP) | NeurIPS'22 | Graph partitioning | (Heterogeneous) node classification; node clustering | link |
Eliciting Structural and Semantic Global Knowledge in Unsupervised Graph Contrastive Learning (S3-CL) | AAAI'23 | Cluster-based instance discrimination (InfoNCE) | Node classification; node clustering | link |
CSGCL: Community-Strength-Enhanced Graph Contrastive Learning | IJCAI'23 | Partition-based instance discrimination (InfoNCE) | Node classification; node clustering; link prediction | link |
HomoGCL: Rethinking Homophily in Graph Contrastive Learning | KDD'23 | Node clustering; cluster-based instance discrimination (InfoNCE) | Node classification; node clustering | link |
CARL-G: Clustering-Accelerated Representation Learning on Graphs | KDD'23 | Node clustering | Node classification; node clustering; similarity search | link |
Towards Alignment-Uniformity Aware Representation in Graph Contrastive Learning (AUAR) | WSDM'24 | Node clustering | Node classification; node clustering | -- |
Deep Contrastive Graph Learning with Clustering-Oriented Guidance (DCGL) | AAAI'24 | Cluster-based instance discrimination (InfoNCE) | Node clustering | link |
StructComp: Substituting propagation with Structural Compression in Training Graph Contrastive Learning | ICLR'24 | Partition-based instance discrimination (JS, InfoNCE, etc.) | Node classification | link |
MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning | WWW'24 | Cluster-based instance discrimination | Node classification; graph classification | link |
Graph Contrastive Learning with Kernel Dependence Maximization for Social Recommendation (CL-KDM) | WWW'24 | Partition-based instance discrimination (BPR) | Recommendation | -- |
Community-Invariant Graph Contrastive Learning (CI-GCL) | ICML'24 | Partition-based instance discrimination (InfoNCE) | Graph classification; graph regression | link |
Global structure
- Global-local instance discrimination: instance discrimination between the representation of each node and a global representation vector, usually aggregated from the whole graph by a readout function
- Group discrimination: a simplified global-local instance discrimination that binarily classifies if a node belongs to the original or the perturbed graph
- Global instance discrimination: to discriminate between global representations of different graph views (small-scale only)
- Global dimension discrimination: dimension discrimination of different graph representations
- Factorized global instance discrimination: to maximize the log-likelihood of the target graph instance given the corresponding instance conditioned on multiple disentangled latent factors
- Graph similarity prediction: to predict various kinds of similarity functions between pairs of graphs, e.g. graph kernels (graphlet kernel, random walk kernel, graph edit distance kernel, etc)
- Half-graph matching: to divide each graph into two halves and predict if two halves are from the same original graph
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Pre-training Graph Neural Networks with Kernels (KernelPred) | arXiv:1811 | Graph similarity prediction | Graph classification | -- |
Deep Graph InfoMax (DGI) | ICLR'19 | Global-local instance discrimination (JS) | Node classification | link |
InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization | ICLR'20 | Global-local instance discrimination (JS) | Graph classification | link |
Graph Contrastive Learning with Augmentations (GraphCL) | NeurIPS'20 | Global instance discrimination (InfoNCE) | Graph classification | link |
Contrastive Multi-View Representation Learning on Graphs (MVGRL) | ICML'20 | Global-local instance discrimination (JS) | Node classification; graph classification | link |
Contrastive Self-supervised Learning for Graph Classification (CSSL1) | AAAI'21 | Global instance discrimination (InfoNCE) | Graph classification | -- |
Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks (PHD); An Effective Self-Supervised Framework for Learning Expressive Molecular Global Representations to Drug Discovery (MPG) | IJCAI'21; Briefings in Bioinformatics'21 | Half-graph matching | Graph classification | link |
Graph Contrastive Learning Automated (JOAO) | ICML'21 | Global instance discrimination (InfoNCE) | Graph classification | link |
Adversarial Graph Augmentation to Improve Graph Contrastive Learning (AD-GCL) | NeurIPS'21 | Global instance discrimination (InfoNCE) | Graph classification | link |
InfoGCL: Information-Aware Graph Contrastive Learning | NeurIPS'21 | Global instance discrimination (Bootstrapping); global-local instance discrimination (Bootstrapping) | Node classification; graph classification | -- |
Graph Adversarial Self-Supervised Learning (GASSL) | NeurIPS'21 | Global instance discrimination (Bootstrapping) | Graph classification | link (Unavailable) |
Disentangled Contrastive Learning on Graphs (DGCL) | NeurIPS'21 | Factorized global instance discrimination | Graph classification | link |
Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations (GraphCL-LP) | WSDM'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
Self-Supervised Graph Neural Networks via Diverse and Interactive Message Passing (DIMP) | AAAI'22 | Global-local instance discrimination (JS) | Node classification; node clustering; graph classification | link |
AutoGCL: Automated Graph Contrastive Learning via Learnable View Generators | AAAI'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
Group Contrastive Self-Supervised Learning on Graphs (GroupCL; GroupIG) | TPAMI'22 | Global instance discrimination (JS; contrastive log-ratio upper bound (CLUB)) | Graph classification | -- |
Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming (G-Zoom) | TNNLS'22 | Global-local instance discrimination (JS) | Node classification | -- |
SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation | WWW'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
Let Invariant Rationale Discovery Inspire Graph Contrastive Learning (RGCL) | ICML'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
M-Mix: Generating Hard Negatives via Multi-sample Mixing for Contrastive Learning | KDD'22 | Global instance discrimination (InfoNCE) | Node classification; node clustering; graph classification; graph edit distance prediction | link |
AdaGCL: Adaptive Subgraph Contrastive Learning to Generalize Large-scale Graph Training | CIKM'22 | Global-local instance discrimination (JS) | Node classification | link |
Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination (GGD) | NeurIPS'22 | Group discrimination | Node classification | link |
Graph Self-supervised Learning with Accurate Discrepancy Learning (D-SLA) | NeurIPS'22 | Group discrimination; graph similarity prediction | Graph classification; link prediction | link |
Deep Graph Structural Infomax (DGSI) | AAAI'23 | Global-local instance discrimination (JS) | Node classification | link |
Spectral Augmentation for Self-Supervised Learning on Graphs (SPAN) | ICLR'23 | Global-local instance discrimination (InfoNCE) | Node classification; graph classification; graph regression | link |
Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules | ICLR'23 | Global instance discrimination (InfoNCE; Triplet margin) | Graph classification; graph regression | link |
Spectral Augmentations for Graph Contrastive Learning (SGCL1) | AISTATS'23 | Global instance discrimination (InfoNCE) | Node classification; graph classification; similarity search | -- |
Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning (CGC) | WWW'23 | Global instance discrimination (InfoNCE) | Graph classification | link (Unavailable) |
Multi-Scale Subgraph Contrastive Learning (MSSGCL) | IJCAI'23 | Global-local instance discrimination (InfoNCE); global instance discrimination (InfoNCE) | Graph classification | link |
Boosting Graph Contrastive Learning via Graph Contrastive Saliency (GCS) | ICML'23 | Global instance discrimination (InfoNCE) | Graph classification | link |
SEGA: Structural Entropy Guided Anchor View for Graph Contrastive Learning | ICML'23 | Global instance discrimination (InfoNCE) | Graph classification | link |
Randomized Schur Complement Views for Graph Contrastive Learning (rLap) | ICML'23 | Global-local instance discrimination (InfoNCE); global instance discrimination (InfoNCE) | Graph classification | link |
Graph Self-Contrast Representation Learning (GraphSC) | ICDM'23 | Global instance discrimination (Triplet margin); global dimension discrimination | Graph classification | -- |
Graph Contrastive Learning with Stable and Scalable Spectral Encoding (Sp2GCL) | NeurIPS'23 | Global instance discrimination (InfoNCE) | Node classification; graph classification; graph regression | link |
Maximizing Mutual Information Across Feature and Topology Views for Representing Graphs (MVMI-FT) | TKDE'23 | Global-local instance discrimination (JS) | Node classification; node clustering | link |
Multi-Scale Self-Supervised Graph Contrastive Learning With Injective Node Augmentation (MS-CIA) | TKDE'23 | Global-local instance discrimination (JS) | Node classification | -- |
Hierarchically Contrastive Hard Sample Mining for Graph Self-Supervised Pretraining (HCHSM) | TNNLS'23 | Global-local instance discrimination (JS) | Node classification; node clustering | link |
Dual Contrastive Learning Network for Graph Clustering (DCLN) | TNNLS'23 | Global-local instance discrimination (JS) | Node classification; node clustering | link |
HeGCL: Advance Self-Supervised Learning in Heterogeneous Graph-Level Representation | TNNLS'23 | Global-local instance discrimination (JS) | (Heterogeneous) node classification; graph classification | link |
Affinity Uncertainty-Based Hard Negative Mining in Graph Contrastive Learning (AUGCL) | TNNLS'23 | Global instance discrimination (InfoNCE) | Graph classification | link |
Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning (HTML) | AAAI'24 | Global instance discrimination (InfoNCE); graph similarity prediction (Jaccard coef-based isomorphic similarity) | Graph classification | link |
TopoGCL: Topological Graph Contrastive Learning | AAAI'24 | Global instance discrimination (InfoNCE) | Graph classification | link (Unavailable) |
Masked Graph Modeling with Multi-View Contrast (GCMAE2) | ICDE'24 | Global instance discrimination (InfoNCE) | Node classification; graph classification; link prediction | link |
Graph Contrastive Learning with Personalized Augmentation (GPA) | TKDE'24 | Global instance discrimination (InfoNCE) | Graph classification | link |
Graph Contrastive Learning with Min-Max Mutual Information (GCLMI) | Information Sciences'24 | Global instance discrimination (InfoNCE) | Graph classification | link |
Manifolds
- Cross-manifold discrimination: to perform instance discrimination between different manifolds (e.g. Euclidean vs. Hyperbolic)
- Ricci curvature prediction: to predict the aggregated Ricci curvature of each node's neighborhood
- Curvature-based node clustering: to assign each node a cluster centroid and maximize/minimize the curvature-based density within/across clusters
- Hyperbolic angle prediction: to pool representations to 2-dimensional angle vectors in a unit hyperbola. These vectors serve as pseudo-labels for regression
Paper | Venue | Pretext | Downstream | Code |
---|---|---|---|---|
Enhancing Hyperbolic Graph Embeddings via Contrastive Learning (HGCL) | NeurIPS Workshop (SSL)'21 | Cross-manifold discrimination (InfoNCE) | Node classification | -- |
A Self-supervised Mixed-curvature Graph Neural Network (SelfMGNN) | AAAI'22 | Cross-manifold discrimination (InfoNCE) | Node classification | -- |
Dual Space Graph Contrastive Learning (DSGC) | WWW'22 | Cross-manifold discrimination (InfoNCE) | Graph classification | link (Unavailable) |
CONGREGATE: Contrastive Graph Clustering in Curvature Spaces | IJCAI'23 | Ricci curvature prediction; cross-manifold discrimination (InfoNCE); curvature-based node clustering | Node clustering | link |
Graph-level Representation Learning with Joint-Embedding Predictive Architectures (GraphJEPA) | arXiv:2309 | Hyperbolic angle prediction | Graph classification; graph regression | link |
Motif-aware Riemannian Graph Neural Network with Generative-Contrastive Learning (MotifRGC) | AAAI'24 | Cross-manifold discrimination (InfoNCE) | Node classification; link prediction | link |
Task generalization strategies
- Multi-task learning: to combine different pretexts and jointly learn them for task-generalizable performance (technically not a task generalization strategy, and is compatible with other three task generalization strategies)
- Fine-tuning: to jointly learn downstream branches as well as the original pre-trained model. Parameter-efficient fine-tuning (PEFT) only updates part of the pre-trained model, e.g. adapter layers
- Probing: to freeze the parameters of the pre-trained model during the downstream task generalization. The probe can be either parameterized or non-parameterized
- Prompting: to jointly encode downstream data and extra learnable task-specific components to instruct the behavior of pre-trained models for downstream generalization
1Can be seen as a GNN probing method prompted by the pre-trained LLM
❤️ Contributions by issues and pull requests to this source list are always welcome! Feel free to initiate a discussion with me, or give me a reminder if there are oversights of papers/hyperlinks or categorical mistakes.