Categorization and summarization of the state-of-the-art Self-supervised Learning (SSL) on Graph Neural Networks (GNNs).
For more details, please refer to our book chapter (link coming soon). If you find this useful and use it in your research, please temporarily cite our work as follows:
@inproceedings{wang2021gnnssl,
author = {Wang, Yu and Jin, Wei and Derr, Tyler},
title = {Graph Neural Networks: Self-supervised Learning},
year = {2021}
}
Pretext tasks are constructed by leveraging different types of supervision information coming from different components of graphs. Based on the components that generate the supervision information, pretext tasks that are prevalent in the literature are categorized into node-level, graph-level and node-graph level. In completing node-level and graph-level pretext tasks, three types of information can be leveraged: graph structure, node features, or hybrid, where the latter combines the information from node features, graph structure, and even information from the known training labels. We summarize the categorization of pretext tasks as a tree where each leaf node represents a specific type of pretext tasks in the following figure while also including the corresponding references.
For node-level pretext tasks, methods have been developed to use easily-accessible data to generate pseudo labels for each node or relationships for each pair of nodes. In this way, the GNNs are then trained to be predictive of the generated pseudo labels or to keep the equivalence between the node embeddings and the original node relationships.
Different nodes have different structure properties in graph topology, which can be measured by the node degree, centrality, node partition, etc. Thus, for structure-based pretext tasks at the node-level, we expect to align node embeddings extracted from the GNNs with their structure properties, in an attempt to ensure this information is preserved while GNNs learn the node embeddings.
- Self-supervised Learning on Graphs: Deep Insights and New Direction (Arxiv 2020) [Code].
- When Does Self-Supervision Help Graph Convolutional Networks? (ICML 2020) [Code].
- Pre-Training Graph Neural Networks for Generic Structural Feature Extraction (ICLR 2019 workshop).
Node features are another important information that can be leveraged to provide extra supervision.
- Self-supervised Learning on Graphs: Deep Insights and New Direction (Arxiv 2020) [Code].
- When Does Self-Supervision Help Graph Convolutional Networks? (ICML 2020) [Code].
- Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks (Openreview 2020) .
- Strategies for Pre-training Graph Neural Networks (ICLR 2020) [Code].
Instead of employing only the topology or only the feature information as the extra supervision, some pretext tasks combine them together as a hybrid supervision, or even utilize information from the known training labels.
- Distance-wise Graph Contrastive Learning (Arxiv 2020).
- Deep Graph Contrastive Representation Learning (Arxiv 2020) [Code].
- Graph Contrastive Learning with Adaptive Augmentation (WWW 2021).
- Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels (AAAI 2020).
After having just presented the node-level SSL pretext tasks, in this section we focus on the graph-level SSL pretext tasks where we desire the node embeddings coming from the GNNs to encode information of graph-level properties.
- Self-supervised Training of Graph Convolutional Networks (Arxiv 2020).
- SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks (Openreview 2020).
- Self-supervised Learning on Graphs: Deep Insights and New Direction (Arxiv 2020).
- Self-Supervised Graph Representation Learning via Global Context Prediction (Openreview 2020).
- How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision (ICLR 2021).
- Motif-Driven Contrastive Learning of Graph Representations (Openreview 2020).
- GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training (KDD 2020) [Code].
- TopoTER: Unsupervised Learning of Topology Transformation Equivariant Representations (Openreview 2020).
Typically, graphs does not come with any feature information and here the graph-level features refer to the graph embeddings obtained after applying a pooling layer on all node embeddings from GNNs.
- Strategies for Pre-training Graph Neural Networks (ICLR 2020) [Code].
- Self-supervised Graph-level Representation Learning with Local and Global Structure (Openreview 2020).
- Self-supervised Learning on Graphs: Deep Insights and New Direction (Arxiv 2020) [Code].
- Strategies for Pre-training Graph Neural Networks (ICLR 2020) [Code].
- GPT-GNN: Generative Pre-Training of Graph Neural Networks (KDD 2020) [Code].
All the above pretext tasks are designed based on either the node or the graph level supervision. However, there is another final line of research combining these two sources of supervision to design pretext tasks, which we summarize in this section.
- Deep Graph Informax (ICLR 2019) [Code].
- Contrastive Multi-View Representation Learning on Graphs (ICML 2020) [Code].
- COAD: Contrastive Pre-training with Adversarial Fine-tuning for Zero-shot Expert Linking (Arxiv 2020).
- Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation (WSDM 2021) [Code].
- Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (ICML 2020).
- Can Graph Neural Networks Go "Online"? An Analysis of Pretraining and Inference (ICLR 2019 workshop).
For an up to date list of SSL GNN papers please refer to awesome-self-supervised-gnn.