Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


- Depending on your transformer toolkit versions, the transformer import code may need to be adjusted, like as follows:
+ from transformers.modeling_bert import BertPreTrainedModel, BertPooler
+ --> from transformers.models.bert.modeling_bert import BertPreTrainedModel, BertPooler
- (Please check your transformer toolikt, and update the import code accordingly.)

Graph-Bert Model Architecture

Graph-Bert: Only Attention is Needed for Learning Graph Representations

Paper URL at IFM Lab:

Paper URL at arXiv:

Graph-Bert Paper List

A list of the latest research papers on graph-bert can be found via the following link
Page List URL:

Two other papers are helpful for readers to follow the ideas in this paper and the code

(1) SEGEN: Sample-Ensemble Genetic Evolutional Network Model

(2) GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation

Graph Neural Networks from IFM Lab

The latest graph neural network models proposed by IFM Lab can be found via the following link


  title={Graph-Bert: Only Attention is Needed for Learning Graph Representations},
  author={Zhang, Jiawei and Zhang, Haopeng and Xia, Congying and Sun, Li},
  journal={arXiv preprint arXiv:2001.05140},

How to run the code?

To run a script, you can just use command line: python3

After downloading the code, you can run


directly for node classification. It seems the random seed cannot control the randomness in parameter initialization in transformer, we suggest to run the code multiple times to get good scores.

What are the scripts used for?

(1) The Graph-Bert model takes (a) node WL code, (b) intimacy based subgraph batch, (c) node hop distance as the prior inputs. These can be computed with the

(2) Pre-training of Graph-Bert based on node attribute reconstruction and graph structure recovery is provided by

(3) Please check the as the entry point to run the model on node classification and graph clustering.

(4) is used for plots drawing and results evaluation purposes.

How to turn on/off the blocks?

You can change the "if 0" to "if 1" to turn on a script block, and the reverse to turn off a script block.

How save/load pre-trained graph-bert?

Please refer to the attached page for model save/load in pytorch:

You can load the pre-trained graph-bert component with pytorch model (e.g., the graph-bert inside MethodGraphBertNodeClassification) with the save and load functions in the above page. According to our experiments, Save/Load Entire Model in the above page works perfectly, whereas Save/Load state_dict doesn't really work.

Several toolkits may be needed to run the code

(1) pytorch (
(2) sklearn (
(3) transformers (
(4) networkx (

Organization of the code?

A simpler template of the code is also available at

The whole program is divided into five main parts:

(1) (for data loading and basic data organization operators, defines abstract method load() )

(2) (for complex operations on the data, defines abstract method run() )

(3) (for saving/loading results from files, defines abstract method load() and save() )

(4) (for result evaluation, defines abstract method evaluate() )

(5) (for experiment settings, defines abstract method load_run_save_evaluate() )

The base class of these five parts are defined in ./code/base_class/, they are all abstract class defining the templates and architecture of the code.

The inherited class are provided in ./code, which inherit from the base classes, implement the abstract methonds.

Detailed information on funtional classes?

a. data

(1) (for dataset loading)

b. method

b-1 pre-computing methods

(1) (for WL code computing)

(2) (for subgraph batching)

(3) (for hop distance computing)

b-2 graph-bert basics

(1) (for graph-bert basic components)

(2) (the graph bert model)

b-3 graph-bert applications





c. result

(1) (for saving results to file)

d. evaluate

(1) (accuracy metric)

(2) (a bunch of clustering metrics)

e. setting

(1) (defines the interactions and data exchange among the above classes)

Learning results of Graph-Bert with graph-raw residual on Cora.

Learning Results of Graph-Bert with Graph Residual on Cora

Learning results of Graph-Bert with raw residual on Cora.

Learning Results of Graph-Bert with Raw Residual on Cora


Source code of Graph-Bert






No releases published


No packages published