GraphLLM: Boosting Graph Reasoning Ability of Large Language Model

This is the implementation for the paper GraphLLM: Boosting Graph Reasoning Ability of Large Language Model.

Setup

• You may need a single 80G GPU to run the experiment. We experiment on CUDA 11.8 and torch 2.0.1.
• Setup up a new conda env and install necessary packages.

  conda create -n graph_llm python=3.10 -y
  pip install -r requirements.txt

• To run the code, you need the checkpoint and tokenizer of LLaMA-2-7B, which you can access at Meta. After downloading LLaMA-2-7B, soft link the checkpoint folder and the tokenizer folder to the folder of this repository:

  ln -s /folder/of/LLaMA-2-7B/checkpoint ./LLaMA-7B-2
  ln -s /folder/of/LLaMA-2-7B/tokenizer ./Llama-2-7b-hf

• Remember to replace the directory /folder/of/LLaMA-2-7B/checkpoint and /folder/of/LLaMA-2-7B/tokenizer with actual directories!
• The four graph reasoning datasets are available on Google Drive or Huggingface. You may download it and place the zip file in the directory of this repository. And then run the command:

  unzip dataset.zip -d ./dataset

• The directory structure should be:

  .
  |- LLaMA-7B-2
  |   |- params.json
  |   |- consolidated.00.pth
  |
  |- Llama-2-7b-hf
  |   |- tokenizer.model
  |
  |- dataset
      |- sc
      |- mts
      |- sp
      |- bgm
  
  

Get Start

Train and evaluate the model with default settings on graph reasoning datasets on GPU 0:

1. Substructure Counting

   ./scripts/sc.sh

2. Maximum Triplet Sum

   ./scripts/mts.sh

3. Shortest Path

   ./scripts/sp.sh

4. Bipartite Graph Matching

   ./scripts/bgm.sh

More hyperparameter settings are at config.py

Hyperparameter explanation:

• --n_encoder_layers number of transformer layers of textual encoder
• --n_decoder_layers number of transformer layers of textual decoder
• --n_mp_layers number of graph transformer layers
• --adapter_dim hidden dimension of textual encoder/decoder and graph transformer
• --adapter_len number of prefix tokens per LLM layer
• --rrwp graph positional encoding dimension
• --batch_size batch size in memory during training
• --grad_steps grad_step $\times$ batch_size = batch size for optimization
• --lr the learning rate
• --num_epochs number of training epochs
• --warmup_epochs number of linear warmup epochs
• --wd weight decay