paper link: https://arxiv.org/abs/2305.10037
Check out the NLGraph dataset! main.json in each task features a supervised fine-tuning setting and is divided into train.json and test.json. The graph directory in each task contains the raw graph data (i.e. graph represented by numbers) divided into easy, (medium) and hard. In the raw graph data, the first two numbers in the first line are the number of nodes and edges and the following lines give the edges and queries.
Environment: conda env create -f environment.yml would generate a conda environment called NLGraph that should be able to run the code.
First set your openai key OEPNAI_API_KEY (and openai organization OPENAI_ORGANIZATION optionally):
$env:OPENAI_API_KEY="your openai key" # for Windows powershell
export OPENAI_API_KEY="your openai key" # for Linux
then run the evaluation code for a specific task:
python evaluation/<task_name>.py --model <name of LM> --mode <difficulty_mode> --prompt <prompting technique> --T <temperature> --token <max number of token> --SC <whether to use self-consistency> --SC_num <sampling number for SC>
For instance,
python evaluation/cycle.py --model text-davinci-003 --mode easy --prompt CoT --SC 1 --SC_num 5
evaluates text-davinci-003 model on the easy subset of cycle task, using chain-of-thought prompting together with self-consistency.
More complete repo coming soon...
If you find this repo useful, please cite our paper:
@inproceedings{
wang2023can,
title={Can Language Models Solve Graph Problems in Natural Language?},
author={Heng Wang and Shangbin Feng and Tianxing He and Zhaoxuan Tan and Xiaochuang Han and Yulia Tsvetkov},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
url={https://openreview.net/forum?id=UDqHhbqYJV}
}