(ACL-2025) SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing

[ Paper 📓 ] [ SurveyBench Benchmark 🤗 ]

🤩 Tired of chaotic structures and inaccurate references in AI-generated survey paper? SurveyForge is here to revolutionize your research experience!

🔥 News

2025.05: 🎉🎉 Congratulations: SurveyForge was accepted by ACL-2025 main conference.

Introduction

Survey papers are vital in scientific research, especially with the rapid increase in research publications. Recently, researchers have started using LLMs to automate survey creation for improved efficiency. However, LLM-generated surveys often fall short compared to human-written ones, particularly in outline quality and citation accuracy. To address this, we introduce SurveyForge, which first creates an outline by analyzing the structure of human-written outlines and consulting domain-related articles. Then, using high-quality papers retrieved by our scholar navigation agent, SurveyForge can automatically generate and refine the content of the survey.

Moreover, to achieve a comprehensive evaluation, we construct SurveyBench, which includes 100 human-written survey papers for win-rate comparison and assesses AI-generated survey papers across three dimensions: reference, outline, and content quality.

🤔How to try out SurveyForge?

Due to the current limitations on API call frequency, please kindly send us an email or open an issue in the repository to inform us of the survey topic you intend to generate.

⏱️Surveyforge only takes about 10 minutes to generate a survey paper. There may be a wait time as the number of users increases, so submit your topic early!

🌟Don’t forget to click the STAR to track if your survey is ready!

Note: Our initial version currently supports survey generation only in the computer science domain, as we need to align with previous evaluation benchmarks in this field. We're actively working on expanding to other academic disciplines, and integration is already in progress. Thank you for your understanding and support!

📝Examples

Topics	Links
Multimodal Large Language Models	Comprehensive Survey on Multimodal Large Language Models Advances, Challenges, and Future Directions.pdf
3D Object Detection in Autonomous Driving	Comprehensive Survey on 3D Object Detection in Autonomous Driving.pdf
Vision Transformers	A Comprehensive Survey on Vision Transformers Foundations, Advances, Applications, and Future Directions.pdf
Generative Diffusion Models	Comprehensive Survey on Generative Diffusion Models Foundations, Innovations, and Applications.pdf
LLM-based Multi-Agent	Comprehensive Survey of Large Language Model-Based Multi-Agent Systems.pdf
Self-Supervised Learning in Computer Vision	A Comprehensive Survey on Self-Supervised Learning in Computer Vision.pdf
Embodied Artificial Intelligence	A Comprehensive Survey on Embodied Artificial Intelligence Foundations, Advances, and Future Directions.pdf
Vector Database Management Systems	A Comprehensive Survey of Vector Database Management Systems Foundations, Architectures, and Future Directions.pdf
Gradient Descent and Its Expanding Frontier	Comprehensive Survey of Gradient Descent and Its Expanding Frontier.pdf
Formal Verification of Neural Networks	Comprehensive Survey on Formal Verification of Neural Networks Foundations, Methods, and Future Directions.pdf
Edge Computing Paradigms and Technologies	A Survey on Edge Computing Paradigms and Technologies.pdf
Automated Machine Learning	Automated Machine Learning Foundations, Advancements, Applications, and Future Directions.pdf
AI in Facial Recognition	Applications of Artificial Intelligence in Facial Recognition Techniques, Challenges, and Future Directions.pdf
Natural Language Processing	Advancements in Natural Language Processing Developments, Trends, and Future Directions.pdf
Adversarial Machine Learning	Adversarial Machine Learning Attack Methods and Defense Mechanisms.pdf
Federated Learning	Federated Learning Privacy-Preserving Collaborative Machine Learning.pdf
Human-Computer Intelligent Interaction	Human-Computer Intelligent Interaction Foundations, Technologies, and Future Perspectives.pdf
AI-Powered Autonomous Scientific Discovery	AI-Powered Autonomous Scientific Discovery Challenges, Innovations, and Future Directions.pdf
LLMs in Mental Health Services	Applications of Large Language Models in Mental Health Services Capabilities, Challenges, and Future Directions.pdf
Quantum Computing Algorithms	Quantum Computing Algorithms Foundations, Advancements, and Frontier Perspectives.pdf
Quantitative Trading with AI in Cryptocurrency	A Comprehensive Survey on Quantitative Trading with Artificial Intelligence in Cryptocurrency.pdf
Large Language Models for Task-Oriented Dialogue Systems	A Comprehensive Survey on Large Language Models for Task-Oriented Dialogue Systems.pdf
Human Feedback in Video Generation	Improving Video Generation with Human Feedback A Comprehensive Survey.pdf
Efficient Video Generation	A Comprehensive Survey on Efficient Video Generation.pdf
Chinese to Malay Speech Translation Based on Fine-Tuning Large Multilingual Translation Models	A Comprehensive Survey on Chinese to Malay Speech Translation System Based on Fine-Tuning Large Multilingual Translation Models.pdf
Controllable Text Generation for Large Language Models	A Comprehensive Survey on Controllable Text Generation for Large Language Models: Methods, Challenges, and Future Directions.pdf
Synthetic Data Generation with Diffusion Models	A Comprehensive Survey on Synthetic Data Generation with Diffusion Models for Object Detection.pdf
Expressive Whole-Body Control for Humanoid Robotic Motions	Expressive Whole-Body Control for Humanoid Robotic Motions Using Model Predictive Control and Reinforcement Learning.pdf
Layout-Guided Controllable Image Synthesis	A Comprehensive Survey on Layout-Guided Controllable Image Synthesis.pdf

🕵️‍♂️How to evaluate the quality of the survey paper?

We offer SurveyBench, a benchmark for academic research and evaluating the quality of AI-generated surveys.

SurveyBench Download

Currently , SurveyBench consists of approximately 100 human-written survey papers across 10 distinct topics, carefully curated by doctoral-level researchers to ensure thematic consistency and academic rigor. The supported topics and the core references corresponding to each topic are as follows:

Topics	# Reference
Multimodal Large Language Models	912
Evaluation of Large Language Models	714
3D Object Detection in Autonomous Driving	441
Vision Transformers	563
Hallucination in Large Language Models	500
Generative Diffusion Models	994
3D Gaussian Splatting	330
LLM-based Multi-Agent	823
Graph Neural Networks	670
Retrieval-Augmented Generation for Large Language Models	608

More support topics coming soon!

🧑‍💻You can evaluate the survey by:

cd SurveyBench && python test.py --is_human_eval

Note set is_human_eval True for human survey evaluation, False for generated surveys.

If you want to evaluate your method on SurveyBench, please follow the format:

generated_surveys
|-- 3D Gaussian Splatting
    |-- exp_1
        |-- ref.json
    |-- exp_2
        |-- ref.json
...
|-- Graph Neural Networks
...

Citations

@misc{yan2025surveyforgeoutlineheuristicsmemorydriven,
      title={SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing}, 
      author={Xiangchao Yan and Shiyang Feng and Jiakang Yuan and Renqiu Xia and Bin Wang and Bo Zhang and Lei Bai},
      year={2025},
      eprint={2503.04629},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.04629}, 
}

SmartJoy-Tech/SurveyForge

ErrorLooks like something went wrong!

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Error
Looks like something went wrong!

Packages