Towards Uncertainty Aware Language Agent

This is the official repo for the paper: Towards Uncertainty-Aware Language Agent.

💡 Introduction

Language Agents utilising Large Language Models to interact with the external world (e.g., through tools) to process collected observations towards solving a task have achieved great improvements in challenging reasoning tasks. A more effective design for language agents should have a better interplay between the implicit knowledge encoded in LLM's weight and the explicit knowledge of the external world. To this end, we present Uncertainty-Aware Language Agent that integrates uncertainty in language agent's cycle of Thought, Action, and Observation. The uncertainty moderates the interaction between the LLM and the external world, facilitating a more effective and efficient dynamic.

💬 Examples

🛠️ Setup

Configure Environment:

pip install -r requirements.txt

Configure OpenAI API (for GPT3.5):

export OPENAI_API_KEY=<YOUR_KEY>

Configure Google Search API (for MMLU):

export SERPAPI_API_KEY=<YOUR_KEY>

🚀 Quick Start

Free-from Question Answering (HotpotQA)

• GPT3.5: python run_hotpotqa_gpt3.5.py
• LLaMA2-70B: python run_hotpotqa_llama2.py

Binary Question Answering (StrategyQA)

• GPT3.5: python run_strategyqa_gpt3.5.py
• LLaMA2-70B: python run_strategyqa_llama2.py

Multiple Choice Question Answering (MMLU)

• GPT3.5: python run_mmlu_gpt3.5.py
• LLaMA2-70B: python run_mmlu_llama2.py

For different settings (standard, cot, react, uala), change the mode variable in script. The uala setting here is UALA-S+Backoff, to use Oracle, set the oracle variable in script as True.

🎯 Results

Citation

@misc{han2024uncertaintyaware,
      title={Towards Uncertainty-Aware Language Agent}, 
      author={Jiuzhou Han and Wray Buntine and Ehsan Shareghi},
      year={2024},
      eprint={2401.14016},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}