Task Alignment

A framework for evaluating LLM assistants on task alignment — the ability to align with users on their intended tasks.

Overview

Installation

pip install -e .

Requires Python 3.10+. See pyproject.toml for dependencies.

Quickstart

from task_alignment import TaskAlignmentEnv

env = TaskAlignmentEnv(task="shopping", ...)   # TODO
obs, info = env.reset()

Repository Layout

task_alignment/       core package
  env.py              POMDP env (Gymnasium-compatible)
  config.py           EnvConfig dataclass
  metrics.py          CommitReward, TotalIG, CommitAcc@k, ...
  llm_utils.py        LLM call helpers (retries, key fallback)
  adapters/           per-domain adapters (shopping, gdpval, terminal_bench)
  user/               user simulator + intent evolver
  ranker/             fork-ranker + score-vector MCQ probe
  prompts/            per-domain prompt templates

benchmark/            batch runner scripts
data/                 dataset placeholders + download scripts
docs/                 documentation

Datasets

https://huggingface.co/datasets/daiandy/task-alignment-dataset

Citation

@article{dai2026interactivetaskalignmentpomdp,
      title={Interactive Task Alignment as a POMDP}, 
      author={Andy Dai and Zexue He and Zhenyu Zhang and Alex Pentland and Jiaxin Pei},
      year={2026},
      eprint={2607.16412},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.16412}, 
}

License

MIT — see LICENSE.