FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

FinToolBench is a real-world, runnable benchmark for evaluating financial tool-using agents. It pairs 760 executable financial tools with 295 tool-required queries and measures not only execution success but also finance-critical compliance dimensions, including timeliness, intent type, and regulatory domain alignment. We also provide FATR, a finance-aware tool retrieval and reasoning baseline that improves stability and compliance. FinToolBench enables auditable financial tool execution and provides a reproducible evaluation protocol for finance-aware agents.

Overview (This Release)

This open release contains the evaluation pipeline, RapidAPI subscription helper, and minimal data required to reproduce results. Full agent training/build scripts and other internal components are not included.

Highlights

• Runnable tool library: 760 free-tier financial tools (RapidAPI + AkShare), normalized into a unified schema.
• Tool-required questions: 295 questions (166 single-tool, 129 multi-tool) adapted from FinanceBench and OpenFinData.
• Trace-level evaluation: capability metrics plus compliance mismatch rates from explicit finance attributes.
• Baseline: FATR (Finance-Aware Tool Routing) with attribute injection, caching, retries, and output compression.

Benchmark Contents

Tool Inventory

• Sources: RapidAPI endpoints and AkShare interfaces.
• Filtering: executability checks, deduplication, and authentication feasibility.
• Normalization: canonical parameter names, consistent date formats, and structured tool signatures.
• Finance attributes: timeliness, intent_type, and regulatory_domains for each tool.

Key files:

• tools/tools_all_annotated.jsonl (tool manifest + finance tags)

Question Set

• Sources: FinanceBench (HF) and OpenFinData release.
• Selection: tool-required only, capped length, top-K tool retrieval, multi-sample verification.
• Composition: 166 single-tool + 129 multi-tool questions.

Key files:

• data/question/select_data_real_remove_duplicates.jsonl (benchmark questions)

Evaluation

Each run produces a tool trace and a final answer. Evaluation separates capability, correctness, and compliance:

• Capability:
  • Tool Invocation Rate (TIR)
  • Tool Execution Success Rate (TESR)
  • Conditional Execution Rate (CER)
  • SoftScore (LLM judge / exact match),
  • CSS (correctness conditioned on successful execution)
• Compliance mismatch:
  • Timeliness (TMR), Intent (IMR), Domain (DMR)

Baseline: FATR

FATR injects finance attributes into tool cards and stabilizes execution via caching, retries, and output compression. It is designed as a lightweight, reproducible baseline for finance-aware tool routing.

What This Repo Includes (Open Parts)

This release is limited to the evaluation pipeline, subscription helper, and minimal tool data needed to reproduce results.

Code

• code_bench/evaluate/ evaluation pipeline
• code_bench/tools/tools_rapidapi_subscribe_url.py subscribe from tools/home_url.json
• code_bench/utils/model_requests.py model request wrapper used by evaluation

Data

• data/question/select_data_real_remove_duplicates.jsonl benchmark questions
• tools/tools_all_annotated.jsonl tool manifest + finance tags
• tools/home_url.json RapidAPI home URLs for subscription

Other folders are kept internal.

Execution Steps

1. Install dependencies:

pip install -r requirements.txt

2. Subscribe RapidAPI tools from tools/home_url.json (requires your RapidAPI account login):

python -u code_bench/tools/tools_rapidapi_subscribe_url.py

3. Configure LLM API (set your model endpoint and credentials in the code or environment as required).

4. Run evaluation:

python -u code_bench/evaluate/run_relative_eval.py \
  --inputs data/result/result/result_model_name_full.jsonl \
  --output_dir data/eval/relative_model

This writes:

• <output_dir>/<setting>_results.jsonl
• <output_dir>/<setting>_metrics.json
• <output_dir>/all_metrics.json

RapidAPI Subscription (Optional)

Subscribe only the APIs listed in tools/home_url.json:

python -u code_bench/tools/tools_rapidapi_subscribe_url.py

If you want to see the browser UI, do not pass --headless.

Prerequisite: you must register and log in to your own RapidAPI account before running this script. The script relies on your RapidAPI login cookies to subscribe APIs.

Notes

• Tool execution can drift over time; report evaluation date and tool manifest version when publishing results.
• Some RapidAPI endpoints may change or become unavailable over time.

Citation

If you use this benchmark in academic work, please cite the FinToolBench paper.