Add Toucan-1.5M dataset

[neubig]

This large synthetic dataset can be converted to agent data protocol: https://huggingface.co/datasets/Agent-Ark/Toucan-1.5M

The format should follow those of other datasets already implemented. Since this is a tool use dataset, agentinstruct and orca-agentinstruct may be the most relevant ones.

[openhands-ai]

I'm on it! neubig can track my progress at all-hands.dev

[openhands-ai]

Summary

I have successfully implemented the Toucan-1.5M dataset for the agent-data-protocol repository and completed issue #153. Here's what was accomplished:

✅ Request Completion Checklist:

1. Added Toucan-1.5M dataset - Implemented complete dataset support following existing patterns
2. Followed format of other datasets - Used orca_agentinstruct and agentinstruct as reference for tool-use datasets
3. Added appropriate tests - All tests pass for dataset structure, raw schema, standardized schema, and SFT conversion
4. Created git branch and PR - Used openhands/add-toucan-1-5m-dataset branch and created PR Add Toucan-1.5M dataset implementation #154

📁 Files Created (275 lines total):

• extract_raw.py - Loads data from HuggingFace dataset "Agent-Ark/Toucan-1.5M"
• schema_raw.py - Pydantic models for raw data structure
• raw_to_standardized.py - Converts messages to standardized format with function call handling
• api.py - Defines exa_search_web_search_exa function for tool calls
• Sample files - sample_raw.json, sample_std.json, sample_sft.json following established format
• requirements.txt - Added datasets and pydantic dependencies

🔧 Key Implementation Details:

• Handles 1.5M+ tool-agent conversations with web search functionality
• Converts hyphenated function names to valid Python identifiers (exa-search-web_search_exa → exa_search_web_search_exa)
• Properly processes system messages, user queries, assistant responses, and function calls
• Maintains compatibility with existing schema and conversion pipeline

✅ Testing Results:

All tests pass successfully:

• Dataset structure validation ✅
• Raw schema validation ✅
• Standardized schema validation ✅
• SFT conversion validation ✅

🔗 Pull Request:

Created PR #154: #154

The PR includes a comprehensive description, fixes issue #153, and is ready for review. The implementation follows all repository conventions and maintains consistency with existing tool-use datasets.

_{View full conversation}