SearchQA Data Split Configuration and max_completion_tokens Setting

[wenny823]

Thank you for this excellent project! I have two questions regarding the SearchQA environment configuration:

1. SearchQA Data Split
   I noticed in SkillOpt/configs/searchqa/default.yaml: train_size: 400
   Could you clarify how the SearchQA dataset is split for training/validation/testing? Specifically:
   Training set: 400 samples (randomly sampled?)
   Validation set: 200 samples?
   Test set: 1400 samples?
   Is this split fixed or randomly sampled each run?

2. max_completion_tokens=512 Truncation Concern
   In SkillOpt/skillopt/envs/searchqa/rollout.py, I observed: max_completion_tokens=512.
   Does this token limit frequently cause output truncation during agent rollout? Have you observed significant truncation issues in practice?

Thank you for your time and clarification!

[Yif-Yang]

Thanks for the suggestions. We will upload the details of our training / validation / testing split design. Currently, the splits are obtained by random sampling.

Regarding max_completion_tokens, we actually used 16k for all runs, including our method as well as baselines such as GEPA and TextGrad, so truncation should not be an issue. The current config is likely a misleading artifact introduced during vibe coding. In our actual experiments, this value was specified through the command line rather than taken from that config file. We will fix this misleading config as soon as possible.

[raytrun]

SearchQA splits and sample IDs are now available. All questions can be found by their IDs.

[wenny823]

Hi, thank you for sharing this update! I noticed that the SearchQA dataset I previously accessed has 6-digit numeric IDs for each question sample. The ID format you uploaded seems inconsistent with what's inside the dataset.

SearchQA dataset: https://github.com/nyu-dl/dl4ir-searchQA
Raw, split, and processed files: https://drive.google.com/drive/u/2/folders/1kBkQGooNyG0h8waaOJpgdGtOnlb1S649

[raytrun]

Hi! Thanks for your feedback. We adopted the SearchQA dataset from https://huggingface.co/datasets/lucadiliello/searchqa. The IDs we used are exactly matched with the key field in this dataset.