SocialData

Description

SocialData is a collection of data science competition environments sourced from DrivenData. It contains 5 multi-turn sandboxed tasks where agents develop machine learning models to solve real-world prediction problems spanning public health, infrastructure, natural language processing, and disaster response.

Capabilities

• Data exploration and feature engineering
• Machine learning model development
• Time series prediction
• Multi-target classification and regression
• Document summarization with LLMs

Compute Requirements

Agents are given a sandboxed environment with 1 CPU and 4 GB RAM, with access to scientific Python libraries (pandas, scikit-learn, etc.).

Tasks

There are 5 environment variants, each with a train split:

Variant	Description	Metric
FluVaccinePrediction	Predict H1N1 and seasonal flu vaccination probabilities	Mean ROC AUC
PumpItUpPrediction	Classify water pump functionality in Tanzania	F1-micro
DocSumTask	Summarize social science research papers	ROUGE-2 F1
DengAIPrediction	Predict weekly dengue fever case counts	Mean Absolute Error
RichterPrediction	Predict earthquake building damage grades	F1-micro

Reward Structure

This is a multi-turn environment. Agents explore data, develop models, generate predictions, and submit via the submit_predictions tool. Each variant uses its specific evaluation metric:

• FluVaccinePrediction: Mean ROC AUC across H1N1 and seasonal targets (0-1)
• PumpItUpPrediction: Micro-averaged F1 across 3 classes (0-1)
• DocSumTask: ROUGE-2 F1 score (0-1)
• DengAIPrediction: Inverted MAE (lower error = higher reward)
• RichterPrediction: Micro-averaged F1 across 3 damage grades (0-1)

Data

Training data is mounted read-only at /orwd_data. Each competition includes:

• Training features and labels
• Test features (labels hidden)
• Data dictionaries and descriptions

Data is sourced from DrivenData competitions and stored on the OpenReward platform.

Tools

Each variant provides CLI tools plus a submission tool:

Tool	Description
bash	Execute shell commands in the sandbox
glob	Find files by pattern
grep	Search file contents
ls	List directory contents
read	Read file contents
write	Write file contents
edit	Edit existing files
multi_edit	Make multiple edits
todo_write	Track task progress
submit_predictions	Submit predictions CSV for evaluation. Ends the episode.

Time Horizon

Multi-turn. Agents explore data, develop and train models, generate predictions, save to submission.csv, and submit for evaluation.

Environment Difficulty

[Put environment difficulty here]

Other Environment Requirements

None. All evaluation is deterministic using competition-specific metrics.

Safety

Agents in SocialData work within sandboxed environments to develop ML models. The environment does not present direct safety risks.

Citation

@software{socialdata_openreward,
  title={SocialData: DrivenData Competition Environments for OpenReward},
  author={GeneralReasoning},
  year={2025},
  url={https://openreward.ai/GeneralReasoning/SocialData}
}