Group 18 AmbiStory Implementation

This folder contains Seyyid Ali's core implementation for SemEval-2026 Task 5. It is designed to run as a standalone directory. The official JSON data is kept under data/raw/.

Expected data files:

• data/raw/train.json
• data/raw/dev.json
• data/raw/test.json

Setup

cd OurImplementation
python3 -m pip install --user -r requirements.txt

If the DeBERTa model is not already cached on the machine, run the commands without HF_HUB_OFFLINE=1 the first time so Hugging Face can download microsoft/deberta-v3-base.

Verified Commands

PYTHONPATH=. python3 -m group18.src.sanity_check
PYTHONPATH=. HF_HUB_OFFLINE=1 python3 -m group18.src.run_baselines --local-files-only
PYTHONPATH=. HF_HUB_OFFLINE=1 python3 -m group18.src.run_step3_regression --local-files-only --batch-size 16
PYTHONPATH=. HF_HUB_OFFLINE=1 python3 -m group18.src.run_step4_ranking --local-files-only --batch-size 8 --val-batch-size 16
PYTHONPATH=. HF_HUB_OFFLINE=1 python3 -m group18.src.run_step5_attention --local-files-only --batch-size 4 --val-batch-size 8
PYTHONPATH=. HF_HUB_OFFLINE=1 python3 -m group18.src.run_step5_joint_attention --local-files-only --batch-size 4 --val-batch-size 8
PYTHONPATH=. python3 -m group18.src.train_step3_smoke --train-size 2 --val-size 2 --epochs 1 --batch-size 1 --max-length 64
PYTHONPATH=. python3 -m group18.src.attention_smoke --val-size 1

The smoke commands use microsoft/deberta-v3-base. Full training should use a GPU and the notebooks in group18/notebooks/.

On a 12GB GPU, reduce Step 3/4/5 batch sizes if CUDA runs out of memory. On a larger GPU such as A100 80GB, the batch sizes shown above are the intended starting point.

run_step5_joint_attention is the joint-encoding attention variant. It keeps the Step 3/4 story-sense encoding and extracts sentence attention from the joint hidden states.

Main Files

• group18/src/data_loader.py: official-data loading, regression instances, pairwise instances, dataloaders.
• group18/src/model.py: DeBERTa regression model and sentence-attention model.
• group18/src/losses.py: MSE and combined MSE + pairwise ranking loss.
• group18/src/evaluate.py: Spearman, official accuracy@std, MSE, ranking accuracy, JSONL export.
• group18/src/train.py: regression and pairwise training loops.
• group18/src/attention.py: sentence splitting, attention datasets, attention extraction, heatmap plotting.