Bias-Amplification and Bias-Elimination Framework for Debiased Cognitive Diagnosis

As one of the fundamental technologies in student modeling and intelligent education, Cognitive Diagnosis~(CD) aims to identify the student proficiency level over knowledge concepts through historical student exercise logs. Plenty of methods have been proposed to fully exploit student exercise logs and achieve impressive student modeling. However, since exercise logs have \textit{inherent bias}~(i.e., top students have more correct answers and weak students have more incorrect answers, termed \textit{bias-aligned logs}), those data-driven CD models may prefer to overly rely on students' historical answer accuracy rates for predictions, overestimating top students and underestimating weak students. To tackle this problem, in this paper, we design a novel BABE for debiased cognitive diagnosis. Specifically, given that bias-aligned logs are the majority in the data and have easy-to-learn patterns, we first use mixup operation to amplify the inherent bias to train a biased model. Then, we leverage the outputs of biased model to weight the exercise logs and use bias-aligned logs to enrich the diversity of bias-conflicting logs, which are used to train the debiased model. Finally, we jointly train the biased and debiased models with a novel weighted optimization target. Extensive experiments on advanced CD models over multiple real-world datasets demonstrate the superiority of BABE.

Install

The current implementation is based on EduStudio library that is a Unified and Templatized Framework for Student Assessment Models including Cognitive Diagnosis(CD) and Knowledge Tracing(KT).

  pip install -r requirments.txt

Run

Running the MIRT Model on the FrcSub Dataset under OOD Settings

1. Run the vanilla MIRT model:

   python Vanilla.py
   

2. Run the MIRT model using the BABE pipeline:

   python BABE.py
   

Below is a detailed introduction to each script

Vanilla.py

This script runs a baseline model using the FrcSub dataset under Out-Of-Distribution (OOD) settings.

import sys
import os

sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/../")
os.chdir(os.path.dirname(os.path.abspath(__file__)))

from edustudio.quickstart import run_edustudio
from module.datatpl.base_datatpl import BABEDataTPL
from module.evaltpl.prediction_evaltpl import PredictionEvalTPL

# backbone models
from module.model.irt_babe import IRT_BABE
from module.model.mirt_babe import MIRT_BABE
from module.model.ncdm_babe import NCDM_BABE
from module.model.kancd_babe import KaNCD_BABE
from module.model.kscd_babe import KSCD_BABE

# baseline methods
from module.traintpl.Vanilla_traintpl import VanillaTrainTPL
from module.traintpl.REW_traintpl import REWTrainTPL
from module.traintpl.LfF_traintpl import LfFTrainTPL
from module.traintpl.SelectMIX_traintpl import SelectMIXTrainTPL
from module.traintpl.GA_traintpl import GATrainTPL
from module.traintpl.BABE_traintpl import BABETrainTPL

run_edustudio(
    dataset="FrcSub",  # Available datasets: [FrcSub, ASSIST_0910, SLP-Math..] at https://edustudio.ai
    cfg_file_name=None,
    traintpl_cfg_dict={
        "cls": VanillaTrainTPL,  # Training templates: [Vanilla, REW, LfF, SelectMix, GA, BABE]
        "batch_size": 1024,
        "lr": 0.005,
        "balance": True,  # Enables OOD setting when True, otherwise IID
    },
    datatpl_cfg_dict={
        "cls": BABEDataTPL,
    },
    modeltpl_cfg_dict={
        "cls": MIRT_BABE,  # Model backbones: [IRT_BABE, MIRT_BABE, NCDM_BABE, KSCD_BABE, KaNCD_BABE]
    },
    evaltpl_cfg_dict={
        "clses": [
            PredictionEvalTPL,
        ]
    },
)

BABE.py

This script runs the MIRT model with the BABE pipeline on the ASSIST_0910 dataset, also under OOD settings.

import sys
import os

sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/../")
os.chdir(os.path.dirname(os.path.abspath(__file__)))

from edustudio.quickstart import run_edustudio
from module.datatpl.base_datatpl import BABEDataTPL
from module.evaltpl.prediction_evaltpl import PredictionEvalTPL

# models
from module.model.irt_babe import IRT_BABE
from module.model.mirt_babe import MIRT_BABE
from module.model.ncdm_babe import NCDM_BABE
from module.model.kancd_babe import KaNCD_BABE
from module.model.kscd_babe import KSCD_BABE

# method
from module.traintpl.Vanilla_traintpl import VanillaTrainTPL
from module.traintpl.REW_traintpl import REWTrainTPL
from module.traintpl.LfF_traintpl import LfFTrainTPL
from module.traintpl.SelectMIX_traintpl import SelectMIXTrainTPL
from module.traintpl.GA_traintpl import GATrainTPL
from module.traintpl.BABE_traintpl import BABETrainTPL

run_edustudio(
    dataset="SLP-Math",  # Available datasets: [FrcSub, ASSIST_0910, SLP-Math..] at https://edustudio.ai
    cfg_file_name=None,
    traintpl_cfg_dict={
        "cls": BABETrainTPL,  # Training templates: [Vanilla, REW, LfF, SelectMix, GA, BABE]
        "batch_size": 1024,
        "lr": 0.005,
        "balance": True,  # Enables OOD setting when True, otherwise IID
    },
    datatpl_cfg_dict={
        "cls": BABEDataTPL,
    },
    modeltpl_cfg_dict={
        "cls": MIRT_BABE,  # Model backbones: [IRT_BABE, MIRT_BABE, NCDM_BABE, KSCD_BABE, KaNCD_BABE]
    },
    evaltpl_cfg_dict={
        "clses": [
            PredictionEvalTPL,
        ]
    },
)