Dataset-Cartography

Dataset Cartography for human alignment

Requirements

conda env create -f environment.yml
conda activate dc
pip install -r requirements.txt

Codebase Structure

Alignment-Data-Map/
├── src
│   ├── calculate
│   ├── config # inference and data map config 
│   ├── correlations
│   ├── figure
│   ├── score
├── README.md                  
├── requirements.txt            
└── environment.yml         

How to Run data-map

1. GPT Inference

Choose model to use and process openAI API Inference
config : ./src/config/gpt-inference-config.yaml

python gpt-inference.py

2. Plotting the Alignment Data Map

You can plot the Alignment Data Map by following a 3-step process:

1. score
2. calculation
3. figure

Each step corresponds to a specific task.
Set the objective to one of the following to execute the corresponding task:

• score: compute similarity scores
• calculation: compute mean and variance statistics
• figure: generate the final visualization

Specify the desired objective in the config file (default is 'score'), and then run the following command:

python main.py

Config path: ./src/config/main-config.yaml