This repository contains code used to run the experiments, and generate the figures, for the paper "Designing Rules to Pick a Rule: Aggregation by Consistency" from Ratip Emin Berker, Ben Armstrong, Vincent Conitzer, Nihar Shah".

The code may be updated as we consider new experiments. Below we document the process used to generate each figure in our paper.

If you are interested in using this code for your work we would be happy to chat with you! We encourage you to reach out before using the code so we can provide any guidance that may be useful.

Our experiments were run using Python 3.11.9. To use the code, please ensure that all packages listed in requirements.txt are installed.

Main Paper Figures

Figure 1 - Ground Truth:

The data in Figure 1 (reference distance vs split distance) is generated by running rule_comparison.py

To plot the resulting data, use plotting.py:plot_kt_distance_vs_ground_truth().

Table 1 - Peer Review Scores:

Table 1 uses the dataset described by Kerzendorf et. al in "Distributed peer review enhanced with natural language processing and machine learning" (2020).

To reproduce the data in the table run scoring_analysis.py This will run data_utils.py:parse_astronomy_csv() and pass the results to scoring_analysis.py:score_data_kt_distance()

Figure 2 - ALMA Proposals:

The data from the Atacama Large Millimeter Array which is used to generate this figure is not publicly accessible.

Figure 3 - Political Elections and F1 Races:

The data underlying this figure is found on Preflib (see the paper for additional details).

To generate the data shown in this figure run preference_data_analysis.py:evaluate_preflib_data().

The resulting data can be plotted using preflib/preflib_plots.py with the methods scatter_plot_preflib() and bar_plot_f1().

Appendix Plots

Figure 4 - Axiom Violations:

Axiom violation rates are calculated in two parts.

For generated preferences, run axioms.py:run_axiom_experiment() to generate violation data.

We omit raw Preflib data due to their file sizes. Instead we include filtered and reformatted preflib elections in ./preflib/preflib-n_profiles=1392-min_alternatives=5-max_alternatives=20-min_voters=4-max_voters=1000.csv To compute violations on preflib data, run axioms.py:test_axioms_on_saved_profiles()

To generate the figure itself run plotting.py:plot_axiom_evaluation_results()

Figure 5 - ALMA Proposals with Jaccard Distance

The data from the Atacama Large Millimeter Array which is used to generate this figure is not publicly accessible.

Figure 6 - F1 Races:

The data underlying this figure is found on Preflib (see the paper for additional details).

To generate the data shown in this figure run preference_data_analysis.py:evaluate_preflib_data().

The resulting plot is made with preflib/preflib_plots.py:bar_plot_f1().

Figure 7 - Olympics

This data is found on Kaggle at the following URL: https://www.kaggle.com/datasets/heesoo37/120-years-of-olympic-history-athletes-and-results/data

To generate the results shown in this figure run preference_data_analysis.py:evaluate_olympics_data()

The resulting plot is made with preflib/preflib_plots.py:bar_plot_olympics().

========

Note that in addition to the packages that must be installed, we also include in our code the sdopt-tearing library, found at https://github.com/baharev/sdopt-tearing.