Pairwise Ranking Aggregation by Non-interactive Crowdsourcing with Budget Constraints
If you find this repository helpful, please cite our paper Pairwise Ranking Aggregation by Non-interactive Crowdsourcing with Budget Constraints.
@inproceedings{cai2017pairwise,
title={Pairwise Ranking Aggregation by Non-interactive Crowdsourcing with Budget Constraints},
author={Cai, Changjiang and Sun, Haipei and Dong, Boxiang and Zhang, Bo and Wang, Ting and Wang, Hui},
booktitle={2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)},
pages={2567--2568},
year={2017},
organization={IEEE}
}
Code Usage:
The main() function is provided in the main_SATD.cpp
file. The experiments in our ICDCS 2017 paper are finished by this function.
Input Parameters:
Note that all the parameters are string variables, which will be automatically changed into appropriate variables (like, string to double via function stod(), string to int via stoi(), etc.).
-
baseAddress, an directory storing the task assignment graphs, each of which is saved in a "txt" file. E.g., =
E:/OpenCVProjects_CCJ/CrowdSourcing2/ICDE-2017/GT/GT-v100/
. Please pay attention the last "/" which is necessary. -
k, finding top-k paths using TA or SA. E.g., k = 3, meaning to return top-3 optimum paths.
-
vertexNum, number of the vertex, this value can be set any initial value, since it be assigned later. E.g., vertexNum = 20;
-
d, degree of the graph, this value can be set any initial value, since it be assigned later. E.g., d = 5.
-
T, temperature used in the Simulation Annealing algorithm. E.g., T = 50000.
-
coolRate, the rate of cooling, still used in the Simulation Annealing algorithm. E.g., coolRate = 0.95.
-
IterationNum, the predefined iteration times for Simulation Annealing algorithm, E.g., IterationNum = 1000.
-
ExperimentTimes, meaning how many times we will do the experiments for getting an averaged result. E.g, IterationNum = 10.
-
distribuion_type, there are two types of distribution used to generate the simulation dataset considering the variety of the workers' qualities. distribuion_type = 0, means Gaussian distribution; distribuion_type = 1, means uniform distribution.
-
permuStartingIdx, please set this value as 1. Actually it does not work during the experiments, it is not deleted just because I do not want to change the order of parameters too much.
-
workerNum, the number of workers for each pairwise comparison, that is, how many workers will work on the pairwise comparison (i.e., one edge). E.g., workerNum = 20, or 50.
-
ratio, selection ratio, different selection ratio means different degree in the task assignment graph. This value can be set any initial value, since it be assigned later, e.g., ratio = 0.5.
-
SA_flag, different methods to find an initial path for Simulation Annealing algorithm. SA_flag = 2 is used in our methods. The different values are explained in the function of TSPalgorithm::Run() located in the file of TSPalgorithm.cpp.
-
isDisplay, a boolean value used to print or not print information. Set it 0 (i.e., false) for disabling the printing.
-
tasks_ratio, the ratio of all the pairwise comparison tasks, e.g, ratio = 0.1, that means each worker will pick at random up to 500 * 0.1 = 50 pairwise comparison tasks to vote, given, like, 500 total pairwise comparison tasks.
-
Worker_quality, Gaussian variance to control the worker's quality.