Compare different techniques for reducing the co-occurrences graph

[albertocottica]

Reduction techniques:

1. The usual: filter by number of co-occurrences
2. Filter by number of co-occurrences, but now co-occurrences in the same thread count as one.
3. Filter by number of co-occurrences, but weigh for number of nodes in the largest connected component in the social network associated to each co-occurrence.
4. Filter by number of co-occurrences, but weightby eigenvector centrality of proponents on the connection.

Method for comparison:

For each technique:

• Apply reduction until number of nodes is ~ 50
• Examine visually
• Apply community detection algo

Finally:

• compare and contrast
• compute the overlap: average probability that a code is in the graph reduced with technique i, conditional to it being in the graph reduced with technique j. Discuss.
• compute the clustering consistency: average probability that two codes are in the same community in the graph reduced with technique i, conditional to them being in the same community in the graph reduced with technique j. Discuss.

[guywiz]

Started working on the data, using the tag co-occurence graph downloaded from GraphRyder (on Oct 4). I am surprised the co-occurence values only vary in the set {16, 23, 31, 39, 46, 54, 61, 69, 77, 84, 100}. Jason, can you confirm the data is ok?

[albertocottica]

This does not look right, no.