Carleton College Computer Science Comps
Advisor: Sneha Narayan
Students: Jackie Chan, James Gardner, Junyi Min, Kirby Mitchell
Source: http://www.cs.carleton.edu/cs_comps/1920/wikipedia/index.php
Paper: https://doi.org/10.1145/2998181.2998232
Creates dendrograms based on hierarichal clustering from correlation values based on pageviews, revisons, and pageview-revisons. NOTE: must run correlation.py
first to get the calculated values.
Contains code that generates pageviewCorr.csv
, revisionCorr.csv
, and pageview-revisionCorr.csv
(top 5). Also allPageviewCorr.csv
, allRevisionCorr.csv
, and allPRCorr.csv
that contain all the correlation data for hierarchical clustering. Uses the data from pull_data.py
and wmlflabs.py
to do the analysis; returns Pearson's coefficient using the built-in Pandas analysis.
Plots unique editors aggregated across corpus - monthly unique editors across corpus. NOTE: must run pull_data.py
first for making plots.
Contains the code that generates the 10 Year Updated.csv
file that concatenates all the .csv files in 10 Year Revision Data
.
Plots fraction of top 10 articles' editors by revison count that edit a page on the non-top 10 articles on daily basis.
Plots aggregate number of edits, editors, and pages edited for different time intervals.
Plots pagecount vs number of editors with normalization based on the number of pages being actively editied.
Plots Jaccard similarity scores for the top 10 articles by revision count in a single plot. Plots individual Jaccard similarity scores for articles.
Plots the median edit size for the top 10 articles by revision count for 14 day periods.
Plots revision activity for the top 10 articles by revision count on a single figure.
Plots pageviews for articles: top 4 most correlated, single pages, and aggregate plots.
Contains all the code that pulls the relevant, necessary data for analysis. This includes: creation/creation_dates.csv
, 10 Year Redirect Data/*
, and 10 Year Revision Data/*
. If data needs to be updated, delete the relevant file and rerun the relevant helper function to pull that data. These user-facing functions include: generate_creation_dates_data()
, generate_revision_data()
, and generate_redirect_data()
.
Note: main.py
access MediaWiki API using a Wiki bot user account with login credentials stored in credentials.txt
which is excluded via .gitignore
. The bot user allows authenciated requests to the MediaWiki API for less restricted access. Follows guidelines here: https://www.mediawiki.org/wiki/API:Login
Manually reformats downloaded pageview file data from the WikiMedia Foundation Labs tool (WMF labs).
List of required Python packages to run files in the repository.
Contains helper functions for use by multiple files.
Generates a table that reviews the top 10 articles by revision count and the totals for the top 10 and the entire corpus.
Contains the code that generates the day_difference_figure.png
file. It generates this data with the creation_dates_analysis.csv
file which contains the event start dates and their page creation dates that were manually inputted and cited in the start_dates_doc.md
.
Contains all the Wikipedia articles in our analysis.
Helper code that automates the downloading and reformatting of pageview data from the WikiMediaFoundation Labs tool (WMF). Uses titles.txt
for reference. Utiles the Selenium library with a Linux Firefox geckodriver for web rendering.