DEPTH+: An Enhanced Depth Metric for Wikipedia Corpora Quality

In this repository, we share all the collected bot activities data for all Wikipedia editions, implementations of all depth metrics (Depth and DEPTH+), and lastly, all the produced results and figures. We also shared a few Jupyter Notebooks illustrating the metrics implementations and data collection procedures.

ACL Citation:

Saied Alshahrani, Norah Alshahrani, and Jeanna Matthews. 2023. DEPTH+: An Enhanced Depth Metric for Wikipedia Corpora Quality. In Proceedings of the Third Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 175–189, Toronto, Canada. Association for Computational Linguistics.

BibTeX Citation:

@inproceedings{alshahrani-etal-2023-depth,
    title = "{DEPTH}+: An Enhanced Depth Metric for {W}ikipedia Corpora Quality",
    author = "Alshahrani, Saied  and Alshahrani, Norah  and Matthews, Jeanna",
    booktitle = "Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)",
    month = July,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.trustnlp-1.16",
    pages = "175--189",
    abstract = "Wikipedia articles are a common source of training data for Natural Language Processing (NLP) research, especially as a source for corpora in languages other than English. However, research has shown that not all Wikipedia editions are produced organically by native speakers, and there are substantial levels of automation and translation activities in the Wikipedia project that could negatively impact the degree to which they truly represent the language and the culture of native speakers. To encourage transparency in the Wikipedia project, Wikimedia Foundation introduced the depth metric as an indication of the degree of collaboration or how frequently users edit a Wikipedia edition{'}s articles. While a promising start, this depth metric suffers from a few serious problems, like a lack of adequate handling of inflation of edits metric and a lack of full utilization of users-related metrics. In this paper, we propose the DEPTH+ metric, provide its mathematical definitions, and describe how it reflects a better representation of the depth of human collaborativeness. We also quantify the bot activities in Wikipedia and offer a bot-free depth metric after the removal of the bot-created articles and the bot-made edits on the Wikipedia articles.",
}