This repository contains the code, data, and evaluation files used in the paper "Leveraging Large Language Models for Automated Definition Extraction with TaxoMatic – a Case Study on Media Bias". The project explores using Large Language Models (LLMs) to extract, classify, and evaluate definitions related to media bias from academic literature.
Since no suitable datasets for testing definition extraction were available, we created one specifically for the media bias domain, where definitions are diverse and often unclear.
The codebase includes scripts and notebooks for automating the research process:
- SES Scraper: Searches the Semantic Scholar database for literature on media bias.
- PDF Scraping: Extracts full-text content from PDFs and converts them to machine-readable formats using GROBID.
- GROBID to df: Formats and transforms extracted content for further processing.
- Relevance Classification: Utilizes five LLMs (e.g., GPT, Claude) and eight prompting strategies to classify article relevance to media bias definitions.
- Definition Extraction: Prompts Claude 3 with five strategies to extract media bias definitions from relevant articles.
- Performance Evaluation: Evaluates classification and definition extraction processes, including manual and automated methods.
The csv and zip files contain data for:
- Manually Evaluated Articles: Ground truth data of articles assessed for relevance.
- LLM-based Relevance Classifications: Outputs from classification by different LLMs.
- Extracted Definitions: Manually and automatically extracted media bias definitions.
- Keywords: A comprehensive list of keywords for scholarly searches.
The evaluation files contain data for:
- Relevance Evaluation: Performance metrics (e.g., accuracy, F1 score) for article classification.
- Definition Evaluation: Cosine similarity analysis, error analysis, and comparisons between manual and LLM-based extractions.
The dataset includes:
- 2,398 articles annotated for relevance.
- 123 definitions manually extracted from 113 relevant articles.
- A filtered sample of 75,151 academic papers based on relevance and citation count.
- Articles filtered by citation count (≥100).
- Minimal noise due to extraction errors during PDF processing.
- No offensive content; articles were collected exclusively for scholarly research purposes.
- Data Sources: Keyword-based searches on Semantic Scholar.
- Extraction: Automated scraping scripts and GROBID for PDF-to-XML conversion.
- Annotation: Manual relevance checks by six researchers with expertise in media bias.
- PDFs converted to XML using GROBID.
- Manual annotation for relevance and definition extraction.
- Raw PDFs and processed XML files are stored for reproducibility.
- All preprocessing scripts are included in this repository.
This dataset and code were used to evaluate the TaxoMatic framework for:
- Classifying relevant articles.
- Extracting definitions.
In the future, TaxoMatic could support research on concept extraction, semantic analysis, and taxonomy building in various domains.
This repository contains annotations and analyses derived from third-party academic publications.
Copyright in the original texts remains with their respective publishers.
Any text excerpts included in the data files are provided solely for scholarly research and text-and-data-mining purposes and are not relicensed under the repository’s open-source license.
The primary contribution of this repository lies in the annotations, extracted structures, and evaluation results rather than in the redistribution of original publication texts.
- The open-source license applies to the code and original annotations created by the authors.
- Third-party academic texts are subject to their original copyright terms.
- No further regulatory restrictions are known.
We are working on an updated version of the project. For any questions, feel free to reach out to us via www.media-bias-research.org.
We hope this repository serves as a valuable resource for researchers exploring automated definition extraction and media bias!