🙈 Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments 💬 🔨

This repository contains code and data for the paper "Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments".

Table of Contents:

• Repository Layout
• Licensing
• Contact

Repository Layout

1. notebooks
   1. dataset_assets.ipynb: Pulling and integrating the ground truth data (*).
   2. preprocessing_and_sampling.ipynb: For pre-processing, sampling, examples and creating data distributions.
   3. classification.ipynb: To execute all experiments.
   4. results.ipynb: For generating the results.
   5. generate_tables.ipynb: For generating the tables.
2. output
   1. classification: Results of each experiment performed.
      1. train-one-one-test-on-another: The results for research question Q2.
      2. to-what-transformers-pay-attention: The results for research question Q3.
      3. leave-one-out: Results of the additional experiment as mentioned in the appendix.
   2. images: Final visualizations of the results.
   3. statistics: Numerical description of dataset features.
3. datasets.csv: Details about all 52 datasets considered.

Note

(*) For TACO and UKP, you need to create an assets folder to store the requested data. Follow the instructions in this notebook to place the folder and files correctly.

Licensing

Limited-Generalizability by Marc Feger is licensed under CC BY-NC-SA 4.0

Contact

If you have any questions, don't hesitate to reach out to marc.feger@hhu.de, katarina.boland@hhu.de or stefan.dietze@gesis.org.