Skip to content
No description, website, or topics provided.
Python HTML JavaScript CSS
Branch: master
Clone or download
Latest commit 5486d80 Nov 11, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
inspector Add demo video Nov 11, 2019
linspector Add a sitemap Jun 24, 2019
.gitignore Add PyCharm May 30, 2019
LICENSE.txt Add license Jul 22, 2019
README.md Update publications Nov 11, 2019
manage.py Initial commit Apr 20, 2019
requirements.txt Bump packages Nov 11, 2019

README.md

LINSPECTOR WEB

LINSPECTOR (Language Inspector) is a multilingual inspector to analyze word embeddings in a web based application. Our goal is to provide researchers with an easily accessible tool to gain quick insights into their word embeddings especially outside of the English language. To do this we employ simple classification tasks called probing tasks for a diverse set of languages.

linspector.ukp.informatik.tu-darmstadt.de

Citation

Please use the following citation:

@inproceedings{eichler-etal-2019-linspector,
    title = "{LINSPECTOR} {WEB}: A Multilingual Probing Suite for Word Representations",
    author = {Eichler, Max  and
      {\c{S}}ahin, G{\"o}zde G{\"u}l  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-3022",
    doi = "10.18653/v1/D19-3022",
    pages = "127--132"
}

Abstract: We present LINSPECTOR WEB, an open source multilingual inspector to analyze word representations. Our system provides researchers working in low-resource settings with an easily accessible web based probing tool to gain quick insights into their word embeddings especially outside of the English language. To do this we employ 16 simple linguistic probing tasks such as gender, case marking, and tense for a diverse set of 28 languages. We support probing of static word embeddings along with pretrained AllenNLP models that are commonly used for NLP downstream tasks such as named entity recognition, natural language inference and dependency parsing. The results are visualized in a polar chart and also provided as a table. LINSPECTOR WEB is available as an offline tool or at https://linspector.ukp.informatik.tu-darmstadt.de.

Contact Person: Gözde Gül Şahin, sahin@ukp.informatik.tu-darmstadt.de

https://www.ukp.tu-darmstadt.de

https://www.tu-darmstadt.de

Overview

  • inspector/ is a Django application structured similar to the official documentation.
  • inspector/nn/ contains an AllenNLP based evaluation suite.

Installation

LINSPECTOR is hosted at linspector.ukp.informatik.tu-darmstadt.de but you can also run a local copy.

  1. Clone this repository.

     git clone https://github.com/UKPLab/linspector-web.git
    
  2. Create a virtual environment using Python 3.6.x.

     pip install virtualenv
     cd linspector-web/
     # As of Python 3.7.3 there is a bug using Eventlet with pathlib
     virtualenv linspectorenv -p python3.6
     source linspectorenv/bin/activate
    
  3. Install requirements.

     pip install -r requirements.txt
    
  4. Run migrations and load fixtures.

     ./manage.py migrate
     ./manage.py loaddata languages probing_tasks
    
  5. Download Bootstrap (4.3) Sass files to inspector/static/inspector/bootstrap/scss/.

  6. Compile Sass file.

     npm install sass postcss-cli autoprefixer
     sass --no-source-map inspector/static/inspector/scss/custom.scss inspector/static/inspector/custom.css
     npx postcss inspector/static/inspector/custom.css --use autoprefixer --replace
    
  7. Install a Celery supported broker, we use RabbitMQ with Eventlet as an execution pool.

  8. Add training data to media/intrinsic_data/ (see Intrinsic Data below).

  9. Start the server (activate virtualenv for Celery and Django).

     rabbitmq-server
     celery -A linspector worker -l info -P eventlet
     ./manage.py runserver
    
  10. Open localhost:8000 in your browser.

Notes

When using Celery without Eventlet as an execution pool there can be issues running AllenNLP.

In a production environment you might encounter performance and / or stability issues using SQLite. We recommend using PostgreSQL.

Probing Tasks

Probing tasks are simple classification tasks aiming to gain insights into information encoded inside embeddings. Our work focuses on word embeddings but should be extendable to other embedding types.

See Conneau et al. (2018), or Şahin et al. (2019) to learn more.

Intrinsic Data

For training data each line consists of a token and a label separated by whitespace e.g. Klammeraffe Noun.

We are using intrinsic data provided by Şahin et al. (2019) with a modified folder structure:

  • Probing tasks are title cased without spaces
  • Use ISO 639-1 codes for languages
  • Folder names have to match database entries except for the spaces
  • media/intrinsic_data/CaseMarking/de/ > train.txt, dev.txt, test.txt

Additionally we renamed some task to be more descriptive:

  • Case to Case Marking
  • Tag Count to Morphological Feature Count
  • Odd Feat to Odd Morphological Feature
  • Pseudo to Pseudoword
  • Same Feat to Shared Morphological Feature
  • Character Bin to Word Length

Also we deleted Character Count. It was replaced by Word Length.

We have attached our fixtures under inspector/fixtures/.

Contrastive Tasks

Odd Morphological Feature and Shared Morphological Feature are contrastive tasks trying to predict a single odd or shared morphological feature between two tokens.

For training data each line consists of two tokens and a label all separated by whitespace e.g. ruckelte getoastet Tense.

A boolean flag contrastive has to be set in the database for each contrastive task.

Commands

There is a command-line script experiment to probe static embeddings files which is designed to work with the intrinsic evaluation of Şahin et al. (2019).

To probe a single embeddings file per language run:

# Folders structured like media/experiment/ar/embeddings.vec
./manage.py experiment ar hy cs fr hu

To probe different embedding types run:

# Folders structured like media/experiment/ar/fasttext/embeddings.vec
./manage.py experiment ar hy cs fr hu --types bpe fasttext w2v

To probe different embedding dimensions run:

# media/experiment/ar/fasttext/100/embeddings.vec
./manage.py experiment ar hy cs fr hu --types bpe fasttext w2v --dims 50 100 200 300

The metrics will be written to a CSV file for each language e.g. media/experiment/ar.csv.

Run ./manage.py help experiment for additional arguments.

You can’t perform that action at this time.