Skip to content
No description, website, or topics provided.
Python Jsonnet Shell
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.
data_util Added contextual probing tasks Dec 23, 2019
extrinsic DEP config cleanup Apr 4, 2019
intrinsic Added contextual probing tasks Dec 23, 2019
.gitignore data_util package Mar 20, 2019
LICENSE.txt Create LICENSE.txt Jul 26, 2019
NOTICE.txt Update NOTICE.txt Jul 26, 2019 Added contextual probing tasks Dec 23, 2019 data_split package Mar 20, 2019


Language Inspector (LINSPECTOR) is an effort to interpret the multilingual black box NLP models. With this work, we release an easy-to-use framework to help researchers,especially the ones that are interested in world languages, understand their word representations better.

  • We have created and released 15 probing tasks for 24 languages.

    • If you only want to download the probing datasets, go to intrinsic/data
    • If you want to create your own probing tasks for other languages or with different settings, see data_split/README
    • If you want to evaluate your new embeddings, or the intermediate representations extracted from a black box model, e.g., neural dependency parser, see the instructions in intrinsic/evaluation/README
  • We have also created 20 contextual probing tasks for 49 languages. Those probing datasets are stored at intrinsic/data_contextual

  • We have compiled and preprocessed (when necessary) dataset for universal part-of-speech tagging, dependency parsing, semantic role labeling, natural language inference and named entity recognition for Turkish, Finnish, German, Spanish and Russian. We provide the necessary data splits (when available) or guide you how to process the downloaded files with our preprocessing scripts.

    • If you want to evaluate your new embeddings on the downstream tasks, we provide config files that are ready to run with AllenNLP.
    • Check extrinsic/README for more details.


Please use the following citation:

  author    = {G{\"{o}}zde G{\"{u}}l {\C{S}}ahin and
               Clara Vania and
               Ilia Kuznetsov and
               Iryna Gurevych},
  title     = {{LINSPECTOR:} Multilingual Probing Tasks for Word Representations},
  journal   = {CoRR},
  volume    = {abs/1903.09442},
  year      = {2019},
  url       = {},
  archivePrefix = {arXiv},
  eprint    = {1903.09442},
  timestamp = {Mon, 01 Apr 2019 14:07:37 +0200},
  biburl    = {},
  bibsource = {dblp computer science bibliography,}

Abstract: Despite an ever growing number of word representation models introduced for a large number of languages, there is a lack of a standardized technique to provide insights into what is captured by these models. Such insights would help the community to get an estimate of the downstream task performance, as well as to design more informed neural architectures, while avoiding extensive experimentation which requires substantial computational resources not all researchers have access to. A recent development in NLP is to use simple classification tasks, also called probing tasks, that test for a single linguistic feature such as part-of-speech. Existing studies mostly focus on exploring the information encoded by the sentence-level representations for English. However, from a typological perspective the morphologically poor English is rather an outlier: the information encoded by the word order and function words in English is often stored on a subword, morphological level in other languages. To address this, we introduce 15 word-level probing tasks such as case marking, possession, word length, morphological tag count and pseudoword identification for 24 languages. We present experiments on several state of the art word embedding models, in which we relate the probing task performance for a diverse set of languages to a range of classic NLP tasks such as semantic role labeling and natural language inference. We find that a number of probing tests have significantly high positive correlation to the downstream tasks, especially for morphologically rich languages. We show that our tests can be used to explore word embeddings or black-box neural models for linguistic cues in a multilingual setting.

Contact person:

Gözde Gül Şahin,

Please send us an e-mail or report an issue, if something is broken or if you have further questions.

You can’t perform that action at this time.