Skip to content

This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish

License

Notifications You must be signed in to change notification settings

CLARIN-PL/LEPISZCZE

Repository files navigation

LEPISZCZE

This is the official code implementation for the LEPISZCZE benchmark experiments. "This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish" (NeurIPS 2022) (Łukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Marcin Wątroba, Arkadiusz Janz, Piotr Szymański, Mikołaj Morzy, Tomasz Kajdanowicz, Maciej Piasecki).

Resources

LEPISZCZE benchmark resources

Name Description URL
Leaderboard LEPISZCZE Leaderboard LEPISZCZE
Libary clarin-pl/embeddings Our library with pre-defined NLP pipelines for text classification, pair text classification and sequence labeling taks GitHub
Experiments dashboard Weight&Biases dashboard with our experiments W&B
Datasets LEPISZCZE Datasets are accessible through our HuggingFace Hub organization page. HuggingFace
KLEJ-Datasets Datasets for KLEJ benchmark are accessible through Allegro HuggingFace organization page. HuggingFace

Citation

@inproceedings{augustyniak2022lepiszcze,
 author = {Augustyniak, Lukasz and Tagowski, Kamil and Sawczyn, Albert and Janiak, Denis and Bartusiak, Roman and Szymczak, Adrian and Janz, Arkadiusz and Szyma\'{n}ski, Piotr and W\k{a}troba, Marcin and Morzy, Miko\l aj and Kajdanowicz, Tomasz and Piasecki, Maciej},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
 pages = {21805--21818},
 publisher = {Curran Associates, Inc.},
 title = {This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish},
 url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/890b206ebb79e550f3988cb8db936f42-Paper-Datasets_and_Benchmarks.pdf},
 volume = {35},
 year = {2022}
}

Contact

In case of any question or concerns about LEPISZCZE benchmark feel free to contact us:

DVC Repository Access Due to the size of pipeline outputs data, we do not provide public access to our DVC Remote Repository. However, if you are interested in any kinds of data artifacts, don't hesitate to get in touch with us.

Installation

Repository can be setup via poetry or via docker.

Requirements installation and environment setup via poetry

Prerequisites:

  • Python: 3.9+
  • Poetry [LINK].
  • CUDA 11.3+ for GPU support (Recommended)

Installation

poetry install

For GPU support

poetry run poe force-torch-cuda

Using docker image

Building image

docker build . -f docker/Dockerfile -t LEPISZCZE

After the container setup use conda env LEPISZCZE

conda activate LEPISZCZE

Reproducibility

Our experiments can be easily reproduced with DVC repro & W&B logging. Using dvc repro command and with W&B token setup.

DISCLAIMER Reproduction of full pipeline could take above 2000 hours to compelete on a single GPU device. We advise to execute stages in parallel on mutiple GPU computing devices.

Experiments

Experiments configs can be found under configs

DISCLAIMER For some of the dataset we had to limit manually maximum sequence length to 512 for Hyper Parameter Search.

Models hyperparameters configuations can be accessed via W&B dashboard. Example: [LINK]

Datasets configurations

dataset name task type input_column_name(s) target_column_name description
clarin-pl/kpwr-ner sequence labeling (named entity recognition) tokens ner KPWR-NER is a part of the Polish Corpus of Wrocław University of Technology (KPWr). Its objective is recognition of named entities, e.g., people, institutions etc.
clarin-pl/polemo2-official classification (sentiment analysis) text target A corpus of consumer reviews from 4 domains: medicine, hotels, products and school.
clarin-pl/2021-punctuation-restoration punctuation restoration text_in text_out Dataset contains original texts and ASR output. It is a part of PolEval 2021 Competition.
clarin-pl/nkjp-pos sequence labeling (part-of-speech tagging) tokens pos_tags NKJP-POS is a part of the National Corpus of Polish. Its objective is part-of-speech tagging, e.g., nouns, verbs, adjectives, adverbs, etc.
clarin-pl/aspectemo sequence labeling (sentiment classification) tokens labels AspectEmo Corpus is an extended version of a publicly available PolEmo 2.0 corpus of Polish customer reviews used in many projects on the use of different methods in sentiment analysis.
laugustyniak/political-advertising-pl sequence labeling (political advertising ) tokens tags First publicly open dataset for detecting specific text chunks and categories of political advertising in the Polish language.
laugustyniak/abusive-clauses-pl classification (abusive-clauses) text class Dataset with Polish abusive clauses examples.
allegro/klej-dyk pair classification (question answering)* (question, answer) target The Did You Know (pol. Czy wiesz?) dataset consists of human-annotated question-answer pairs.
allegro/klej-psc pair classification (text summarization)* (extract_text, summary_text) label The Polish Summaries Corpus contains news articles and their summaries.
allegro/klej-cdsc-e pair classification (textual entailment)* (sentence_A, sentence_B) entailment_judgment The polish sentence pairs which are human-annotated for textual entailment.

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •