To participate in the challenge, please see the competition page here. The dataset files for the development and test phases are available under the data folder. Note that the reference data for the development data is available (public_reference_data_dev.zip) to evaluate against using the instructions in evaluation_script.zip.
Please refer to the github pages for detailed information on the project: https://ukplab.github.io/PuzzLing-Machines/
Please use the following citation:
@inproceedings{DBLP:conf/acl/SahinKRG20,
author = {G{\"{o}}zde G{\"{u}}l Sahin and
Yova Kementchedjhieva and
Phillip Rust and
Iryna Gurevych},
editor = {Dan Jurafsky and
Joyce Chai and
Natalie Schluter and
Joel R. Tetreault},
title = {PuzzLing Machines: {A} Challenge on Learning From Small Data},
booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, {ACL} 2020, Online, July 5-10, 2020},
pages = {1241--1254},
publisher = {Association for Computational Linguistics},
year = {2020},
url = {https://doi.org/10.18653/v1/2020.acl-main.115},
doi = {10.18653/v1/2020.acl-main.115},
timestamp = {Thu, 14 Oct 2021 09:46:03 +0200},
biburl = {https://dblp.org/rec/conf/acl/SahinKRG20.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
Abstract: Deep neural models have repeatedly proved excellent at memorizing surface patterns from large datasets for various ML and NLP benchmarks. They struggle to achieve human-like thinking, however, because they lack the skill of iterative reasoning upon knowledge. To expose this problem in a new light, we introduce a challenge on learning from small data, PuzzLing Machines, which consists of Rosetta Stone puzzles from Linguistic Olympiads for high school students. These puzzles are carefully designed to contain only the minimal amount of parallel text necessary to deduce the form of unseen expressions. Solving them does not require external information (e.g., knowledge bases, visual signals) or linguistic expertise, but meta-linguistic awareness and deductive skills. Our challenge contains around 100 puzzles covering a wide range of linguistic phenomena from 81 languages. We show that both simple statistical algorithms and state-of-the-art deep neural models perform inadequately on this challenge, as expected. We hope that this benchmark, available at https://ukplab.github.io/PuzzLing-Machines/, inspires further efforts towards a new paradigm in NLP---one that is grounded in human-like reasoning and understanding.
Contact person: Gözde Gül Şahin, goezde{dot}guel{at}gmail.com
https://www.ukp.tu-darmstadt.de/
Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.