BECEL: BEnchmark for Consistency Evaluation of Language Models

This is the repository of the paper: "BECEL: Benchmark for Consistency Evaluation of Language Models (TBD)

Directory Description

• data: BECEL datasets for 7 downstream tasks. Please refer README.md files in each data repository for more information.

  • ag_news: includes additive and semantic consistency datasets.
  • boolq: includes semantic and negational consistency datasets.
  • mrpc: includes semantic, negational, and symmetric consistency datasets.
  • rte: includes semantic, negational, and symmetric consistency datasets.
  • snli: includes semantic, negational, symmetric, and transitive consistency datasets.
  • sst2: includes additive and semantic consistency datasets are provided.
  • wic: includes semantic, negational, symmetric, and transitive consistency datasets.

• src: Scripts for evaluation metrics and examples.

Citation

@inproceedings{jang-etal-2022-becel,
    title = "{BECEL}: Benchmark for Consistency Evaluation of Language Models",
    author = "Jang, Myeongjun  and
      Kwon, Deuk Sin  and
      Lukasiewicz, Thomas",
    booktitle = "Proceedings of the 29th International Conference on Computational Linguistics",
    month = oct,
    year = "2022",
    address = "Gyeongju, Republic of Korea",
    publisher = "International Committee on Computational Linguistics",
    url = "https://aclanthology.org/2022.coling-1.324",
    pages = "3680--3696",
}