Code and Data for the paper

    Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game
    Prisha Samadarshi, Mariam Mustafa, Anushka Kulkarni, Raven Rothkopf, Tuhin Chakrabarty, Smaranda Muresan

Look at

    games.txt for all 200 games
    knowledge_types.csv for 800 annotations of knowledge types. These are arrange chronologically so each 4 rows = 1 game
    automated_call/prompt_llm.txt contains the actual prompt
    scoring folder has all the LLM responses

If you use our data or annotations please cite us at

      @article{samadarshi2024connecting,
        title={Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game},
        author={Samadarshi, Prisha and Mustafa, Mariam and Kulkarni, Anushka and Rothkopf, Raven and Chakrabarty, Tuhin and Muresan, Smaranda},
        journal={arXiv preprint arXiv:2406.11012},
        year={2024}
      }