Irene1 Problem-Set

This is a problem-set for the SherlockBench AI benchmarking system.

This problem-set is designed to test fluid intelligence. It only has three problems because

• current frontier models are extremely slow and expensive to test
• we're not testing knowledge but investigation ability so having a large problem-set seems unnecessary

Usage

Copy the source file into your checkout of the SherlockBench-API under src/clj/.

Then edit config.edn and add it to :problem-namespaces.

Note this problem-set contains procedurally generated problems. To make them reproducable, make a custom problem-set and set seed. For example

"crack-lock-seeded" {:names [("irene1" "crack lock")] :seed 42}

Development

It's just a single file.

Run the tests like this:

clj -X:test

Version

This is Irene1v1