Skip to content

Benchmark Projects

Jump to bottom
Shaun A. Noordin edited this page Sep 6, 2025 · 8 revisions

Benchmark Projects for the Zooniverse

aka The Northstar Project: a Zooniverse Project for Benchmarking Projects (by the Zooniverse)

aka So You Want to Build a Specific Kind of Zooniverse Project and Need a Template or Example

Benchmark Projects are Zooniverse projects that serve as templates or examples for others of its kind. This document lists all Benchmark Projects, their purpose, their features, and how to create one from scratch.

Currently, this document is written specifically by & for the internal Zooniverse team, but we intend to expand the intended audience to include all project owners & collaborators. We assume readers already know the basic anatomy of a Zooniverse Project, and know the differences between Projects, Workflows, Subject Sets, and Tasks.

Notes:

  • 🔐 if you see this lock & key icon, it means that the associated link/resource/etc is only available to Zooniverse team members.
  • 🛠️ if you see this hammer & wrench icon, it means expect technical nerdery that may only make sense to Zooniverse developers. Please ping one of them if you have questions about their alien jargon.

Why Benchmark Projects?

The Zooniverse has a lot of features and a lot of moving parts, and sometimes the more complex project types (e.g. Indexed Transcription-type Projects) require several features or a unique setup to even start working. (e.g. Transcription Tasks, Caesar workflow config, Subject indexing)

  • For Zooniverse developers:
    • Benchmark Projects are a great way to test if a feature works (or still works following updates), using actual data.
    • This avoids scenarios where a fancy new feature (FNF) was built for a "real project", and that "real project" gets retired or changed significantly, causing that FNF to be unavailable even for local testing.
  • For everyone else: Benchmark Projects are a great way to discover the various features that the Zooniverse has to offer.

Related Documentation:

Minimal Benchmark Project

The Minimal Benchmark Project™ is the simplest version of a Zooniverse project that makes sense. It contains simple image Subjects, and a Workflow with one Yes/No Question Task.

Notable Features:

  • None.
  • Although simple, this is useful for baseline tests. If you're unable to run the project, then you know something has gone horribly wrong somewhere. (e.g. new code changes break the classifier, cat has unplugged WiFi router again, Panoptes server is on fire, etc)

Also see:

  • 🐱 I Fancy Cats, the original Benchmark Project that uses photos of kittens to test various Task types.

Anatomy / Configuration

Project: standard project setup. No special configurations.

Workflow: one standard workflow built in the FEM Pages Editor (aka Workflow Editor). Contains 1x Single Answer Question Task, with a "Yes" choice and a "No" choice.

Subjects: a collection of single-image Subjects, preferably of cute cats.

If you really want technical details, here's...

How To Make a Minimal Benchmark Project (Step-by-Step)

Project:

  • Sign in the Zooniverse and go to the Project Builder.
  • Project: create a new Project. Provide name, description, and introduction.
    • Visibility: set to state=public and visibility=live

🛠️ The Project Resource should look like this:

minimalBenchmarkProject = {
  configuration: {},
  description: "The Minimal Benchmark Project is the simplest version of a Zooniverse project that makes sense.",
  display_name: "Minimal Benchmark Project",
  introduction: "The Minimal Benchmark Project is the simplest version of a Zooniverse project that makes sense.",
  live: true,
  private: false
}

Subjects & Subject Set:

  • Create a new Subject Set, call it "Example Cat Photos"
    • Just drag & drop any number of kitten JPEGs into Project Builder => Subject Set => "Example Cat Photos".
    • A manifest file isn't required... but if you really want to, we have an example manifest.csv with cat photos you can find on Wikimedia Commons:
filename,origin
cat-001a.jpg,"https://commons.wikimedia.org/wiki/File:Six_weeks_old_cat_(aka)%2Ejpg"
cat-002a.jpg,"https://commons.wikimedia.org/wiki/File:Felis_catus-cat_on_snow%2Ejpg"
cat-003a.jpg,"https://commons.wikimedia.org/wiki/File:Our_feral_cats_4%2Ejpg"
cat-004a.jpg,"https://commons.wikimedia.org/wiki/File:Domestic_cat_2011_G02%2Ejpg"
cat-005a.jpg,"https://commons.wikimedia.org/wiki/File:Domestic_cat_by_Shagil_Kannur%2Ejpg"

Workflow:

  • Create a new Workflow, call it "Is This Cute?"
    • Add a Question Task with...
      • question/instructions/main text: _"Is this cat adorable?"
      • two answers: "Yes" and "No", both which immediately submit the Classification.
    • Add "Example Cat Photos" to its Associated Subject Sets.
    • Set the Workflow as 'active'. (Note: this can be done on the Workflows listing page, not from within the edit Workflow page)

🛠️ The Workflow Resource should look like this:

isThisCuteWorkflow = {
  active: true,
  configuration: {},
  display_name: "Is This Cute?",
  links: {
    project: "((Project ID))",
    subjects: [ "((SubjectSet ID of Example Cat Photos))" ]
  },
  steps: [
    [
      "P0",
      { stepKey: "P0", taskKeys: ["T0"] }
    ]
  ],
  tasks: {
    "T0": {
      answers: [
        { label: "Yes" },
        { label: "No" }
      ],
      help: "",
      required: false,
      question: "Is this cat adorable?",
      type: "single"
    }
  }
}

🛠️⚠️ Advanced technical nerdery: we can tell that this Workflow was built for the FEM Classifier, since it has explicit steps. Workflows built for the classic PFE Classifier don't use steps, and would need something like first_task = "T0". That said, this Workflow will still work on the FEM classifier even if steps = [], since there's some backwards compatibility that lets it extrapolate steps at runtime.

Notes

This entry was last reviewed 6 Sep 2025.

Clone this wiki locally