Skip to content

Benchmark Projects

Jump to bottom
Shaun A. Noordin edited this page Sep 12, 2025 · 8 revisions

Benchmark Projects for the Zooniverse

aka The Northstar Project: a Zooniverse Project for Benchmarking Projects (by the Zooniverse)

aka So You Want to Build a Specific Kind of Zooniverse Project and Need a Template or Example

Benchmark Projects are Zooniverse projects that serve as templates or examples for others of its kind. This document lists all Benchmark Projects, their purpose, their features, and how to create one from scratch.

Currently, this document is written specifically by & for the internal Zooniverse team, but we intend to expand the intended audience to include all project owners & collaborators. We assume readers already know the basic anatomy of a Zooniverse Project, and know the differences between Projects, Workflows, Subject Sets, and Tasks.

Notes:

  • 🔐 if you see this lock & key icon, it means that the associated link/resource/etc is only available to Zooniverse team members.
  • 🛠️ if you see this hammer & wrench icon, it means expect technical nerdery that may only make sense to Zooniverse developers. Please ping one of them if you have questions about their alien jargon.

Why Benchmark Projects?

The Zooniverse has a lot of features and a lot of moving parts, and sometimes the more complex project types (e.g. Indexed Transcription-type Projects) require several features or a unique setup to even start working. (e.g. Transcription Tasks, Caesar workflow config, Subject indexing)

  • For Zooniverse developers:
    • Benchmark Projects are a great way to test if a feature works (or still works following updates), using actual data.
    • This avoids scenarios where a fancy new feature (FNF) was built for a "real project", and that "real project" gets retired or changed significantly, causing that FNF to be unavailable even for local testing.
  • For everyone else: Benchmark Projects are a great way to discover the various features that the Zooniverse has to offer.

Related Documentation:

Minimal Benchmark Project

Minimal Benchmark Project

The Minimal Benchmark Project™ is the simplest version of a Zooniverse project that makes sense. It contains simple image Subjects, and a Workflow with one Yes/No Question Task.

Notable Features:

  • None.
  • Although simple, this is useful for baseline tests. If you're unable to run the project, then you know something has gone horribly wrong somewhere. (e.g. new code changes break the classifier, cat has unplugged WiFi router again, Panoptes server is on fire, etc)

Also see:

  • 🐱 I Fancy Cats, the original Benchmark Project that uses photos of kittens to test various Task types.

Anatomy / Configuration

Project: standard project setup. No special configurations.

Workflow: one standard workflow built in the FEM Pages Editor (aka Workflow Editor). Contains 1x Single Answer Question Task, with a "Yes" choice and a "No" choice.

Subjects: a collection of single-image Subjects, preferably of cute cats.

If you really want technical details, here's...

How To Make a Minimal Benchmark Project (Step-by-Step)

Project:

  • Sign in the Zooniverse and go to the Project Builder.
  • Project: create a new Project. Provide name, description, and introduction.
    • Visibility: set to state=public and visibility=live

🛠️ The Project Resource should look like this:

minimalBenchmarkProject = {
  configuration: {},
  description: "The Minimal Benchmark Project is the simplest version of a Zooniverse project that makes sense.",
  display_name: "Minimal Benchmark Project",
  introduction: "The Minimal Benchmark Project is the simplest version of a Zooniverse project that makes sense.",
  live: true,
  private: false
}

Subjects & Subject Set:

  • Create a new Subject Set, call it "Example Cat Photos"
    • Just drag & drop any number of kitten JPEGs into Project Builder => Subject Set => "Example Cat Photos".
    • A manifest file isn't required... but if you really want to, we have an example manifest.csv with cat photos you can find on Wikimedia Commons:
filename,origin
cat-001a.jpg,"https://commons.wikimedia.org/wiki/File:Six_weeks_old_cat_(aka)%2Ejpg"
cat-002a.jpg,"https://commons.wikimedia.org/wiki/File:Felis_catus-cat_on_snow%2Ejpg"
cat-003a.jpg,"https://commons.wikimedia.org/wiki/File:Our_feral_cats_4%2Ejpg"
cat-004a.jpg,"https://commons.wikimedia.org/wiki/File:Domestic_cat_2011_G02%2Ejpg"
cat-005a.jpg,"https://commons.wikimedia.org/wiki/File:Domestic_cat_by_Shagil_Kannur%2Ejpg"

Workflow:

  • Create a new Workflow, call it "Is This Cute?"
    • Add a Question Task with...
      • question/instructions/main text: _"Is this cat adorable?"
      • two answers: "Yes" and "No", both which immediately submit the Classification.
    • Add "Example Cat Photos" to its Associated Subject Sets.
    • Set the Workflow as 'active'. (Note: this can be done on the Workflows listing page, not from within the edit Workflow page)

🛠️ The Workflow Resource should look like this:

isThisCuteWorkflow = {
  active: true,
  configuration: {},
  display_name: "Is This Cute?",
  links: {
    project: "((Project ID))",
    subjects: [ "((SubjectSet ID of Example Cat Photos))" ]
  },
  steps: [
    [
      "P0",
      { stepKey: "P0", taskKeys: ["T0"] }
    ]
  ],
  tasks: {
    "T0": {
      answers: [
        { label: "Yes" },
        { label: "No" }
      ],
      help: "",
      question: "Is this cat adorable?",
      required: false,
      type: "single"
    }
  }
}

🛠️⚠️ Advanced technical nerdery: we can tell that this Workflow was built for the FEM Classifier, since it has explicit steps. Workflows built for the classic PFE Classifier don't use steps, and would need something like first_task = "T0". That said, this Workflow will still work on the FEM classifier even if steps = [], since there's some backwards compatibility that lets it extrapolate steps at runtime.

Notes

This entry was last reviewed 6 Sep 2025.

Transcription Benchmark Project

TODO: add screenshot

The Transcription Benchmark Project™ demonstrates how we can use Transcription Tasks to allow volunteers to transcribe text from (an image of) a page. Transcription Projects on the Zooniverse are usually used by researchers to help understand & record the contents of handwritten letters, or ancient manuscripts, or similar things that automated OCR has difficulty recognising.

Features:

  • Transcription Task: contains two sub-parts
    • The Actual Transcription Task: lets volunteers mark the start & end of a line of text on a page, and then input the content of that line of text.
    • A Question Task: asks "has everything on this page been fully transcribed?" (Used to determine if a Subject should be retired.)
  • Aggregated transcriptions on Subjects: i.e. transcriptions from other volunteers are displayed while you're transcribing.
    • This lets you "compare notes" with others, and figure out which parts of the page have been fully transcribed.
    • This is powered by a Caesar workflow configuration.

TODO: explain retirement rules.

Also note:

  • This project has the 'transcription-task' and 'fem-lab' experimental tools enabled.

Also see:

Anatomy / Configuration

Project: enable the transcription-task and (optionally) fem-lab Experimental Tools.

Workflow:

  • contains one Transcription Task (which consists of two Tasks packaged in one Step).
  • NOTE: at the moment, this can/should only be built on the classic Project Builder edit workflow page (FEM-Lab ver).

Caesar: TODO

Subjects:

  • a collection of image Subjects, with each image being a scan/photo of a handwritten page/ancient manuscript/etc.
    • Note: a Subject can have multiple images. This is useful if you want each subject as a "book" with multiple "pages" (images).
🛠️ Technical Details: TODO

Workflow:

simpleTranscriptionWorkflow = {
  active: true,
  configuration: {
    classifier_version: "2.0",
    subject_viewer: "multiFrame"
  },
  display_name: "Simple Transcription WF",
  links: {
    project: "((Project ID))",
    subjects: [ "((SubjectSet ID))" ]
  },
  steps: [
    [
      "S0",
      { stepKey: "S0", taskKeys: ["T0", "T1"] }
    ]
  ],
  tasks: {
    "T0": {
      help: "",
      instruction: "Underline a single row of text by clicking at the start and end of the line, and follow the instructions on the pop-up for transcribing the text you’ve just underlined.",
      tools: [
        {
          color: "",
          details: [
            {
              help: "",
              instruction: "Transcribe the line of text that you've marked.",
              required: "true",
              type: "text"
            }
          ],
          label: "Single line of text",
          type: "transcriptionLine"
        }
      ],
      type: "transcription"
    },
    "T1": {
      answers: [
        { label: "Yes" },
        { label: "No" }
      ],
      help: "",
      question: "Have all the volunteer-made underline marks turned grey?",
      required: true,
      type: "single"
    }
  }
}

Notes

This entry was last reviewed 13 Sep 2025.

Clone this wiki locally