Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

TagRuler: Interactive Tool for Span-Level Data Programming by Demonstration

This repo contains the source code and the user evaluation data for TagRuler, a data programming by demonstration system for span-level annotation. Check out our demo video to see TagRuler in action!

Demonstration Video:

TagRuler synthesizes labeling functions based on your annotations, allowing you to quickly and easily generate large amounts of training data for span annotation, without the need to program.

What is TagRuler?

In 2020, we introduced Ruler, a novel data programming by demonstration system that allows domain experts to leverage data programming without the need for coding. Ruler generates document classification rules, but we knew that there was a bigger challenge left to tackle: span-level annotations. This is one of the more time-consuming labelling tasks, and creating a DPBD system for this proved to be a challenge because of the sheer magnitude of the space of labeling functions over spans.

We feel that this is a critical extension of the DPBD paradigm, and that by open-sourcing it, we can help with all kinds of labelling needs.

How to use the source code in this repo

Follow these instructions to run the system on your own, where you can plug in your own data and save the resulting labels, models, and annotations.

1. Server

1-1. Install Dependencies πŸ”§

cd server
pip install -r requirements.txt
python -m spacy download en_core_web_sm

1-2. (Optional) Download Data Files

1-3. Run πŸƒ

python api/

2. User Interface

2-1. Install Node.js

You can download node.js here.

To confirm that you have node.js installed, run node - v

2-2. Run

cd ui
npm install 
npm start

By default, the app will make calls to localhost:5000, assuming that you have the server running on your machine. (See the instructions above).

Once you have both of these running, navigate to localhost:3000.


...or other inquiries, contact and/or