Skip to content
searching large heterogenous data dumps with Universal Sentence Encoder
Jupyter Notebook Python
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.
.python-version First commit Jan 16, 2020
LICENSE adds license Jan 16, 2020
Searching with USE.ipynb
nyc_docs.jsonl First commit Jan 16, 2020
requirements.txt First commit Jan 16, 2020 properly warn (but don't crash) if GPU is absent Jan 22, 2020 First commit Jan 16, 2020

Searching Bill de Blasio's Emails with the Universal Sentence Encoder

By Jeremy B. Merrill, Quartz

As part of Quartz's participation in the Luanda Leaks investigation in partnership with ICIJ, we built an system for searching large heterogenous document sets with AI. Read more about the system here.

Here's the code demo so you can reproduce our work. For this demo, we'll be searching in a set of a few thousand emails between Bill de Blasio and some advisors, informally called the "Agent of the City documents."

This isn't a full library or off-the-shelf software, but rather a demonstration that you can adapt for yourself.

This workflow works with huge document sets. The demo documents are small, so that indexing only takes a few seconds, rather than hours. The Luanda Leaks investigation involved 356 gigabytes of files (and 11 gigabytes of plain text).

how to

There are a few steps you'll have to take to run the demo.

  1. Get a computer with Python 3.7. You probably want a server with a GPU, but it's not strictly necessary. Without a GPU, this'll be very slow.
  2. Set up Elasticsearch locally.
  3. Split the provided nyc_docs.jsonl file (which has a text copy of every document in the document set) into sentence-size chunks by running python We need to split the documents into sentence-length chunks because Universal Sentence Encoder can only handle blocks of text shorter than 128 words.
  4. Index the chunks and the full documents to ElasticSearch by running python
  5. Embed the chunks into vector space with USE and index those with Annoy by running python
  6. Run the "Searching with USE.ipynb" Jupyter notebook to run your own searches.


about tech choices

Universal Sentence Encoder is a tensorflow library that, without any additional training, embeds sentences in several languages into a shared vector representation. Basically, sentences with a similar meaning have vectors that are close together.

Annoy is a Python library that makes nearest neighbor searches for vectors very fast. This lets us index thousands, or millions, of vectors and instantly get back which ones are close by.

Combined, these tools let us ask the computer, "what sentences mean something similar to this?" and we'll get back high quality results, even if the similar sentences don't overlap in any keywords. We also indexed the text of each sentence to ElasticSearch—and each sentence had a single ID number used in both the Annoy and ElasticSearch indexes.

You can’t perform that action at this time.