Skip to content
/ harvest Public

A toolkit for extracting posts and post metadata from web forums.


Notifications You must be signed in to change notification settings


Folders and files

Last commit message
Last commit date

Latest commit


Repository files navigation

Harvest - A toolkit for extracting posts and post metadata from web forums

Actions Status codecov PyPI version

Automatic extraction of forum posts and metadata is a challenging task since forums do not expose their content in a standardized structure. Harvest performs this task reliably for many web forums and offers an easy way to extract data from web forums.


At the command line:

$ pip install harvest-webforum

If you want to install from the latest sources, you can do:

$ git clone
$ cd harvest
$ python3 install

Python library

Embedding harvest into your code is easy, as outlined below:

from urllib.request import urlopen, Request
from harvest import extract_data

USER_AGENT = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0"

url = ""
req = Request(url, headers={'User-Agent': USER_AGENT})
html = urlopen(req).read().decode('utf-8')

result = extract_data(html, url)

WEB-FORUM-52 gold standard

The corpus currently contains from 52 different web forums gold standard documents. These documents are also used by the integrations test of harvest.