Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

56 lines (37 sloc) 1.86 kb

dbpediakit

Python utilities to do analytics and perform transformations on the DBpedia dumps.

DBpedia is an extraction of the structured content of Wikipedia articles (links, redirect, infobox properties) augmented with a generic ontology (hierarchy of classes for the entities described by Wikipedia articles and schemas for their properties).

Quick example

>>> import dbpediakit as dbk

>>> archive_file = dbk.archive.fetch("long_abstracts")
>>> archive_file
'/home/ogrisel/data/dbpedia/long_abstracts_en.nt.bz2'

>>> tuples = dbk.archive.extract_text(archive_file)
>>> tuples
<generator object extract_text at 0x41e8aa0></generator>

>>> first = tuples.next()
>>> first.id
Autism

>>> first.text[:60] + u"..."
u'Autism is a disorder of neural development characterized by ...'

Overview

  • dbpediakit.archive provides lightweight utilities to fetch and parse the compressed ntriples (RDF) archive files.

    Parsed content is extracted as generators of python namedtuples. Those streams of tuples can then be filtered, processes and serialized to file or pipe using the CSV format understood by many databases and ETL.

  • dbpediakit.postgres provides utilities to load the tuples into a PostgreSQL database using the psql command line client for efficient bulk loading of the parsed data using the COPY command from stdin. The decompressed archive is never fully loaded in memory nor in a temporary file but directly streamed to the PostgreSQL database instead.

Complete examples

  • examples/topics shows how to process the DBpedia dump to extract a topic taxonomy out of the Wikipedia categories useable for document classification and navigation.
Jump to Line
Something went wrong with that request. Please try again.