Skip to content
Pipeline for distributed Natural Language Processing, made in Python http://pypln.org
Python Shell Other
Find file
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
doc
pypln
requirements
scripts
tests
.gitignore
CONTRIBUTING.rst
COPYING
Makefile
README.rst
make-docs.sh
run_celery.sh
setup.py

README.rst

PyPLN

PyPLN is a distributed pipeline for natural language processing, made in Python. We use NLTK and Celery as our foundations. The goal of the project is to create an easy way to use NLTK for processing big corpora, with a Web interface.

PyPLN is sponsored by Fundação Getulio Vargas.

License

PyPLN is free software, released under the GPLv3 https://gnu.org/licenses/gpl-3.0.html.

Documentation

Our documentation is hosted using GitHub Pages:

Requirements

You will need some Python packages, libmagic,
poppler utils, libfreetype <http://www.freetype.org/>'s development headers and aspell dictionaries for english and portuguese.

To install dependencies (on a Debian-like GNU/Linux distribution):

sudo apt-get install python-setuptools libmagic-dev poppler-utils libfreetype6-dev fonts-dejavu aspell-en aspell-pt
pip install virtualenv virtualenvwrapper
mkvirtualenv pypln.backend
# we need to install Cython first because of the way pip handles C extensions
pip install Cython
pip install -r requirements/production.txt

You will also need to download some NLTK data packages. You can do so executing:

python -m nltk.downloader genesis maxent_treebank_pos_tagger punkt stopwords averaged_perceptron_tagger

Developing

To run tests:

workon pypln.backend
pip install -r requirements/development.txt
echo "MONGODB_CONFIG = {'host': 'localhost', 'port': 27017, 'database': 'test_pypln_dev', 'gridfs_collection': files}" >> pypln/backend/local_config.py
make test

See our code guidelines.

Creating a new Task

All analyses in PyPLN are performed by our workers. Every worker is a Celery task that can be included in the canvas that will run when a document is received in pypln.web.

New workers are very easy to create. All you need to do is write a subclass of PyPLNTask <https://github.com/NAMD/pypln.backend/blob/develop/pypln/backend/celery_task.py#L36> that implements a "process" method. This method will receive the document as a dictionary, and should return a dictionary that will be used to update the existing document. As an example:

from pypln.backend.celery_task import PyPLNTask

class FreqDist(PyPLNTask):
    def process(self, document):
        value = document['value']
        square = value ** 2
        return {'squared_value': square}

This worker assumes that a previous worker has already included "value" in the document and uses it to create a new one, called "squared_value".

Something went wrong with that request. Please try again.