Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
transmogrifier source blueprints for crawling html
Python
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
docs
transmogrify
.gitignore
README.rst
bootstrap.py
buildout.cfg
setup.cfg
setup.py

README.rst

Crawling - html to import

transmogrify.webcrawler will crawl html to extract pages and files as a source for your transmogrifier pipeline. transmogrify.webcrawler.typerecognitor aids in setting '_type' based on the crawled mimetype. transmogrify.webcrawler.cache helps speed up crawling and reduce memory usage by storing items locally.

These blueprints are designed to work with the funnelweb pipeline but can be used independently.

Something went wrong with that request. Please try again.