Skip to content
Modularly extensible semantic metadata validator
HTML Web Ontology Language Python
Latest commit 7002316 Dec 10, 2015 @emmett9001 emmett9001 fix readme formatting
Failed to load latest commit information.
schemato update test Dec 10, 2015
.gitignore add .cache to gitignore Dec 10, 2015
.travis.yml add travis.yml Dec 10, 2015
AUTHORS.rst Init scrubbed version of schemato May 6, 2014
LICENSE add Apache v2 license Nov 26, 2014
README.rst fix readme formatting Dec 10, 2015
setup.py version bump Mar 20, 2015

README.rst

https://travis-ci.org/Parsely/schemato.svg?branch=master

schemato

Schemato is a validator for HTML-embedded metadata standards. It knows the location of the official schema definitions, and uses these documents as validation templates. As a contributor, you can easily subclass the base validator class to plug into this functionality.

To see the validator in action:

from schemato import Schemato sc = Schemato("my_test.html") res = sc.validate() [a.to_dict() for a in res]

The first time you run schemato, it will make requests for the latest versions of the official schema definitions. Schemato will then call the validate() method of the Validator subclasses listed in settings.py.

There are a few test documents available for validation in the test_documents subdirectory.

Download

Download the source from PyPI with

pip install schemato

You can also clone this repo and take a closer look at the code and test documents

git clone https://github.com/Parsely/schemato.git

Then, install the library with

python setup.py install

Tests

To run the tests for Schemato:

pip install pytest cd schemato py.test test.py

Distiller

Schemato's distiller framework lets you implement strategies for creating a "normalized" set of metadata by mixing and matching metadata from different supported standards.

Supported so far:

  • parsely-page
  • OpenGraph
  • Schema.org NewsArticle

Take a look at the clean Python class definitions that describe the strategies:

https://github.com/Parsely/schemato/blob/master/schemato/distillery.py

There are two examples -- one that tries pp and falls back on Schema.org/OpenGraph (called ParselyDistiller) and another the tries Schema.org and falls back on OpenGraph (called NewsDistiller).

The distiller returns a clean Python dictionary that has all the extracted fields, as well as a dictionary describing which metadata standard was used to source each field. The framework is defined here:

https://github.com/Parsely/schemato/blob/master/schemato/distillers.py

Here is an example of usage:

from schemato import Schemato
from schemato.distillery import ParselyDistiller, NewsDistiller
mashable = Schemato("http://mashable.com/2012/10/17/iphone-5-supply-problems/")
ParselyDistiller(mashable).distill()

{'author': u'Seth Fiegerman',
'image_url': u'http://5.mshcdn.com/wp-content/uploads/2012/10/iphone-lineup.jpg',
'link': u'http://mashable.com/2012/10/17/iphone-5-supply-problems/',
'page_type': u'post',
'post_id': u'1432059',
'pub_date': u'2012-10-17T11:36:40+00:00',
'section': u'bus',
'site': 'Mashable',
'title': u"Apple's Manufacturing Partner Explains iPhone 5 Supply Problems"}

In this case, Mashable implements the parsely-page metadata field, which is used to source all the defined properties for this distiller.

d = NewsDistiller(mashable)
d.distill()

{'author': None,
'id': None,
'image_url': 'http://5.mshcdn.com/wp-content/uploads/2012/10/iphone-lineup.jpg',
'link': 'http://mashable.com/2012/10/17/iphone-5-supply-problems/',
'pub_date': None,
'section': None,
'title': "Apple's Manufacturing Partner Explains iPhone 5 Supply Problems"}

d.sources

{'author': None,
'id': None,
'image_url': 'og:image',
'link': 'og:url',
'pub_date': None,
'section': None,
'title': 'og:title'}

In this case, our strategy did not involve parsely-page, and instead used Schema.org and OpenGraph. Since Mashable does not implement Schema.org but does implement OpenGraph, it comes up with the fields it can. The sources property shows which fields were populated and how they got their values.

Support

If you need help using Schemato, or have found a bug, please create an issue on the [Github repo](https://github.com/Parsely/schemato/issues?state=open).

Something went wrong with that request. Please try again.