-
Notifications
You must be signed in to change notification settings - Fork 1
Analyzes HTML content for RDFa, Microdata and Microformats
digitalbazaar/cc-structured-data
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
This is a library that analyzes the CommonCrawl dataset for structured data expressed as RDFa, Microdata or Microformats. To build -------- You'll need to have Apache Ant (http://ant.apache.org/manual/install.html) installed, and once you do, just run a: # ant dist This step will compile the libraries and Hadoop code into an Elastic MapReduce- friendly JAR at dist/lib/StructuredDataAnalyzer.jar, suitable for use as a custom JAR-based Elastic MapReduce workflow. To run locally -------------- You'll need to be running Hadoop, and if you don't have it installed, Cloudera provides a useful set of OS-specific Hadoop packages which will make it easy. Check out their site: https://ccp.cloudera.com/display/SUPPORT/Downloads Once you've got Hadoop installed, you can use the 'hadoop jar' task to execute the tutorial code. Here's the pattern: hadoop jar <checkout location>/dist/lib/StructuredDataAnalyzer.jar com.digitalbazaar.analyzer.StructuredDataAnalyzer <Amazon AWS access key ID> <Amazon AWS secret access key> <CommonCrawl crawl files to use as input> <HDFS output location>
About
Analyzes HTML content for RDFa, Microdata and Microformats
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published