Link ranking with Apache Giraph for Apache Nutch
Java
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is 2 commits behind AGMLab:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
giraph-nutch
LICENSE.txt
NOTICE
README.md
checkstyle.xml
findbugs-exclude.xml
license-header.txt
pom.xml

README.md

Giranking

Giranking project contains Link ranking algorithms developed with Apache Giraph to be used in Apache Nutch 2.x. It can be used in different applications as well.

Apache Giraph allows us perform graph operations efficiently in a distributed manner.

For sample jobs with proper configurations, look at https://github.com/AGMLab/nutch/tree/2.1-emre/src/java/org/apache/nutch/scoring. There exists UsageRank in this package as well which just normalizes visit counts of hosts.

Current Problems

  • Cycles cause unfair advantage to some vertices, they get 9.9+ score out of 10.0

LinkRank

Derivation of PageRank. Ported from Apache Nutch 1.x LinkRank to make it more efficient with Giraph.

  • Uses full URLs (pages).
  • Uses 10 iteration.
  • Removes duplicate links.
  • Performs cumulative probability distribution normalization, yields scores [0..10].
  • Use Nutch2WebpageInputFormat and Nutch2WebpageOutputFormat when HBase is your DB.
  • Use LinkRankVertexInputFormat (if you have initial scores for vertices) or LinkRankVertexUniformInputFormat (gives 1.0 for each vertex). LinkRankEdgeInputFormat for edge list.
  • Use LinkRankEdgeFilter for skipping malformed URLs.

HostRank

Derivation of LinkRank. Works on HBase host table.

  • Use LinkRankComputation with Nutch2HostInputFormat and Nutch2HostOutputFormat.
  • Uses hosts from the HBase host table generated by Apache Nutch 2. (./nutch updatedb -linkDb)

TrustRank

  • Use TrustRankComputation with Nutch2HostTrustInputFormat and Nutch2HostOutputFormat with output fields set as mtdt:_tr_