Skip to content
master
Go to file
Code
This branch is 6 commits behind AGMLab:master.

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

README.md

Giranking

Giranking project contains Link ranking algorithms developed with Apache Giraph to be used in Apache Nutch 2.x. It can be used in different applications as well.

Apache Giraph allows us perform graph operations efficiently in a distributed manner.

For sample jobs with proper configurations, look at https://github.com/AGMLab/nutch/tree/2.1-emre/src/java/org/apache/nutch/scoring. There exists UsageRank in this package as well which just normalizes visit counts of hosts.

Current Problems

  • Cycles cause unfair advantage to some vertices, they get 9.9+ score out of 10.0

LinkRank

Derivation of PageRank. Ported from Apache Nutch 1.x LinkRank to make it more efficient with Giraph.

  • Uses full URLs (pages).
  • Uses 10 iteration.
  • Removes duplicate links.
  • Performs cumulative probability distribution normalization, yields scores [0..10].
  • Use Nutch2WebpageInputFormat and Nutch2WebpageOutputFormat when HBase is your DB.
  • Use LinkRankVertexInputFormat (if you have initial scores for vertices) or LinkRankVertexUniformInputFormat (gives 1.0 for each vertex). LinkRankEdgeInputFormat for edge list.
  • Use LinkRankEdgeFilter for skipping malformed URLs.

HostRank

Derivation of LinkRank. Works on HBase host table.

  • Use LinkRankComputation with Nutch2HostInputFormat and Nutch2HostOutputFormat.
  • Uses hosts from the HBase host table generated by Apache Nutch 2. (./nutch updatedb -linkDb)

TrustRank

  • Use TrustRankComputation with Nutch2HostTrustInputFormat and Nutch2HostOutputFormat with output fields set as mtdt:_tr_

About

Link ranking with Apache Giraph for Apache Nutch

Resources

License

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.