Skip to content

aladagemre/giranking

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code
This branch is 14 commits behind AGMLab:master.

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

Giranking

Giranking project contains Link ranking algorithms developed with Apache Giraph to be used in Apache Nutch 2.x. It can be used in different applications as well.

Apache Giraph allows us perform graph operations efficiently in a distributed manner.

For sample jobs with proper configurations, look at https://github.com/AGMLab/nutch/tree/2.1-emre/src/java/org/apache/nutch/scoring. There exists UsageRank in this package as well which just normalizes visit counts of hosts.

Current Problems

  • Cycles cause unfair advantage to some vertices, they get 9.9+ score out of 10.0

LinkRank

Derivation of PageRank. Ported from Apache Nutch 1.x LinkRank to make it more efficient with Giraph.

  • Uses full URLs (pages).
  • Uses 10 iteration.
  • Removes duplicate links.
  • Performs cumulative probability distribution normalization, yields scores [0..10].
  • Use Nutch2WebpageInputFormat and Nutch2WebpageOutputFormat when HBase is your DB.
  • Use LinkRankVertexInputFormat (if you have initial scores for vertices) or LinkRankVertexUniformInputFormat (gives 1.0 for each vertex). LinkRankEdgeInputFormat for edge list.
  • Use LinkRankEdgeFilter for skipping malformed URLs.

HostRank

Derivation of LinkRank. Works on HBase host table.

  • Use LinkRankComputation with Nutch2HostInputFormat and Nutch2HostOutputFormat.
  • Uses hosts from the HBase host table generated by Apache Nutch 2. (./nutch updatedb -linkDb)

TrustRank

  • Use TrustRankComputation with Nutch2HostTrustInputFormat and Nutch2HostOutputFormat with output fields set as mtdt:_tr_

About

Link ranking with Apache Giraph for Apache Nutch

Resources

License

Apache-2.0, Unknown licenses found

Licenses found

Apache-2.0
LICENSE.txt
Unknown
license-header.txt

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 100.0%