Example application using neo4j.rb
JavaScript Ruby CoffeeScript
Latest commit 507cd09 Jan 29, 2013 @andreasronge andreasronge Merge pull request #1 from dmzza/patch-1
Fixed some typos


Neo4j.rb example application

This application shows how to use neo4j together with rails 3.1 by parsing Twitter feeds and creating a social graph. It first shows some basic operations which you are probably used to from the ActiveRecord/ActiveModel API. In the last section we take real advantage of the neo4j engine by implementing a recommendation algorithm for finding new twitter users.

Have fun and feel free to clone it.


Make sure you have Java JDK 1.6+ installed

Install latest JRuby, example

rvm use jruby-1.6.7

Install Rails (>= 3.1.1)

gem install rails

1. Create a rails project

Use my rails template which will disable active record and enable neo4j instead

rails new kvitter -m http://andreasronge.github.com/neo4j/rails.rb

Edit the Gemfile and add the twitter gem

cd kvitter
emacs Gemfile # and add
gem 'twitter', '1.7.2'

Download all the dependencies


2. Scaffold

Run the following commands:

rails generate scaffold Tweet text:string link:string date:datetime tweet_id:string --indices tweet_id date text --has_n tags mentions links --has_one tweeted_by:tweeted
rails generate scaffold User twid:string link:string --indices twid --has_n tweeted follows knows used_tags mentioned_from:mentions
rails generate scaffold Link url:string --indices url --has_n tweets:links short_urls:redirected_link --has_one redirected_link
rails generate scaffold Tag name:string --indices name --has_n tweets:tags used_by_users:used_tags

There is nothing magical happening here like neo4j configuration or migrations. It only creates plain new Ruby classes (models, controllers and views) and routing. The relationships and properties are only specified in the model classes.

3. Start Rails

Test the basic crud operations

rails s

Open browser: localhost:3000/tags

4 Search using Twitter API

Add the following:


The following code does a twitter search and parses the result. It creates and connects the Tweet, Link, User and Tag model classes.

def search
  @tag = Tag.find(params[:id])

  search = Twitter::Search.new
  result = search.hashtag(@tag.name)

  curr_page = 0
  while curr_page < 2 do
    result.each do |item|
      parsed_tweet_hash = Tweet.parse(item)
      next if Tweet.find_by_tweet_id(parsed_tweet_hash[:tweet_id])
      tweet = Tweet.create!(parsed_tweet_hash)

      twid = item['from_user'].downcase
      user = User.find_or_create_by(:twid => twid)
      user.tweeted << tweet

      parse_tweet(tweet, user)
    curr_page += 1

  redirect_to @tag

def parse_tweet(tweet, user)
  tweet.text.gsub(/(@\w+|https?:\/\/[a-zA-Z0-9\-\.~\:\?#\[\]\!\@\$&,\*+=;,\/]+|#\w+)/).each do |t|
    case t
      when /^@.+/
        t = t[1..-1].downcase
        next if t.nil?
        other = User.find_or_create_by(:twid => t)
        user.knows << other unless t == user.twid || user.knows.include?(other)
        tweet.mentions << other
      when /#.+/
        t = t[1..-1].downcase
        tag = Tag.find_or_create_by(:name => t)
        tweet.tags << tag unless tweet.tags.include?(tag)
        user.used_tags << tag unless user.used_tags.include?(tag)
      when /https?:.+/
        link = Link.find_or_create_by(:url => t)
        tweet.links << (link.redirected_link || link)


Change index on text to fulltext lucene:

property :text, :type => String, :index => :fulltext

Add a to_s and parse method and change the index type of text to fulltext (we need that later on, see below).

def to_s

def self.parse(item)
  {:tweet_id => item['id_str'],
   :text => item['text'],
   :date => Time.parse(item['created_at']),
   :link => "http://twitter.com/#{item['from_user']}/statuses/#{item['id_str']}"

Notice : in Neo4j it is not necessary to specify the types of properties. By setting :type => String we force that each Tweet object will store the property 'type' as strings.



resource :tags


resources :tags do
  get :search, :on => :member


Add a button to the view:

<%= button_to "Search", [:search, @tag], :method => :get %>

Test the application now by opening a browser localhost:3000/tags Create a new tag and press the button 'search' You will now found tweets, users and links

Follow URL shortenings

When you looked at all the links, most of them are short urls like t.co Since we are more interested in the real link and who has tweeted about them we must follow the URL by doing a HTTP head request.

We use a before save callback to create a the real link which correspond to where the short url is directed to. The short link and the real link are connected in a redirected_link relationship.


class Link < Neo4j::Rails::Model
  property :url, :type => String, :index => :exact
  has_one :redirected_link

  # Add the following:
  before_save :create_redirect_link

  SHORT_URLS = %w[t.co bit.ly ow.ly goo.gl tiny.cc tinyurl.com doiop.com readthisurl.com memurl.com tr.im cli.gs short.ie kl.am idek.net short.ie is.gd hex.io asterl.in j.mp].to_set

  def to_s


  def self.short_url?(url)
    domain = url.split('/')[2]
    domain && SHORT_URLS.include?(domain)

  def create_redirect_link
    return if !self.class.short_url?(url)
    uri = URI.parse(url)
    http = Net::HTTP.new(uri.host, uri.port)
    http.read_timeout = 200
    req = Net::HTTP::Head.new(uri.request_uri)
    res = http.request(req)
    redirect = res['location']
    if redirect && url != redirect
      self.redirected_link = Link.find_or_create_by(:url => redirect.strip)
  rescue Timeout::Error
    puts "Can't acccess #{url}"
  rescue Error
    puts "Can't call #{url}"
  rescue Net::HTTPBadResponse
    puts "Bad response for #{url}"


In order to show both outgoing redirected_link and incoming redirected_link (by using short_urls method) add the following:

  <b>Short Urls:</b>
  <% @link.short_urls.each do |link| %>
    <%= link_to link, link %> <br/>
  <% end %>

<% if @link.redirected_link %>
  <b>Redirects to</b>
  <%= link_to @link.redirected_link, @link.redirected_link %>
<% end %>

Don't display URL shortening

The index page shows all the links, including links like bit.ly and the the real one. To only return the real URLs we can use rules, which is a bit similar to scope in ActiveRecord.


rule(:real) { redirected_link.nil?}

This means that it will group all links under the rule :real which does not have a redirected_link To return all those nodes, use the class method #real. Notice you can do some really interesting queries using rules and cypher together, see “Rules-Cypher”:github.com/andreasronge/neo4j/wiki/Neo4j%3A%3AWrapper-Rules-and-Functions

Btw, the Neo4j::Rails::Model#all method is also implemented as a rule. You can also chain rules, just like scopes in Active Record.


def index
   @links = Link.real

   respond_to do |format|
     format.html # index.html.erb
     format.json { render :json => @links }

Since the rules do work by creating relationships when new nodes are created/updated/deleted we must do another search or stop the rails server and delete the database 'db/neo4j-developement'


The list of all tweets (localhost:3000/tweets ) does not look good. It needs some pagination. Add the two gem in Gemfile:

gem 'neo4j-will_paginate', :git => 'git://github.com/andreasronge/neo4j-will_paginate.git'
gem 'will_paginate'


Neo4j.rb comes included with the will_paginate gem. Change the index method

@tweets = Tweet.all


@tweets = Tweet.all.paginate(:page => params[:page], :per_page => 10)

Pagination is support for all traversals and lucene queries.


Add the following line before the table:

<%= will_paginate(@tweets) %>

Searching and Sorting

Lets say we want to sort the tweets by the text. Lucene has two types of indexes: exact and fulltext. The exact index is perfect for keywords while the fulltext is for longer texts. We have already changed the index of text to fulltext for the app/models/tweets.rb See lucene.apache.org/java/3_0_0/queryparsersyntax.html for the query syntax.


Add the following form before the table

<%= form_for(:tweets, :method => :get) do |f| %>
<div class="field">
  <%= text_field_tag :query %>
    <%= f.submit "Search" %>
<% end %>


In the index method we now should handle the query parameter. Add the following before the respond_to.

def index
  query = params[:query]
    if query.present?
      @tweets = Tweet.all("text:#{query}", :type => :fulltext).paginate(:page => params[:page], :per_page => 10)
      @tweets = Tweet.all.paginate(:page => params[:page], :per_page => 10)

  # respond_to ...

Test it !

Add SVG Visualization !

Would it not be cool to visualize who knows who ?

Download the D3 Javascript library

Download the D3 version 2.4.6 javascript library from github.com/mbostock/d3/archives/master.

Unzip it and move the following files to the app/assets/javascript folder

  • d3.geom.js

  • d3.js

  • d3.layout.js


Create a JSON API needed by the javascript

def index
  @users = User.all

  respond_to do |format|
    format.html # index.html.erb
    format.json do
      nodes = @users.map{|u| {:name => u.twid, :value => u.tweeted.size}}
      links = []
      @users.each do |user|
        links += user.knows.map {|other| { :source => nodes.find_index{|n| n[:name] == user.twid}, :target => nodes.find_index{|n| n[:name] == other.twid}}}
      render :json => {:nodes => nodes, :links => links}

Test the API, open a browser localhost:3000/users.json it should return something like this:



Add the following coffeescript (be careful with the indentation):

# Place all the behaviors and hooks related to the matching controller here.
# All this logic will automatically be available in application.js.
# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

$(document).ready ->
  w = 1260
  h = 1300
  fill = d3.scale.category20()

  vis = d3.select("#graph").append("svg:svg").attr("width", w).attr("height", h)

  d3.json("/users.json", (json) ->
    force = d3.layout.force()
      .size([w, h])

    link = vis.selectAll("line.link")
      .attr("class", "link")
      .style("stroke-width", (d) -> Math.sqrt(d.value))
      .attr("x1", (d) -> d.source.x)
      .attr("y1", (d) -> d.source.y)
      .attr("x2", (d) -> d.target.x)
      .attr("y2", (d) -> d.target.y)

    node = vis.selectAll("g.node")
      .attr("transform", (d) -> "translate(" + d.x + "," + d.y + ")")
      .attr("class", "node")

      .attr("r", (d) -> if d.value > 25 then 50 else d.value*2 + 5)
      .style("fill", (d) -> '#fea')

    node.append("svg:title").text((d) -> d.name)
      .attr("text-anchor", "middle")
      .attr("dy", ".3em")
      .text((d) -> d.name)

    vis.style("opacity", 1e-6)
      .style("opacity", 1)

    force.on("tick", ->
      link.attr("x1", (d) -> d.source.x).attr("y1", (d) -> d.source.y).attr("x2", (d) -> d.target.x).attr("y2", (d) -> d.target.y)
      node.attr("transform", (d) -> "translate(" + d.x + "," + d.y + ")")


Add the following at the bottom of the file

<div id='graph'> </div>


Make sure things are loading in the correct order.

//= require jquery
//= require jquery_ujs
//= require d3
//= require d3.geom
//= require d3.layout
//= require users


circle.node {
    stroke: #fff;
    stroke-width: 1.5px;

line.link {
    stroke: #999;
    stroke-opacity: .6;

Test it

Open a browser localhost:3000/users and scroll down

Better Navigation for Users


Add the following before the link_to lines

  <b>Used tags</b>
  <% @user.used_tags.each do |tag| %>
    <%= link_to tag.name, tag %> <br/>
  <% end %>

  <% @knows.each do |user| %>
    <%= link_to user, user %> <br/>
  <% end %>
  <%= will_paginate(@knows) %>

  <b>Mentioned from:</b><br/>
  <% @mentioned_from.each do |tweet| %>
    <%= link_to tweet, tweet %> <br/>
  <% end %>

  <% @user.tweeted.each do |tweet| %>
    <%= link_to tweet, tweet %> <br/>
  <% end %>


Change the show function to:

def show
  @user = User.find(params[:id])
  @knows = @user.knows.paginate(:page => params[:page], :per_page => 10)

  @mentioned_from = @user.mentioned_from

  respond_to do |format|
    format.html # show.html.erb
    format.json { render :json => @user }

Notice that we do pagination of a traversal (knows).

Better Navigation for Tweets


Show the relationships from and to a tweet:

  <%= @tweet.text %>

  <%= link_to @tweet.link, @tweet.link  %>

  <%= @tweet.date %>

  <%= @tweet.tweet_id %>

  <b>Tweeted by</b>
  <%= link_to @tweet.tweeted_by, @tweet.tweeted_by %>

  <% @tweet.tags.each do |tag| %>
    <%= link_to tag.name, tag %> <br/>
  <% end %>

  <% @tweet.links.each do |link| %>
    <%= link_to link, link %> <br/>
  <% end %>

Better Navigation for Tags



<% @tweets.each do |tweet| %>
  <%= link_to tweet, tweet %> <br/>
<% end %>
<%= will_paginate(@tweets) %>



Add the following line in the show method after the Tag.find line

@tweets = @tag.tweets.paginate(:page => params[:page], :per_page => 10)



The following algorithm works like this:

1. Get all the users who are also using my tags.
2. For each of those users get their used tags and compare with mine.
3. Select the user who has the most similar tags to mine.

Add the following at the bottom of the file user_controllers.rb


def recommend(user)
  my_tags = user.used_tags.to_a
  my_friends = user.knows.to_a

  # we are here using the raw java API - that's why using _java_node, raw and wrapper
  other_users =  user._java_node.outgoing(:used_tags).incoming(:used_tags).raw.depth(2).filter{|path| path.length == 2 && !my_friends.include?(path.end_node)}

  # for all those users, find the person who has the max number of same tags as I have
  found = other_users.max_by{|friend| (friend.outgoing(:used_tags).raw.map{|tag| tag[:name]} & my_tags).size }

  found && found.wrapper # load the ruby wrapper around the neo4j java node


Add one line to the show method:

def show
  @user = User.find(params[:id])
  @recommend = recommend(@user)

  respond_to do |format|
    format.html # show.html.erb
    format.json { render :json => @user }


Display a recommendation if available

<% if @recommend %>
  <%= link_to @recommend.twid, @recommend %>
<% end %>