Skip to content
This repository
Newer
Older
100644 205 lines (141 sloc) 9.339 kb
5083d1f4 » Darrick Wiebe
2011-03-28 Switch to markdown
1 # Pacer
2
aac71a64 » Darrick Wiebe
2010-12-20 Updated the readme.
3 Pacer is a JRuby library that enables very expressive graph traversals.
4
ca459f63 »
2012-08-10 Update README
5 It currently supports 2 major graph database: [Neo4j](http://neo4j.org)
6 and [Dex](http://www.sparsity-technologies.com/dex) using the
7 [Tinkerpop](http://tinkerpop.com) graphdb stack. Plus there's a very
8 convenient in-memory graph called TinkerGraph which is part of
9 [Blueprints](http://blueprints.tinkerpop.com).
10
11 Pacer allows you to create, modify and traverse graphs using very fast
12 and memory efficient stream processing thanks to the very cool
13 [Pipes](http://pipes.tinkerpop.com) library. That also means that almost
14 all processing is done in pure Java, so when it comes the usual Ruby
15 expressiveness vs. speed problem, you can have your cake and eat it too,
16 it's very fast!
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
17
4aafc292 » Darrick Wiebe
2011-08-15 Mailing list link
18 ## Mailing List
19
20 With the release of 0.8.1, I just set up a brand new [pacer google
21 group](http://groups.google.com/group/pacer-users?lnk=gcimv). Join and
22 let's get the conversation going!
23
dbc9a476 » Darrick Wiebe
2011-04-08 bit more doco
24 ## Documentation
25
26 Pacer is documented with a comprehensive RSpec test suite and with a
27 thorough YARD documentation. [Dig in!](http://rubydoc.info/github/pangloss/pacer/develop/frames)
28
29 If you like, you can also use the documentation locally via
30
31 gem install yard
32 yard server
33
fc7b165e »
2012-08-24 Require JRuby 1.7.
34 ## JRuby 1.7 Required
35
36 As of Pacer 1.0.0, [JRuby](http://jruby.org/) 1.7 is required because it is both faster and
37 fixes some problems that I previously had to work around in Pacer.
38
39 The easiest way (ie. the way I do it) to get the most recent version of
40 JRuby is to use [RVM](https://rvm.io/)
41
dbc9a476 » Darrick Wiebe
2011-04-08 bit more doco
42 ## Installation
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
43
dbc9a476 » Darrick Wiebe
2011-04-08 bit more doco
44 The easiest way to get Pacer is `gem install pacer`.
45
46 If you want to hack on Pacer, you'll need to have
47 [maven](http://maven.apache.org/) installed (I recommend `brew install
48 maven`), then use `rake jar` to set up maven's pom.xml file and run the
ca459f63 »
2012-08-10 Update README
49 maven build script. Then `rake gemfile_devel` to set up a default
50 Gemfile that you can customize as needed to include dependencies
51 to the various pacer-... graphs you want the test suite to run against.
dc893495 » Darrick Wiebe
2011-04-09 Note about building Pipes.
52
b6be4306 » Darrick Wiebe
2011-04-08 List of graph db plugins.
53 ## Graph Database Support
54
55 The Tinkerpop suite supports a number of graph data stores. They are all
56 compatible with Pacer, but I have not yet implemented the simple
57 adapters Pacer needs to use them yet. Here is the list of what's
58 supported so far:
59
60 * TinkerGraph - In-memory graph db, built in and ready to use without
61 additional dependencies.
62 * [Neo4J](http://neo4j.org) - The industry-leading graph db. `gem
63 install pacer-neo4j`
64 [pangloss/pacer-neo4j](https://github.com/pangloss/pacer-neo4j)
65 * [Dex](http://sparsity-technologies.com) - A very fast, relatively new graph db. `gem
66 install pacer-dex`
67 [pangloss/pacer-dex](https://github.com/pangloss/pacer-dex)
68
69 You can run any or all of the above graph databases. Pacer supports
70 running them simultaneuosly and even supports having many of any given
71 type open at once.
72
bcbc3712 » Darrick Wiebe
2011-08-15 Describe interoperability with neo4j gem
73 ### Interoperation with the neo4j gem
74
75 Pacer can work together with other Ruby GraphDB libraries, too. The
76 first functioning example is with theo neo4j gem. Hopefully more will
77 follow soon as I find out about them or get requests to support them.
78
79 To use Pacer together with the neo4j gem, get your Pacer graph instance
80 as follows:
81
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
82 ```ruby
1bc3a6f2 » Darrick Wiebe
2011-08-16 Show a bit more detail in the neo4j gem example.
83 require 'neo4j'
84 require 'pacer-neo4j'
85 Neo4j.db.start
bcbc3712 » Darrick Wiebe
2011-08-15 Describe interoperability with neo4j gem
86 graph = Pacer.neo4j(Neo4j.db.graph)
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
87 ```
bcbc3712 » Darrick Wiebe
2011-08-15 Describe interoperability with neo4j gem
88
89 After that, you can continue to use the graph as normal with *both*
90 gems. Any update that's committed with one gem will be visible
91 immediately to the other because they are now both pointing to the same
92 Java graphdb instance.
93
b6be4306 » Darrick Wiebe
2011-04-08 List of graph db plugins.
94 ### A note on safely exiting
95
96 Some databases need to be shutdown cleanly when the program exits. You
97 can shut a database down anytime by calling `graph.shutdown`, but you
98 don't need to worry about it much. Pacer uses Ruby's `at_exit` hook to
99 automatically shut down all open databases!
100
dbc9a476 » Darrick Wiebe
2011-04-08 bit more doco
101 ## Example traversals
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
102
103 Friend recommendation algorithm expressed in basic traversal functions:
104
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
105 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
106 friends = person.out_e(:friend).in_v(:type => 'person')
107 friends.out_e(:friend).in_v(:type => 'person').except(friends).except(person).most_frequent(0...10)
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
108 ```
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
109
110 or using Pacer's route extensions to create your own query methods:
111
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
112 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
113 person.friends.friends.except(person.friends).except(person).most_frequent(0...10)
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
114 ```
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
115
116 or to take it one step further:
117
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
118 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
119 person.recommended_friends
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
120 ```
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
121
5083d1f4 » Darrick Wiebe
2011-03-28 Switch to markdown
122 ## Create and populate a graph
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
123
124 To get started, you need to know just a few methods. First, open up a graph (if one doesn't exist it will be automatically created) and add some vertices to it:
125
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
126 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
127 dex = Pacer.dex '/tmp/dex_demo'
128 pangloss = dex.create_vertex :name => 'pangloss', :type => 'user'
129 okram = dex.create_vertex :name => 'okram', :type => 'user'
130 group = dex.create_vertex :name => 'Tinkerpop', :type => 'group'
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
131 ```
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
132
133 Now, let's see what we've got:
134
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
135 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
136 dex.v
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
137 ```
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
138
139 produces:
140
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
141 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
142 #<V[1024]> #<V[1025]> #<V[1026]>
143 Total: 3
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
144 => #<GraphV>
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
145 ```
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
146
147 There are our vertices. Let's look their properties:
148
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
149 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
150 dex.v.properties
151
152 {"name"=>"pangloss", "type"=>"user"} {"name"=>"okram", "type"=>"user"}
153 {"name"=>"Tinkerpop", "type"=>"group"}
154 Total: 3
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
155 => #<GraphV -> Obj-Map>
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
156 ```
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
157
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
158 Now let's put an edge between them:
159
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
160 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
161 dex.create_edge okram, pangloss, :inspired
162 => #<E[2048]:1025-inspired-1024>
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
163 ```
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
164
165 That's great for creating an edge but what if I've got lots to create? Try this method instead which can add edges to the cross product of all vertices in one route with all vertices in the other:
166
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
167 ```ruby
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
168 group.add_edges_to :member, dex.v(:type => 'user')
169
170 #<E[4097]:1026-member-1024> #<E[4098]:1026-member-1025>
171 Total: 2
172 => #<Obj 2 ids -> lookup>
12ac36dc » Darrick Wiebe
2011-10-03 Syntax highlighting in readme
173 ```
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
174
175 There is plenty more to see as well! Please dig into the code and the spec suite to find loads of examples and edge cases. And if you think of a case that I've missed, I'll greatly appreciate your contributions!
176
5083d1f4 » Darrick Wiebe
2011-03-28 Switch to markdown
177 ## Design Philosophy
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
178
179 I want Pacer and its ecosystem to become a repository for real implementations of ideas, best practices and techniques for streaming data manipulation. I've got lots of ideas that I'd like to add, and although Pacer seems to be quite rock solid right now -- and I am using it in limited production environments -- it is still in flux. If we find a better way to do something, we're going to do it that way even if that means breaking changes from one release to another.
180
181 Once Pacer matures further, a decision will be made to 'lock it down' at least a little more, hopefully there will be a community in place by then to help determine the right time for that to happen!
182
2fac2a13 » Darrick Wiebe
2011-03-29 Note on plugins in the Readme
183 ## Pluggable Architecture
184
7b0cd89f » Darrick Wiebe
2011-03-29 editing
185 Pacer is meant to be extensible and is built on a very modular architecture. Nearly every chainable route method is actually implemented in an independent module that is plugged into the route only when it's in use. That allows great flexibility in adding methods to routes without clogging up the method namespace. It also makes it natural to make pacer plugin gems.
186
187 There are lots of examples of route extensions right inside Pacer. Have a look at the [lib/pacer/filter](https://github.com/pangloss/pacer/tree/develop/lib/pacer/filter), [side_effect](https://github.com/pangloss/pacer/tree/develop/lib/pacer/side_effect) and [transform](https://github.com/pangloss/pacer/tree/develop/lib/pacer/transform) folders to see the modules that are built into Pacer. They vary widely in complexity, so take a look around.
188
189 If you want to add a traversal technique to Pacer, you can fork Pacer and send me a pull request or just create your own pacer-&lt;feature name&gt; plugin! To see how to build your own Pacer plugin, see my [example pacer-bloomfilter plugin](https://github.com/pangloss/pacer-bloomfilter) which also has a readme file that goes into considerable detail on the process of creating plugins and provides some additional usage examples as well.
2fac2a13 » Darrick Wiebe
2011-03-29 Note on plugins in the Readme
190
7b0cd89f » Darrick Wiebe
2011-03-29 editing
191 As a side note, don't worry about any magic happening behind the scenes to discover or automatically load pacer plugins, there is none of that! If you want to use a pacer plugin, treat it like any other gem, add it to your Gemfile (if that's what you use) and <code>require</code> the gem as normal if you need it.
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
192
5083d1f4 » Darrick Wiebe
2011-03-28 Switch to markdown
193 ## Gremlin
f41741df » Darrick Wiebe
2011-03-27 Working on the Readme
194
5083d1f4 » Darrick Wiebe
2011-03-28 Switch to markdown
195 If you're already familiar with [Gremlin](http://gremlin.tinkerpop.com), please look at my [Introducing Pacer](http://ofallpossibleworlds.wordpress.com/2010/12/19/introducing-pacer) post for a simple introduction and explanation of how Pacer is at once similar to and quite different from Gremlin, the project that inspired it. That post is a little out of date at this point since it refers to the original version of Gremlin. Groovy Gremlin is the latest version, inspired in turn by Pacer!
aac71a64 » Darrick Wiebe
2010-12-20 Updated the readme.
196
c5b807b5 » jayniz
2011-09-06 Added a link to Marko's introduction to the concept of pipes
197 A great introduction to the underlying concept of pipes can be found in Marko Rodriguez' post [On the Nature of Pipes](http://markorodriguez.com/2011/08/03/on-the-nature-of-pipes/)
c1a3970b » jayniz
2011-09-06 Added a link to Marko's introduction to the concept of pipes
198
5083d1f4 » Darrick Wiebe
2011-03-28 Switch to markdown
199 ## Test Coverage
aac71a64 » Darrick Wiebe
2010-12-20 Updated the readme.
200
2690b812 » Darrick Wiebe
2011-03-28 Finally, a proper readme
201 I'm aiming for 100% test coverage in Pacer and am currently nearly there in the core classes, but there is a way to go with the filter, transform and side effect route modules. Open coverage/index.html to see the current state of test coverage. And of course contributions would be much apreciated.
b7a95687 » Darrick Wiebe
2012-03-22 Add style guide note to readme
202
203 ## Style Guide
204
ca459f63 »
2012-08-10 Update README
205 Please follow Github's [Ruby style guide](https://github.com/styleguide/ruby) when contributing to make your patches more likely to be accepted!
Something went wrong with that request. Please try again.