Skip to content

it is very difficult to install flockdb #14

Open
daywednes opened this Issue Oct 6, 2010 · 23 comments

5 participants

@daywednes

The installation tutorial doesn't cover everything. I had to google alot different things but ended up with nothing.-

@robey
robey commented Oct 6, 2010

can you tell me what you wish the instructions included?

@daywednes

i'm not familiar with scala, so i don't know how to config it.

there is no help to show what is sbt ? i had to google and found out http://gist.github.com/510080 . It was helpful. But when i tried to sbt clean update package-dist , it was running well, but then i got a request asking for svn password which i did not understant . there was something like :

mdoan@mdoan:~/work/git/flockdb_git$ sbt clean update package-dist
[info] Standard project rules 0.7.5 loaded (2010-09-21).
[warn] No .svnrepo file; no svn repo will be configured.
[info] Building project flockdb 1.4.0 against Scala 2.7.7
[info] using FlockDBProject with sbt 0.7.4 and Scala 2.7.7
[info]
[info] == clean-thrift ==
[info] Deleting directory /home/mdoan/work/git/flockdb_git/target/gen-java
[info] Deleting directory /home/mdoan/work/git/flockdb_git/target/gen-rb
[info] == clean-thrift ==
[info]
[info] == clean-dist ==
[info] == clean-dist ==
[info]
[info] == clean ==
[info] Deleting directory /home/mdoan/work/git/flockdb_git/target
[info] == clean ==
[success] Successful.
[info]
[info] Total time: 0 s, completed Oct 6, 2010 11:51:19 AM
[info]
[info] == update ==
IvySvn Build-Version: null
IvySvn Build-DateTime: null
SVN repository password:

And one thing is i don't understand how to install gizzmo. should i install it before running sbt clean update package-dist .

anyhelp will be appreciated.

@robey
robey commented Oct 7, 2010

ok, i just edited the docs for gizzard & flockdb to include more info on sbt and thrift. hopefully that helps a bit. it does suck that the thrift site is still so shaky.

i think the gizzard version info in that gist is old and isn't needed anymore.

@daywednes

Thanks robey so much.
is the higher version than thrift 0.2.0 of thrift is ok ? I still got a compiling problem.

@robey
robey commented Oct 7, 2010

i don't know why it's asking you for an svn password. but i can see that too, so i'm going to dig into that. you should be able to bypass the svn password prompt by just typing in gibberish and hitting enter.

@robey
robey commented Oct 7, 2010

no, you have to use version 0.2.0 exactly for now. we hope to upgrade soon, but it requires code changes.

@daywednes

I got the followings:

mdoan@mdoan:~/work/git/flockdb_git$ NO_TESTS=1 sbt package-dist
[info] Standard project rules 0.7.5 loaded (2010-09-21).
[warn] No .svnrepo file; no svn repo will be configured.
[info] Building project flockdb 1.4.0 against Scala 2.7.7
[info]    using FlockDBProject with sbt 0.7.4 and Scala 2.7.7
[info] 
[info] == compile-thrift-ruby ==
[info] == compile-thrift-ruby ==
[info] 
[info] == compile-thrift-java ==
[info] == compile-thrift-java ==
[info] 
[info] == compile ==
[info]   Source analysis: 65 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[error] Note: Some input files use or override a deprecated API.
[error] Note: Recompile with -Xlint:deprecation for details.
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/Edge.scala:19: value xrayspecs is not a member of package com.twitter
[error] import com.twitter.xrayspecs.Time
[error]                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/EdgeQuery.scala:19: value results is not a member of package com.twitter
[error] import com.twitter.results.Page
[error]                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/EdgesService.scala:19: value gizzard is not a member of package com.twitter
[error] import com.twitter.gizzard.Future
[error]                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/EdgesService.scala:20: value gizzard is not a member of package com.twitter
[error] import com.twitter.gizzard.jobs.CopyFactory
[error]                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/EdgesService.scala:21: value gizzard is not a member of package com.twitter
[error] import com.twitter.gizzard.nameserver.NameServer
[error]                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/EdgesService.scala:22: value gizzard is not a member of package com.twitter
[error] import com.twitter.gizzard.scheduler.PrioritizingJobScheduler
.....
@robey
robey commented Oct 7, 2010

i think i fixed the svn password prompt. for the missing classes, can you try running "sbt update" again? all of those classes should be coming from dependent jars.

@daywednes

I just got another error.

info] Standard project rules 0.7.8 loaded (2010-10-06).
[warn] No .svnrepo file; no svn repo will be configured.
[info] Building project flockdb 1.4.0 against Scala 2.7.7
[info]    using FlockDBProject with sbt 0.7.4 and Scala 2.7.7
[info] 
[info] == clean-thrift ==
[info] Deleting directory /home/mdoan/work/git/flockdb_git/target/gen-java
[info] Deleting directory /home/mdoan/work/git/flockdb_git/target/gen-rb
[info] == clean-thrift ==
[info] 
[info] == clean-dist ==
[info] == clean-dist ==
[info] 
[info] == clean ==
[info] Deleting directory /home/mdoan/work/git/flockdb_git/target
[info] == clean ==
[success] Successful.
[info] 
[info] Total time: 0 s, completed Oct 6, 2010 8:19:41 PM
[info] 
[info] == update ==
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/xrayspecs/1.0.7/xrayspecs-1.0.7.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/xrayspecs/1.0.7/xrayspecs-1.0.7.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/org/scala-tools/vscaladoc/1.1-md-3/vscaladoc-1.1-md-3.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/org/scala-tools/vscaladoc/1.1-md-3/vscaladoc-1.1-md-3.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/net/lag/kestrel/1.2/kestrel-1.2.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/net/lag/kestrel/1.2/kestrel-1.2.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/results/1.0/results-1.0.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/results/1.0/results-1.0.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/thrift/libthrift/0.2.0/libthrift-0.2.0.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/thrift/libthrift/0.2.0/libthrift-0.2.0.jar
[info] 
[error] :: problems summary ::
[error] :::: ERRORS
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/xrayspecs/1.0.7/xrayspecs-1.0.7.pom
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/xrayspecs/1.0.7/xrayspecs-1.0.7.jar
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/org/scala-tools/vscaladoc/1.1-md-3/vscaladoc-1.1-md-3.pom
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/org/scala-tools/vscaladoc/1.1-md-3/vscaladoc-1.1-md-3.jar
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/net/lag/kestrel/1.2/kestrel-1.2.pom
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/net/lag/kestrel/1.2/kestrel-1.2.jar
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/results/1.0/results-1.0.pom
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/results/1.0/results-1.0.jar
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/thrift/libthrift/0.2.0/libthrift-0.2.0.pom
[error]     Server access Error: Connection refused url=http://maven.reucon.com/public/thrift/libthrift/0.2.0/libthrift-0.2.0.jar
[info] 
[info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
[info] :: retrieving :: com.twitter#flockdb [sync]
[info]  confs: [compile, runtime, test, provided, system, optional, sources, javadoc]
[info]  0 artifacts copied, 38 already retrieved (0kB/36ms)
[info] == update ==
[success] Successful.
[info] 
[info] Total time: 48 s, completed Oct 6, 2010 8:20:29 PM
[info] 
[info] == compile-thrift-ruby ==
[info] == compile-thrift-ruby ==
[info] 
[info] == compile-thrift-java ==
[info] == compile-thrift-java ==
[info] 
[info] == compile ==
[info]   Source analysis: 65 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[error] Note: Some input files use or override a deprecated API.
[error] Note: Recompile with -Xlint:deprecation for details.
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/QueryTerm.scala:35: value toLongArray is not a member of java.nio.ByteBuffer
[error]       Some(term.destination_ids.toLongArray)
[error]                                 ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/Results.scala:25: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]     def toThrift = new thrift.Results(resultWindow.toList.pack, resultWindow.nextCursor.position,
[error]                                                           ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/FlockDB.scala:154: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]     edges.count(queries.toSeq.map { _.toSeq.map { _.fromThrift }}).pack
[error]                                                                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/QueryTerm.scala:26: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]       term.destinationIds.map { x => thriftTerm.setDestination_ids(x.pack) }
[error]                                                                      ^
[error] four errors found
[info] == compile ==
[info] 
[info] == copy-resources ==
[info] == copy-resources ==
[info] 
[info] == copy-test-resources ==
[info] == copy-test-resources ==
[info] 
[info] == write-build-properties ==
[info] == write-build-properties ==
[error] Error running compile: Compilation failed
[info] 
[info] Total time: 7 s, completed Oct 6, 2010 8:20:37 PM
[info] 
[info] Total session time: 56 s, completed Oct 6, 2010 8:20:37 PM
[error] Error during build.
@ngonpham
ngonpham commented Oct 7, 2010

I try to install but got 'Server access Error: Connection refused ', could someone please help? Thanks ahead

$ sudo sbt clean update package-dist
[info] Standard project rules 0.7.8 loaded (2010-10-06).
[warn] No .svnrepo file; no svn repo will be configured.
[info] Building project flockdb 1.4.0 against Scala 2.7.7
[info] using FlockDBProject with sbt 0.7.4 and Scala 2.7.7
[info]
[info] == clean-thrift ==
[info] Deleting directory /home/ngon/flockdb/target/gen-java
[info] Deleting directory /home/ngon/flockdb/target/gen-rb
[info] == clean-thrift ==
[info]
[info] == clean-dist ==
[info] == clean-dist ==
[info]
[info] == clean ==
[info] Deleting directory /home/ngon/flockdb/target
[info] == clean ==
[success] Successful.
[info]
[info] Total time: 0 s, completed Oct 6, 2010 10:24:08 PM
[info]
[info] == update ==
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/org/scala-tools/vscaladoc/1.1-md-3/vscaladoc-1.1-md-3.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/org/scala-tools/vscaladoc/1.1-md-3/vscaladoc-1.1-md-3.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/xrayspecs/1.0.7/xrayspecs-1.0.7.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/com/twitter/xrayspecs/1.0.7/xrayspecs-1.0.7.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/thrift/libthrift/0.2.0/libthrift-0.2.0.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/thrift/libthrift/0.2.0/libthrift-0.2.0.jar
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/net/lag/kestrel/1.2/kestrel-1.2.pom
[error] Server access Error: Connection refused url=http://maven.reucon.com/public/net/lag/kestrel/1.2/kestrel-1.2.jar
.....

@robey
robey commented Oct 7, 2010

you can ignore the messages about reucon being down. we don't depend on anything from that repo right now, so the error messages are just maven whining. ugh, maven.

@daywednes

Thank robey,

How about 4 compiling errors mentioned above ? Something is wrong with thrift. Could you look into them ?

@robey
robey commented Oct 11, 2010

that weirdly seems like an implicit isn't being picked up. let me blow away my ivy cache and start over.

@robey
robey commented Oct 11, 2010

seems to build okay when i do that. in case you haven't tried this yet, blow away your own ivy cache and try a fresh "sbt clean update package-dist". i think something is messed up with getting the gizzard dependency.

@daywednes

does the 'sbt clean update package-dist' command install the gizzard? I think i did not install gizzard.

Another thing is I have not done the following things required from TODO file. Might it be the problem ?

move com.twitter.service.flock.Reset into gizzard
move com.twitter.results into gizzard or its own package.
move ByteSwapper into gizzard
merge in Gizzard's no_reflection branch.
move StatsCollectingQuery where??
create a `main` file by copying glock
create simple exceptionWrappingProxy using new gizzard niceness interface (cf Rowz)
rename Edges.scala to Flockdb.scala
figure out where Flock.thrift goes
move State into com.twitter.flockdb; make a copy for Groups; they're only identical on accident. not the same thing.
copy the production.conf config from glock (but be careful with passwords!!!!)
rename Edges.thrift Flock.thrift
figure out how to package conf file??
@daywednes

just tried to do compiling again. it remained a few errors as follows:
I'm not familiar with scala, and gizzard. Could you make a guess for them ?

[info] Compiling main sources...
[error] Note: Some input files use or override a deprecated API.
[error] Note: Recompile with -Xlint:deprecation for details.
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/QueryTerm.scala:35: value toLongArray is not a member of java.nio.ByteBuffer
[error]       Some(term.destination_ids.toLongArray)
[error]                                 ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/Results.scala:25: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]     def toThrift = new thrift.Results(resultWindow.toList.pack, resultWindow.nextCursor.position,
[error]                                                           ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/FlockDB.scala:154: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]     edges.count(queries.toSeq.map { _.toSeq.map { _.fromThrift }}).pack
[error]                                                                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/QueryTerm.scala:26: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]       term.destinationIds.map { x => thriftTerm.setDestination_ids(x.pack) }
[error]                                                                      ^
[error] four errors found
[info] == compile ==
@robey
robey commented Oct 11, 2010

sorry, i don't know. :( if you really did blow away your ivy cache and run update again, i don't know why you aren't getting the right version of gizzard.

@daywednes

oh, the clean and update tasks have been done successfully. I might have made your misunderstanding.

in my previous reply, i found 4 errors in compiling like :

[info] == compile ==
[info]   Source analysis: 65 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[error] Note: Some input files use or override a deprecated API.
[error] Note: Recompile with -Xlint:deprecation for details.
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/QueryTerm.scala:35: value toLongArray is not a member of java.nio.ByteBuffer
[error]       Some(term.destination_ids.toLongArray)
[error]                                 ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/Results.scala:25: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]     def toThrift = new thrift.Results(resultWindow.toList.pack, resultWindow.nextCursor.position,
[error]                                                           ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/FlockDB.scala:154: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]     edges.count(queries.toSeq.map { _.toSeq.map { _.fromThrift }}).pack
[error]                                                                    ^
[error] /home/mdoan/work/git/flockdb_git/src/main/scala/com/twitter/flockdb/conversions/QueryTerm.scala:26: type mismatch;
[error]  found   : Array[Byte]
[error]  required: java.nio.ByteBuffer
[error]       term.destinationIds.map { x => thriftTerm.setDestination_ids(x.pack) }
[error]                                                                      ^
[error] four errors found
[info] == compile ==

@ryd
ryd commented Oct 25, 2010

I got exactly the same type mismatch.

@robey
robey commented Oct 25, 2010

those look like errors in loading Sequences.scala, but i don't know why that would happen.

@ryd
ryd commented Oct 27, 2010

Ok - got it running - the problem is the wrong thrift version in PATH

http://apache.mirroring.de//incubator/thrift/0.2.0-incubating/

This Version is running - newer version have changes which make it incompatible with
the version that is needed for flockdb - one example is binary, which is changed to BitSet in newer versions to reduce memory footprint.

I will try to write a patch fixing that - but I don't know if I have enough time for that in near future.

@robey
robey commented Nov 8, 2010

ah, yeah, we definitely still need 0.2.0 because they made incompatible changes in later versions.

if you can think of a way to highlight that more in the readme, i'll do it. a couple of other people had the same problem.

@psitt
psitt commented Jun 18, 2011

Hi. I installed thrift-0.2.0 and sbt-launch-0.7.4. I downloaded and unzipped flockbd-1.6.15. Then, I run "sbt clean update package-dist". After a lot of compilations errors (90 to be more specific) I changed libthrift-0.5.0.jar (in lib_managed/compile/ dir) by libthrift-0.2.0.jar.

Now, I still have the four errors previously mentionated by ryd and daywednes. Can you teach us how did you resolve this problem?

Thank you very much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.