Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Updated plugin to work with the latest Flume version. #1

Merged
merged 7 commits into from
+22,238 −11,519

2 participants

@swillis11

I would appreciate the time you take to review these changes. I have tested these changes and they work on Flume fresh off of the git repo, 0.9.5-SNAPSHOT I believe. The only thing I really had to change was some byte arrays to ByteBuffers in some of the code and also the thrift generated bindings generated from cassandra.thrift from cassandra-0.8.2 with thrift-0.6.1 installed. It seems to be working just as the old plugin did on Flume-0.9.1. So far it's working great for me with the latest version of Cassandra and the latest version of Flume. It might be good to be more clear in the Docs about what version of thrift should be used in order to build. The ANT build xml points to a libthrit.jar which doesn't indicate a version of thrift, I had to download like 5 thrift versions until I found the right one, it happened to be thrift-0.2.0.jar. It might be better to include a libthrift-0.6.1.jar or something.

@thobbs
Owner

Awesome! Thanks for tackling this. I'll review your changes as soon as I can get to them, which should be this week.

@swillis11

Yeah no problem. I'm new to flume/cassandra and git for that matter.. so just let me know if I didn't do something quite right.

Thanks!

@thobbs
Owner

Would you mind sharing what you needed to change in Flume and where you put the plugin jar(s) in order to get the plugin to work?

The Flume project layout and build process has changed considerably since the last time I looked at it.

@swillis11
@swillis11
@swillis11

Alright, so to my run configuration classpath for the master and the node in eclipse I added /flume-cassandra-plugin/build/classes (OR cassandra_plugin.jar compiled from the ant build) and jug-asl.2.0.0.jar.

Also in run configurations, an environment variable FLUME_HOME that points to flume/flume-core. And inside of the flume core directory I linked the conf directory because the flume_home var tells flume where to look for the config. You could set it to just the top flume dir too probably.

After that I just added the plugin to load in the flume-site.xml based on your instructions and then configured the node from the flume-shell within eclipse.

@swillis11

Also when you're building, make sure thrift-0.6.x.jar is in the build path (with jug-asl-2.0.0.jar and slf4j-api-1.5.11.jar) since the generated bindings are from thrift 0.6.1. Also since my flume-cassandra-plugin directory wasn't in flume's directory, I had to change the flume.base in the build.xml..

@thobbs
Owner

Finally figured it out, and it looks like the simpleCassandraSink works well. (The logsandraSink is out of date, but that's my own doing.)

Thanks for the pull request and for the tips! Very much appreciated.

@thobbs thobbs merged commit 72beeff into thobbs:master
@swillis11

Yeah no problem, anything I can do to save someone else some grief. The new docs and the download are great, thanks for doing that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 3, 2011
  1. Thrift bindings for Thrift 0.6.0

    swillis authored
  2. New for thrift 0.6.0

    swillis authored
  3. new for thrift 0.6.0

    swillis authored
  4. new for thrift 0.6.0

    swillis authored
  5. Old Bindings

    swillis authored
  6. Updated thrift bindings for Thrift-0.6.1: Compatible with

    swillis authored
    Flume-0.9.3+, including Flume-0.9.5-SNAPSHOT.
Something went wrong with that request. Please try again.