Skip to content

Clojure tuples #94

Merged
merged 15 commits into from Feb 2, 2012

2 participants

@schleyfox

These commits address #25 and add some improved semantics to the Clojure DSL.

Stream/component/task info on a tuple is now accessible from the meta data.

Tuples now implement ILookup, IPersistentMap, Indexed, Seqable, IFn, and Map so that things like (nth tuple 1), (tuple :foo-field), (:foo-field tuple), etc. work.

Added an IndifferentAccessMap to allow tuple field names to be treated as either keywords or strings.

Tuples can now be "modified" like clojure maps with assoc/dissoc. This is useful for building up values to emit. This is kind of a naive implementation, but I think it works pretty well for the task at hand.

From the Clojure side, emit-bolt!/emit-bolt-direct! can now take its values as either a list or a map. If it is provided a map, then it will find the output fields for the stream and construct the values list from the map based on that (currently null for any fields not present in the map, which I think is sensible). We've been using a pattern like this on our project at NabeWise and I find it to be very powerful/convenient.

I apologize in advance that my java-fu is somewhat weak, so let me know if I need to fix any style or other issues.

@schleyfox

ok, I need to fix some lazy seq bugs. Working on running my project on this version.

@nathanmarz
Owner

I'm not totally sold yet on exposing the TopologyContext in the output collector. I'll have to think if there's a better way to get at that information.

I think that mk-tuple-values should be changed to a protocol. That should be faster and more flexible.

@schleyfox

Yeah, that was a dirty hack. I think the better approach would be to let output collectors take a list or a map, but that would require fairly far reaching changes across the system.

I also really need to switch out stringify keys in mk-tuple-values with a comprehension that doesn't mess with values that are maps.

@nathanmarz
Owner

Instead of exposing the TopologyContext in OutputCollector, how about we make the Clojure DSL "collector" be a map containing :output-collector and :context. Then the emit and ack functions can be modified appropriately to make use of that and we don't have to change the Java interfaces.

@nathanmarz
Owner

BTW, when you update a pull request leave a comment. Just adding commits doesn't send me a notification :(

@schleyfox

Okay, I made those changes. Everything seems to work well (at least in storm's test and my project).

schleyfox added some commits Jan 17, 2012
@schleyfox schleyfox silly mistake in args hinting 8a64e43
@schleyfox schleyfox Merge branch 'master' into clojure_tuples
Conflicts:
	src/clj/backtype/storm/testing.clj
	src/jvm/backtype/storm/drpc/CoordinatedBolt.java
	src/jvm/backtype/storm/tuple/Tuple.java
526dcc0
@schleyfox

Merged to latest master

@nathanmarz nathanmarz merged commit 526dcc0 into nathanmarz:master Feb 2, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.