Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add support for composite column types #8

Merged
merged 14 commits into from

2 participants

@mstump

This is in reference to gh-7, adding composite column support to clj-hector.

It's not quite ready to be merged yet, but I wanted to show you the direction I was headed so you could give feedback early on. It's been several years since I've last written Clojure code so any criticisms would be welcome.

The major issue outstanding is that the Component values that are returned as part of a Composite column when performing a query are always HeapByteBuffers instead of the expected type. After this is fixed I think I might be done.

mstump added some commits
@mstump mstump I have composite columns mostly done, the only problem is that compos…
…ite column values in query results are being returned as HeapByteBuffer instead of string
60806c3
@mstump mstump in the toClojure protocol switched from using the concrete type Compo…
…site to AbstractComposite so that we can also deserialze DynamicComposite which shares the same base
3b6ef9a
@mstump mstump added dynamic-composite serializer ba9401f
@mstump mstump finished implementing DynamicComposite and it doesn't look like it's …
…affected by the bug that's holding up Composite
6073997
@mstump mstump added test for dynamic composite columns and column range queries 45573ed
@pingles
Owner

Thanks for all this Matt- looks very interesting.

Apologies I've not taken a look through so far- I've been pretty busy with work but I'm hoping to get a day or two over the next week or so to catch up.

From checking your repository it looks like you've progressed further than these commits?

@mstump

No problems regarding the delay. I'm not in a huge to push out a new release of clj-hector. I'm pulling in my branch as a leiningen sub project. I just want to be as transparent as possible.

I've completed support for DynamicComposite, and tests for both row fetch, column slice, and column range queries pass tests.

The Composite issue I was running into was most likely a bug in Hector, as other people are seeing it too. I might be able to work around it.

I'm working on keyspace and column family creation with Composite columns, I'm not sure if the issue I'm running into is my bug or Hector's.

I'm going to spend some time working on it today, but starting tomorrow I need to run off and spend time with the relatives so I'm not sure how much progress I'll make.

Adding this feature is taking a little bit longer than I had hoped. I'm paying two penalties, one for coming up to speed on Clojure, and another because Hector isn't particularly well documented.

@pingles
Owner

Hi Matt,

I just merged in a bunch of changes + refactorings that Nick had done- it changes some of the assumptions around using maps for super-columns and sounds like it may make some of the composite stuff easier (#12).

I'll see if I can get an hour or two tomorrow to play around with the changes you've been working on. Thanks again for your work!

mstump added some commits
@mstump mstump merged upstream bc06329
@mstump mstump discovered bug in CFDef creation where even though we took comparator…
… as an input parameter we weren't setting it on the CFDef
84c9920
@mstump mstump added eclipse artifacts to .gitignore 4daeae2
@mstump mstump added support for specifying deserializers for Composites, but to do …
…this I needed to add an options parameter to the to-clojure protocol which involved touching lots of code
6d61686
@mstump

Alright, so after a bunch of back and froth about how Composite deserialization is supposed to work on hector-users we finally came to the conclusion that it doesn't which explains the behavior I was seeing.

http://groups.google.com/group/hector-users/browse_thread/thread/ba03b9435689dbbc

As a result I added an option called :c-serializer which allows the user to specify an array of serializers to use for deserializing the Component values of a Composite.

All tests are passing and this pull request is ready for inclusion.

@pingles pingles merged commit c0951a4 into pingles:master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 19, 2011
  1. @mstump

    I have composite columns mostly done, the only problem is that compos…

    mstump authored
    …ite column values in query results are being returned as HeapByteBuffer instead of string
Commits on Dec 20, 2011
  1. @mstump

    in the toClojure protocol switched from using the concrete type Compo…

    mstump authored
    …site to AbstractComposite so that we can also deserialze DynamicComposite which shares the same base
  2. @mstump
  3. @mstump

    finished implementing DynamicComposite and it doesn't look like it's …

    mstump authored
    …affected by the bug that's holding up Composite
  4. @mstump
Commits on Dec 21, 2011
  1. @mstump
  2. @mstump

    added test helper so that I can mark tests as pending. Made the compo…

    mstump authored
    …site-serializer test pending. added test for column range fetch for dynamic colums
  3. @mstump

    merged conflicts

    mstump authored
Commits on Dec 22, 2011
  1. @mstump

    cross refrenced the validators, comparators, and column types in clj-…

    mstump authored
    …hector with hector and cassandra. populated clj-hector with missing types
  2. @mstump
Commits on Dec 27, 2011
  1. @mstump

    merged upstream

    mstump authored
Commits on Dec 29, 2011
  1. @mstump

    discovered bug in CFDef creation where even though we took comparator…

    mstump authored
    … as an input parameter we weren't setting it on the CFDef
  2. @mstump
  3. @mstump

    added support for specifying deserializers for Composites, but to do …

    mstump authored
    …this I needed to add an options parameter to the to-clojure protocol which involved touching lots of code
This page is out of date. Refresh to see the latest.
View
9 .gitignore
@@ -5,3 +5,12 @@ classes
autodoc/**
.lein-*
tmp/
+*.swp
+*~
+.DS_Store
+.#*
+#*
+/checkouts
+.project
+.settings/
+bin/
View
101 src/clj_hector/core.clj
@@ -1,13 +1,15 @@
(ns clj-hector.core
^{:author "Paul Ingles"
:doc "Hector-based Cassandra client"}
- (:require [clj-hector.consistency :as c])
(:require [clj-hector.serialize :as s])
+ (:require [clj-hector.ddl :as ddl])
+ (:require [clj-hector.consistency :as c])
(:import [java.io Closeable]
[me.prettyprint.hector.api.mutation Mutator]
[me.prettyprint.hector.api Cluster]
[me.prettyprint.hector.api.factory HFactory]
[me.prettyprint.hector.api.query Query]
+ [me.prettyprint.hector.api.beans Composite DynamicComposite]
[me.prettyprint.cassandra.service CassandraHostConfigurator]
[me.prettyprint.cassandra.serializers TypeInferringSerializer]))
@@ -31,6 +33,7 @@
(defn shutdown-cluster! [c] (HFactory/shutdownCluster c))
(defn cluster-name [c] (.describeClusterName c))
+
(defn partitioner [c] (.describePartitioner c))
(defn keyspace
@@ -54,7 +57,8 @@
:ttl nil
:s-serializer :bytes
:n-serializer :bytes
- :v-serializer :bytes}]
+ :v-serializer :bytes
+ :c-serializer nil}]
(merge defaults (apply hash-map opts) (get *schemas* column-family))))
(defn- query-options
@@ -65,11 +69,12 @@
:limit Integer/MAX_VALUE}]
(merge defaults (apply hash-map opts))))
-(defn- extract-options
+(defn extract-options
[opts cf]
(merge (query-options opts) (schema-options opts cf)))
(def serializer-keys [:n-serializer :v-serializer :s-serializer])
+
(defn convert-serializers [opts]
(reduce (fn [m [k v]] (assoc m k (s/serializer v)))
opts
@@ -77,9 +82,52 @@
(defn- super? [opts]
(= (opts :type :standard) :super))
+
(defn- counter? [opts]
(get opts :counter false))
+(defn- populate-composite
+ "populate the component values of a DynamicComposite or Composite"
+ [composite components]
+ (doseq [c components]
+ (if (map? c)
+ (let [opts (merge {:n-serializer :bytes
+ :comparator :bytes
+ :equality :equal} c)]
+ (.addComponent composite
+ (:value opts)
+ (s/serializer (:n-serializer opts))
+ (.getTypeName (ddl/comparator-types (:comparator opts)))
+ (ddl/component-equality-type (:equality opts))))
+ (.add composite -1 c)))
+ composite)
+
+(defn create-composite
+ "Given a list create a Composite
+
+ Supply a list of hashes to specify Component options for each element in the composite
+
+ ex: [\"col\" \"name\"]
+ ex: [{:value \"col\" :n-serializer :string :comparator :utf-8 :equality :equal}
+ {:value 2 :n-serializer :string :comparator :integer :equality :less_than_equal}]"
+
+ [& components]
+ (let [^Composite composite (new Composite)]
+ (populate-composite composite components)))
+
+(defn create-dynamic-composite
+ "Given a list create a DynamicComposite
+
+ Supply a list of hashes to specify Component options for each element in the composite
+
+ ex: [\"col\" \"name\"]
+ ex: [{:value \"col\" :n-serializer :string :comparator :utf-8 :equality :equal}
+ {:value 2 :n-serializer :string :comparator :integer :equality :less_than_equal}]"
+
+ [& components]
+ (let [^DynamicComposite composite (new DynamicComposite)]
+ (populate-composite composite components)))
+
(defn- create-column
"Creates Column and SuperColumns.
@@ -118,8 +166,18 @@
(doseq [[k v] m] (.addInsertion mut pk cf (create-column k v opts))))
(.execute mut)))
-(defn- execute-query [^Query query]
- (s/to-clojure (.execute query)))
+(defn create-counter-column
+ [name value & {:keys [n-serializer v-serializer s-serializer]
+ :or {n-serializer type-inferring
+ v-serializer type-inferring
+ s-serializer type-inferring}}]
+ (if (map? value)
+ (let [cols (map (fn [[n v]] (create-counter-column n v :n-serializer n-serializer :v-serializer v-serializer)) value)]
+ (HFactory/createCounterSuperColumn name cols s-serializer n-serializer))
+ (HFactory/createCounterColumn name value n-serializer)))
+
+(defn- execute-query [^Query query & [opts]]
+ (s/to-clojure (.execute query) opts))
(defn get-super-rows
"In keyspace ks, from Super Column Family cf, retrieve the rows identified by pks. Executed
@@ -137,7 +195,8 @@
(.setColumnFamily cf)
(.setKeys (into-array pks))
(.setColumnNames (into-array scs))
- (.setRange (:start opts) (:end opts) (:reversed opts) (:limit opts))))))
+ (.setRange (:start opts) (:end opts) (:reversed opts) (:limit opts)))
+ opts)))
(defn get-rows
"In keyspace ks, retrieve rows for pks within column family cf."
@@ -149,7 +208,8 @@
(s/serializer (:v-serializer opts)))
(.setColumnFamily cf)
(.setKeys (into-array pks))
- (.setRange (:start opts) (:end opts) (:reversed opts) (:limit opts))))))
+ (.setRange (:start opts) (:end opts) (:reversed opts) (:limit opts)))
+ opts)))
(defn get-super-columns
"In keyspace ks, for row pk, retrieve columns in c from super column sc."
@@ -163,7 +223,8 @@
(.setColumnFamily cf)
(.setKey pk)
(.setSuperColumn sc)
- (.setColumnNames (into-array c))))))
+ (.setColumnNames (into-array c)))
+ opts)))
(defn get-column-range
"In keyspace ks, retrieve columns between start and end from column family cf."
@@ -174,7 +235,8 @@
(execute-query (doto (HFactory/createSliceQuery ks type-inferring ns vs)
(.setColumnFamily cf)
(.setKey pk)
- (.setRange start end (:reversed opts) (:limit opts))))))
+ (.setRange start end (:reversed opts) (:limit opts)))
+ opts)))
(defn get-columns
"In keyspace ks, retrieve c columns for row pk from column family cf"
@@ -186,11 +248,13 @@
(execute-query (doto (HFactory/createSliceQuery ks type-inferring ns vs)
(.setColumnFamily cf)
(.setKey pk)
- (.setColumnNames (into-array c))))
+ (.setColumnNames (into-array c)))
+ opts)
(execute-query (doto (HFactory/createColumnQuery ks type-inferring ns vs)
(.setColumnFamily cf)
(.setKey pk)
- (.setName c))))))
+ (.setName c))
+ opts))))
(defn get-counter-columns
"Queries counter column values. c is a sequence of column names to
@@ -202,11 +266,13 @@
(execute-query (doto (HFactory/createCounterSliceQuery ks type-inferring ns)
(.setColumnFamily cf)
(.setKey pk)
- (.setColumnNames (into-array c))))
+ (.setColumnNames (into-array c)))
+ opts)
(execute-query (doto (HFactory/createCounterColumnQuery ks type-inferring ns)
(.setColumnFamily cf)
(.setKey pk)
- (.setName (into-array c)))))))
+ (.setName (into-array c)))
+ opts))))
(defn get-counter-rows
"Load data for specified keys and columns"
@@ -217,7 +283,8 @@
(execute-query (doto (HFactory/createMultigetSliceCounterQuery ks kser ns)
(.setKeys pks)
(.setColumnFamily cf)
- (.setColumnNames (into-array cs))))))
+ (.setColumnNames (into-array cs)))
+ opts)))
(defn get-counter-column-range
"Queries for a range of counter columns."
@@ -227,7 +294,8 @@
(execute-query (doto (HFactory/createCounterSliceQuery ks type-inferring ns)
(.setColumnFamily cf)
(.setKey pk)
- (.setRange start end (:reversed opts) (:limit opts))))))
+ (.setRange start end (:reversed opts) (:limit opts)))
+ opts)))
(defn get-counter-super-columns
"Queries for counter values in a super column column family."
@@ -239,7 +307,8 @@
(.setKey pk)
(.setColumnFamily cf)
(.setColumnNames (into-array c))
- (.setRange (:start o) (:end o) (:reversed o) (:limit o))))))
+ (.setRange (:start o) (:end o) (:reversed o) (:limit o)))
+ opts)))
(defn delete-columns
"Deletes columns identified in cs for row pk."
View
79 src/clj_hector/ddl.clj
@@ -4,23 +4,38 @@
(:import [me.prettyprint.hector.api.factory HFactory]
[me.prettyprint.hector.api Cluster]
[me.prettyprint.cassandra.service ThriftCfDef]
+ [me.prettyprint.hector.api.beans Composite AbstractComposite$ComponentEquality]
[me.prettyprint.hector.api.ddl ComparatorType ColumnFamilyDefinition ColumnType KeyspaceDefinition]))
-(def comparator-types {:ascii ComparatorType/ASCIITYPE
- :bytes ComparatorType/BYTESTYPE
- :integer ComparatorType/INTEGERTYPE
- :lexical-uuid ComparatorType/LEXICALUUIDTYPE
- :long ComparatorType/LONGTYPE
- :time-uuid ComparatorType/TIMEUUIDTYPE
- :utf-8 ComparatorType/UTF8TYPE})
-
-(def validator-types {:ascii "AsciiType"
- :bytes "BytesType"
- :counter "CounterColumnType"
- :integer "IntegerType"
- :lexical-uuid "LeixcalUUIDType"
- :long "LongType"
- :utf-8 "UTF8Type"})
+(def component-equality-type {:less_than_equal AbstractComposite$ComponentEquality/LESS_THAN_EQUAL
+ :equal AbstractComposite$ComponentEquality/EQUAL
+ :greater_than_equal AbstractComposite$ComponentEquality/GREATER_THAN_EQUAL})
+
+(def comparator-types {:ascii ComparatorType/ASCIITYPE
+ :bytes ComparatorType/BYTESTYPE
+ :integer ComparatorType/INTEGERTYPE
+ :lexical-uuid ComparatorType/LEXICALUUIDTYPE
+ :local-partitioner ComparatorType/LOCALBYPARTITIONERTYPE
+ :long ComparatorType/LONGTYPE
+ :time-uuid ComparatorType/TIMEUUIDTYPE
+ :utf-8 ComparatorType/UTF8TYPE
+ :composite ComparatorType/COMPOSITETYPE
+ :dynamic-composite ComparatorType/DYNAMICCOMPOSITETYPE
+ :uuid ComparatorType/UUIDTYPE
+ :counter ComparatorType/COUNTERTYPE})
+
+(def validator-types {:ascii "AsciiType"
+ :bytes "BytesType"
+ :integer "IntegerType"
+ :lexical-uuid "LeixcalUUIDType"
+ :local-partitioner "LocalByPartionerType"
+ :long "LongType"
+ :time-uuid "TimeUUIDType"
+ :utf-8 "UTF8Type"
+ :composite "CompositeType"
+ :dynamic-composite "DynamicCompositeTYpe"
+ :uuid "UUIDType"
+ :counter "CounterColumnType"})
(defn- make-column-family
"Returns an object defining a new column family"
@@ -42,14 +57,24 @@
(defn- make-keyspace-definition
([keyspace strategy-class replication-factor column-families]
- (let [column-families (map (fn [{:keys [name comparator type validator]}]
+ (let [column-families (map (fn [{:keys [name comparator type validator comparator-alias]}]
(let [cf-def (if (nil? comparator)
(make-column-family keyspace name)
(make-column-family keyspace name comparator))]
+
(doto ^ThriftCfDef cf-def
(.setColumnType (column-type type))
- (.setDefaultValidationClass (default-validation-class validator)))))
+ (.setDefaultValidationClass (default-validation-class validator)))
+
+ (if comparator
+ (.setComparatorType cf-def (comparator-types comparator)))
+
+ (if comparator-alias
+ (.setComparatorTypeAlias cf-def comparator-alias))
+
+ cf-def))
column-families)]
+
(HFactory/createKeyspaceDefinition keyspace
strategy-class
replication-factor
@@ -101,14 +126,18 @@
:super
:standard))
-(def types {"org.apache.cassandra.db.marshal.UTF8Type" :utf-8
- "org.apache.cassandra.db.marshal.AsciiType" :ascii
- "org.apache.cassandra.db.marshal.BytesType" :bytes
- "org.apache.cassandra.db.marshal.IntegerType" :integer
- "org.apache.cassandra.db.marshal.LexicalUUIDType" :lexical-uuid
- "org.apache.cassandra.db.marshal.LongType" :long
- "org.apache.cassandra.db.marshal.TimeUUIDType" :time-uuid
- "org.apache.cassandra.db.marshal.CounterColumnType" :counter})
+(def types {"org.apache.cassandra.db.marshal.AsciiType" :ascii
+ "org.apache.cassandra.db.marshal.BytesType" :bytes
+ "org.apache.cassandra.db.marshal.IntegerType" :integer
+ "org.apache.cassandra.db.marshal.LexicalUUIDType" :lexical-uuid
+ "org.apache.cassandra.db.marshal.LocalByPartionerType" :local-partitioner
+ "org.apache.cassandra.db.marshal.LongType" :long
+ "org.apache.cassandra.db.marshal.TimeUUIDType" :time-uuid
+ "org.apache.cassandra.db.marshal.UTF8Type" :utf-8
+ "org.apache.cassandra.db.marshal.CompositeType" :composite
+ "org.apache.cassandra.db.marshal.DynamicCompositeType" :dynamic-composite
+ "org.apache.cassandra.db.marshal.UUIDType" :uuid
+ "org.apache.cassandra.db.marshal.CounterColumnType" :counter})
(defn- parse-comparator
[^ComparatorType comparator-type]
View
174 src/clj_hector/serialize.clj
@@ -1,80 +1,13 @@
-(ns ^{:author "Paul Ingles"
- :description "Utilities for serializing and deserializing Clojure and Hector types."}
- clj-hector.serialize
- (:import [me.prettyprint.cassandra.serializers StringSerializer IntegerSerializer LongSerializer TypeInferringSerializer BytesArraySerializer SerializerTypeInferer UUIDSerializer BigIntegerSerializer BooleanSerializer DateSerializer ObjectSerializer AsciiSerializer ByteBufferSerializer FloatSerializer CharSerializer DoubleSerializer ShortSerializer]
+(ns clj-hector.serialize
+ ^{:author "Paul Ingles"
+ :description "Utilities for serializing and deserializing Clojure and Hector types."}
+ (:import [me.prettyprint.cassandra.serializers StringSerializer IntegerSerializer LongSerializer TypeInferringSerializer BytesArraySerializer SerializerTypeInferer UUIDSerializer BigIntegerSerializer BooleanSerializer DateSerializer ObjectSerializer AsciiSerializer ByteBufferSerializer FloatSerializer CharSerializer DoubleSerializer ShortSerializer CompositeSerializer DynamicCompositeSerializer]
[me.prettyprint.cassandra.model QueryResultImpl HColumnImpl ColumnSliceImpl RowImpl RowsImpl SuperRowImpl SuperRowsImpl HSuperColumnImpl CounterSliceImpl HCounterColumnImpl CounterSuperSliceImpl HCounterSuperColumnImpl CounterRowsImpl CounterRowImpl]
[me.prettyprint.hector.api.ddl KeyspaceDefinition ColumnFamilyDefinition ColumnDefinition]
+ [me.prettyprint.hector.api.beans AbstractComposite AbstractComposite$Component]
[me.prettyprint.hector.api Serializer]
[java.nio ByteBuffer]))
-(defprotocol ToClojure
- (to-clojure [_] "Convert hector types to Clojure data structures."))
-
-(extend-protocol ToClojure
- ColumnDefinition
- (to-clojure [c] {:name (.getName c)
- :index (.getIndexName c)
- :index-type (.getIndexType c)
- :validation-class (.getValidationClass c)})
-
- ColumnFamilyDefinition
- (to-clojure [c] {:name (.getName c)
- :comment (.getComment c)
- :column-type (.getColumnType c)
- :comparator-type (.getComparatorType c)
- :sub-comparator-type (.getSubComparatorType c)
- :columns (map to-clojure (.getColumnMetadata c))})
-
- KeyspaceDefinition
- (to-clojure [k] {(.getName k) {:strategy (.getStrategyClass k)
- :replication (.getReplicationFactor k)
- :column-families (map to-clojure (.getCfDefs k))}})
- CounterRowsImpl
- (to-clojure [s] (into {} (map to-clojure (iterator-seq (.iterator s)))))
- CounterRowImpl
- (to-clojure [s] {(.getKey s) (to-clojure (.getColumnSlice s))})
-
- SuperRowsImpl
- (to-clojure [s]
- (map to-clojure (iterator-seq (.iterator s))))
- SuperRowImpl
- (to-clojure [s]
- {(.getKey s) (map to-clojure (seq (.. s getSuperSlice getSuperColumns)))})
- HSuperColumnImpl
- (to-clojure [s]
- {(.getName s) (into {} (map to-clojure (.getColumns s)))})
- RowsImpl
- (to-clojure [s]
- (map to-clojure (iterator-seq (.iterator s))))
- RowImpl
- (to-clojure [s]
- {(.getKey s) (to-clojure (.getColumnSlice s))})
- ColumnSliceImpl
- (to-clojure [s]
- (into (hash-map) (for [c (.getColumns s)] (to-clojure c))))
- HColumnImpl
- (to-clojure [s]
- {(.getName s) (.getValue s)})
- HCounterColumnImpl
- (to-clojure [s]
- {(.getName s) (.getValue s)})
- CounterSuperSliceImpl
- (to-clojure [s]
- (into {} (map to-clojure (.getSuperColumns s))))
- HCounterSuperColumnImpl
- (to-clojure [s]
- {(.getName s) (into {} (map to-clojure (.getColumns s)))})
- CounterSliceImpl
- (to-clojure [s]
- (into {} (map to-clojure (.getColumns s))))
- Integer
- (to-clojure [s]
- {:count s})
- QueryResultImpl
- (to-clojure [s]
- (with-meta (to-clojure (.get s)) {:exec_us (.getExecutionTimeMicro s)
- :host (.getHostUsed s)})))
-
(def serializers {:integer (IntegerSerializer/get)
:string (StringSerializer/get)
:long (LongSerializer/get)
@@ -90,6 +23,8 @@
:double (DoubleSerializer/get)
:float (FloatSerializer/get)
:short (ShortSerializer/get)
+ :dynamic-composite (new DynamicCompositeSerializer)
+ :composite (new CompositeSerializer)
:type-inferring (TypeInferringSerializer/get)})
(defn serializer
@@ -110,3 +45,98 @@
(cond (keyword? x) (x serializers)
(instance? Serializer x) x
:else (SerializerTypeInferer/getSerializer x)))
+
+(defn- deserialize-composite
+ "Given a composite and a list of deserializers deserialize the
+ component values of the composite returning a vector of values.
+ Unfortunatly this is required due to a limition of the Composite
+ implementation in Hector. this isn't neccessary for
+ DynamicComposite."
+
+ [composite deserializers]
+ (into []
+ (map (fn [component deserializer]
+ (.fromByteBuffer (serializer deserializer) (.getBytes component)))
+ (.getComponents composite)
+ deserializers)))
+
+
+(defprotocol ToClojure
+ (to-clojure [_ _] "Convert hector types to Clojure data structures."))
+
+(defn- tc-swap
+ [opts s]
+ (to-clojure s opts))
+
+(extend-protocol ToClojure
+ ColumnDefinition
+ (to-clojure [c _] {:name (.getName c)
+ :index (.getIndexName c)
+ :index-type (.getIndexType c)
+ :validation-class (.getValidationClass c)})
+
+ ColumnFamilyDefinition
+ (to-clojure [c opts] {:name (.getName c)
+ :comment (.getComment c)
+ :column-type (.getColumnType c)
+ :comparator-type (.getComparatorType c)
+ :sub-comparator-type (.getSubComparatorType c)
+ :columns (map (partial tc-swap opts) (.getColumnMetadata c))})
+
+ KeyspaceDefinition
+ (to-clojure [k opts] {(.getName k) {:strategy (.getStrategyClass k)
+ :replication (.getReplicationFactor k)
+ :column-families (map (partial tc-swap opts) (.getCfDefs k))}})
+ CounterRowsImpl
+ (to-clojure [s opts]
+ (into {} (partial tc-swap opts) (iterator-seq (.iterator s))))
+ CounterRowImpl
+ (to-clojure [s opts]
+ {(.getKey s) (to-clojure (.getColumnSlice s) opts)})
+ SuperRowsImpl
+ (to-clojure [s opts]
+ (map (partial tc-swap opts) (iterator-seq (.iterator s))))
+ SuperRowImpl
+ (to-clojure [s opts]
+ {(.getKey s) (map (partial tc-swap opts) (seq (.. s getSuperSlice getSuperColumns)))})
+ HSuperColumnImpl
+ (to-clojure [s opts]
+ {(.getName s) (into {} (map (partial tc-swap opts) (.getColumns s)))})
+ RowsImpl
+ (to-clojure [s opts]
+ (map (partial tc-swap opts) (iterator-seq (.iterator s))))
+ RowImpl
+ (to-clojure [s opts]
+ {(.getKey s) (to-clojure (.getColumnSlice s) opts)})
+ ColumnSliceImpl
+ (to-clojure [s opts]
+ (into (hash-map) (for [c (.getColumns s)] (to-clojure c opts))))
+ HColumnImpl
+ (to-clojure [s opts]
+ {(let [col (.getName s)] (if (instance? AbstractComposite col)
+ (to-clojure col opts) col)) (.getValue s)})
+ HCounterColumnImpl
+ (to-clojure [s opts]
+ {(.getName s) (.getValue s)})
+ CounterSuperSliceImpl
+ (to-clojure [s opts]
+ (into {} (map (partial tc-swap opts) (.getSuperColumns s))))
+ HCounterSuperColumnImpl
+ (to-clojure [s opts]
+ {(.getName s) (into {} (map (partial tc-swap opts) (.getColumns s)))})
+ CounterSliceImpl
+ (to-clojure [s opts]
+ (into {} (map (partial tc-swap opts) (.getColumns s))))
+ Integer
+ (to-clojure [s _]
+ {:count s})
+ AbstractComposite
+ (to-clojure [s opts]
+ (let [serializers (:c-serializer opts)]
+ (if serializers
+ (deserialize-composite s serializers)
+ (into [] (map #(.getValue %1) (.getComponents s))))))
+ QueryResultImpl
+ (to-clojure [s opts]
+ (with-meta (to-clojure (.get s) opts) {:exec_us (.getExecutionTimeMicro s)
+ :host (.getHostUsed s)})))
View
67 test/clj_hector/test/core.clj
@@ -1,12 +1,53 @@
(ns clj-hector.test.core
(:import [me.prettyprint.cassandra.serializers StringSerializer])
+ (:require [clj-hector.serialize :as s])
(:use [clojure.test]
+ [clj-hector.test.test-helper]
[clj-hector.test.cassandra-helper :only (with-test-keyspace)]
[clj-hector.core] :reload))
+(deftest composite-serializer
+ (let [column-family "A"]
+ (with-test-keyspace keyspace [{:name column-family
+ :comparator :composite
+ :comparator-alias "(UTF8Type, UTF8Type)"}]
+ (testing ":composite serializer"
+ (let [opts [:v-serializer :string
+ :n-serializer :composite
+ :c-serializer [:string :string]]
+ comp (create-composite {:value "col"
+ :n-serializer :string
+ :comparator :utf-8}
+ {:value "name"
+ :n-serializer :string
+ :comparator :utf-8})]
+
+ (put keyspace column-family "row-key" {comp "v"} :n-serializer :composite)
+ (is (= [{"row-key" {["col" "name"] "v"}}]
+ (apply get-rows keyspace column-family ["row-key"] opts))
+ (= {["col" "name"] "v"}
+ (apply get-columns keyspace column-family "row-key" [comp] opts))))))))
+
(deftest serializers
(let [column-family "A"]
(with-test-keyspace keyspace [{:name column-family}]
+ (testing ":dynamic-composite serializer"
+ (let [opts [:v-serializer :string
+ :n-serializer :dynamic-composite]
+ comp (create-dynamic-composite {:value "col"
+ :n-serializer :ascii
+ :comparator :ascii}
+ {:value "name"
+ :n-serializer :ascii
+ :comparator :ascii})]
+
+ (put keyspace column-family "row-key" {comp "v"} :n-serializer :dynamic-composite)
+ (is (= [{"row-key" {["col" "name"] "v"}}]
+ (apply get-rows keyspace column-family ["row-key"] opts))
+ (= {["col" "name"] "v"}
+ (apply get-columns keyspace column-family "row-key" [comp] opts))))))
+
+ (with-test-keyspace keyspace [{:name column-family}]
(testing ":string serializer"
(let [opts [:v-serializer :string
:n-serializer :string]]
@@ -65,6 +106,32 @@
"c" "v"}
(apply get-column-range keyspace column-family "row-key" "a" "c" opts))))))
+(deftest dynamic-composite-column-ranges
+ (with-test-keyspace keyspace [{:name "A"}]
+ (let [column-family "A"
+ opts [:v-serializer :string
+ :n-serializer :dynamic-composite]]
+
+ (let [cols (into {}
+ (interleave
+ (doseq [i (range 0 3)]
+ (create-dynamic-composite {:value "col"
+ :n-serializer :string
+ :comparator :utf-8}
+ {:value i
+ :n-serializer :integer
+ :comparator :integer}))
+ (cycle "v")))]
+ (put keyspace column-family "row-key" cols)
+ (is (= cols
+ (apply get-column-range
+ keyspace
+ column-family
+ "row-key"
+ (first (keys cols))
+ (last (keys cols))
+ opts)))))))
+
;; count-columns is different to counter columns, it's not
;; an O(1) operation.
(deftest counting-with-count-columns
View
11 test/clj_hector/test/test_helper.clj
@@ -0,0 +1,11 @@
+(ns clj-hector.test.test-helper
+ (:use [clojure.test]))
+
+;; A Simple macro that enable to mark your test
+;; to pending
+;; Taken from: http://techbehindtech.com/2010/06/01/marking-tests-as-pending-in-clojure/
+(defmacro deftest-pending
+ [name & body]
+ (let [message (str "\n========\n" name " is pending !!\n========\n")]
+ `(deftest ~name
+ (println ~message))))
Something went wrong with that request. Please try again.