Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Unanalyzed Fields #34

Closed
wants to merge 10 commits into from

2 participants

@adamalix

This pull request introduces Unanalyzed fields that have the ability to be used as filter queries that are optionally cached in ElasticSearch.

adamalix and others added some commits
@adamalix adamalix Initial stab at Term queries and filter caching:
This commit adds the ability to have unanalyzed fields and to also cache filter
queries to speed up queries on Elasticsearch.
TODOs:
- Debug commented out line in testUnanalyzed Unit test (ElasticQueryTest:308)
  `"terms" : { "field" : [ [ "value1, "value2" ] ] } ` queries don't work
  because of a SearchParseException.
- Make SlashemGeoField extend SlashemUnanalyzedStringField
- Decide what to do for Solr queries on this field type (in the case of
  SlashemGeoField).  Do we want two separate field types one for both backends?
  The current plan is to not implement Term queries for Solr because we receive
  no benefit there.
3fb6928
@holdenk holdenk Add test for empty nin/in queries. 003122a
@holdenk holdenk Bump version & update some documentation c9dd050
@holdenk holdenk Fix query gen for elasitcboost weights of 1.0 5e9d133
@holdenk holdenk Fix query gen for ES Empty[T] queries.
TODO: We should optimize our tree before generating queries
4888ca3
@holdenk holdenk Bump version for bugfix 9f72c4f
@adamalix adamalix Fixed Terms queries and filters. 47cb673
@adamalix adamalix Tests for Term filter queries. 77e3ab9
@adamalix adamalix Changed SlashemGeoField to be unanalyzed:
- SlashemGeoField now extends SlashemUnanalyzedStringField
- Fixed extend() for Term[T] queries because of failing unit tests
8978eaf
@holdenk holdenk commented on the diff
src/main/scala/com/foursquare/slashem/Schema.scala
@@ -782,28 +788,54 @@ trait SolrSchema[M <: Record[M]] extends SlashemSchema[M] {
}
+/**
+ * A field type for unanalyzed queries. Results in using Term[V] queries.
@holdenk
holdenk added a note

Maybe add available for ES only.

Then do we want separate SlashemGeoFields for Solr / ES? This should behave the same for ES / Solr with the current impl.

@holdenk
holdenk added a note

It does, you're not seeing the latest because of the way this thing does reviews.

@holdenk
holdenk added a note

Fuck github. Sorry for the dumb comment :p

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...t/scala/com/foursquare/slashem/ElasticQueryTest.scala
@@ -328,6 +346,21 @@ class ElasticQueryTest extends SpecsMatchers with ScalaCheckMatchers {
Assert.assertEquals(ids2.intersect(idsWithFavNums).length, 3)
}
+ @Test
+ def testTermQueries {
@holdenk
holdenk added a note

Add a test for in with empty list, also add a test for nin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/main/scala/com/foursquare/slashem/Ast.scala
@@ -405,6 +408,47 @@ object Ast {
}
}
+ /**
+ * A term query. Used for queries that don't need to be analyzed
+ *
+ * By default, elasticFilter() will always be cached!
+ */
+ case class Term[T](query: Iterable[T], escaped: Boolean = true, cached: Boolean = true) extends Query[T] {
+ // hack for single term queries
+ def this(query: T) = this(List(query))
+ /** @inheritdoc */
+ //def extend() = throw new UnimplementedException("Slashem does not support Term queries Solr")
+ def extend(): String = {
+ escaped match {
+ // hack to fix wrapping the queries in a List()
+ case true => {'"' + escape(query.mkString("")) + '"'}
+ case false => '"' + query.toString + '"'
@holdenk
holdenk added a note

Don't you want to do query.mkString("") here to?
Also add a test case for single element Term[T] query against Solr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@holdenk holdenk commented on the diff
src/main/scala/com/foursquare/slashem/Ast.scala
((14 lines not shown))
+ def extend(): String = {
+ escaped match {
+ // hack to fix wrapping the queries in a List()
+ case true => {'"' + escape(query.mkString("")) + '"'}
+ case false => '"' + query.toString + '"'
+ }
+ }
+ /** @inheritdoc */
+ def elasticExtend(qf: List[WeightedField], pf: List[PhraseWeightedField], mm: Option[String]): ElasticQueryBuilder = {
+ val fieldName = qf.head.fieldName
+ val weight = qf.head.weight.toFloat
+ query match {
+ case term::Nil => EQueryBuilders.termQuery(fieldName, term).boost(weight)
+ case terms => {
+ val moarTerms = terms.toSeq.map(_.toString)
+ EQueryBuilders.termsQuery(fieldName, moarTerms: _*).boost(weight)
@holdenk
holdenk added a note

Is this going to work with an empty terms? Or do we need to special case it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@holdenk holdenk commented on the diff
src/main/scala/com/foursquare/slashem/Ast.scala
((25 lines not shown))
+ query match {
+ case term::Nil => EQueryBuilders.termQuery(fieldName, term).boost(weight)
+ case terms => {
+ val moarTerms = terms.toSeq.map(_.toString)
+ EQueryBuilders.termsQuery(fieldName, moarTerms: _*).boost(weight)
+ }
+ }
+ }
+ /** @inheritdoc */
+ override def elasticFilter(qf: List[WeightedField]): ElasticFilterBuilder = {
+ val fieldName = qf.head.fieldName
+ query match {
+ case term::Nil => EFilterBuilders.termFilter(fieldName, term).cache(cached)
+ case terms => {
+ val moarTerms = terms.toSeq.map(_.toString)
+ EFilterBuilders.termsFilter(fieldName, moarTerms: _*).cache(cached)
@holdenk
holdenk added a note

Is this going to work with an empty terms? Or do we need to special case it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/main/scala/com/foursquare/slashem/Schema.scala
@@ -843,6 +875,14 @@ trait SlashemField[V, M <: Record[M]] extends OwnedField[M] {
//Slashem field types
class SlashemStringField[T <: Record[T]](owner: T) extends StringField[T](owner, 0) with SlashemField[String, T]
+/**
+ * Field type that can be queried without analyzing whitespace.
@holdenk
holdenk added a note

I'd recommend "Field type that be queried without analyzing. An example of this would be a multi-value field or a whitespace tokenized field where search terms are always for a specific token". What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@adamalix adamalix Added changes from Pull Request 34 feedback:
- Added Solr tests for Terms queries
- Added more comprehensive docstrings
- More comprehensive Elastic Tests
- Fixed Solr Terms queries
dd61da6
@adamalix

Deleted branch because it got corrupt. Opening new request.

@adamalix adamalix closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 10, 2012
  1. @adamalix

    Initial stab at Term queries and filter caching:

    adamalix authored
    This commit adds the ability to have unanalyzed fields and to also cache filter
    queries to speed up queries on Elasticsearch.
    TODOs:
    - Debug commented out line in testUnanalyzed Unit test (ElasticQueryTest:308)
      `"terms" : { "field" : [ [ "value1, "value2" ] ] } ` queries don't work
      because of a SearchParseException.
    - Make SlashemGeoField extend SlashemUnanalyzedStringField
    - Decide what to do for Solr queries on this field type (in the case of
      SlashemGeoField).  Do we want two separate field types one for both backends?
      The current plan is to not implement Term queries for Solr because we receive
      no benefit there.
  2. @holdenk @adamalix

    Add test for empty nin/in queries.

    holdenk authored adamalix committed
  3. @holdenk @adamalix

    Bump version & update some documentation

    holdenk authored adamalix committed
  4. @holdenk @adamalix

    Fix query gen for elasitcboost weights of 1.0

    holdenk authored adamalix committed
  5. @holdenk @adamalix

    Fix query gen for ES Empty[T] queries.

    holdenk authored adamalix committed
    TODO: We should optimize our tree before generating queries
  6. @holdenk @adamalix

    Bump version for bugfix

    holdenk authored adamalix committed
  7. @adamalix
  8. @adamalix
  9. @adamalix

    Changed SlashemGeoField to be unanalyzed:

    adamalix authored
    - SlashemGeoField now extends SlashemUnanalyzedStringField
    - Fixed extend() for Term[T] queries because of failing unit tests
Commits on May 14, 2012
  1. @adamalix

    Added changes from Pull Request 34 feedback:

    adamalix authored
    - Added Solr tests for Terms queries
    - Added more comprehensive docstrings
    - More comprehensive Elastic Tests
    - Fixed Solr Terms queries
This page is out of date. Refresh to see the latest.
View
17 README.md
@@ -24,6 +24,11 @@ The other hook is only useful if you are using Solr for geospatail information,
we provide a trait called SolrGeoHash which has two required functions, namely
coverString and rectCoverString. Most people will not need to implement this.
+### Multiple Solr cores & non-stanrd query paths
+
+Support for multiple cores is done by overriding "core" in the model (wich is an Option[String]).
+If you have a non-standard query path you can override queryPath in your model.
+
## Examples
[QueryTest.scala](https://github.com/foursquare/slashem/blob/master/src/test/scala/com/foursquare/slashem/QueryTest.scala) contains sample queries and shows the corresponding query.
@@ -48,11 +53,17 @@ to do this:
## Dependencies
lift, joda-time, junit, finagle, jackson. These dependencies are managed by
-the build system.
+the build system. Note: some of the transitive dependencies may fail to resolve
+from the central maven. If you are using sbt you can fix this by adding
-## Warnings
+ ivyXML := (
+ <dependencies>
+ <exclude module="jmxtools"/>
+ <exclude module="jmxri"/>
+ </dependencies>
+ )
-This is still a very early version. There are likely bugs (sorry!). Let us know
+## Warnings still a very early version. There are likely bugs (sorry!). Let us know
if you find any. While we can't promise timely fixes, it will help :)
## Maintainers
View
9 build.sbt
@@ -1,6 +1,6 @@
name := "slashem"
-version := "0.9.13"
+version := "0.9.15a"
organization := "com.foursquare"
@@ -123,4 +123,11 @@ pomExtra := (
<email>aalix@foursquare.com</email>
</developer>
</developers>
+)
+
+ivyXML := (
+<dependencies>
+ <exclude module="jmxtools"/>
+ <exclude module="jmxri"/>
+</dependencies>
)
View
58 src/main/scala/com/foursquare/slashem/Ast.scala
@@ -7,6 +7,7 @@ import org.elasticsearch.index.query.{FilterBuilder => ElasticFilterBuilder,
QueryBuilder => ElasticQueryBuilder,
QueryBuilders => EQueryBuilders,
QueryStringQueryBuilder}
+import scalaj.collection.Imports._
/**
* Abstract Syntax Tree used to represent queries.
@@ -242,7 +243,7 @@ object Ast {
}
def elasticBoost(): Pair[List[String],String] = {
weight match {
- case 1.0 => Pair(Nil,"(doc['" + fieldName + "'].value)")
+ case 1.0 => Pair(Nil,"doc['" + fieldName + "'].value")
case _ => Pair(Nil,"(doc['" + fieldName + "'].value *" + weight.toString + ")")
}
}
@@ -359,8 +360,10 @@ object Ast {
def extend(): String = "\"\""
/** @inheritdoc */
def elasticExtend(qf: List[WeightedField], pf: List[PhraseWeightedField], mm: Option[String]): ElasticQueryBuilder = {
- val q = EQueryBuilders.queryString(this.extend())
- qf.map(f => q.field(f.fieldName,f.weight.toFloat))
+ //An empty query matches no documents, so it is the same as the negation of matchAll
+ //Note: this is kind of ugly since this is may likely an OR clause or negated up above
+ //so we should try and avoid generating this
+ val q = EQueryBuilders.boolQuery.mustNot(EQueryBuilders.matchAllQuery())
q
}
}
@@ -405,6 +408,51 @@ object Ast {
}
}
+ /**
+ * A term query. Used for queries that don't need to be analyzed
+ *
+ * By default, elasticFilter() will always be cached!
+ */
+ case class Term[T](query: Iterable[T], escaped: Boolean = true, cached: Boolean = true) extends Query[T] {
+ // hack for single term queries
+ def this(query: T) = this(List(query))
+ /** @inheritdoc */
+ //def extend() = throw new UnimplementedException("Slashem does not support Term queries Solr")
+ def extend(): String = {
+ escaped match {
+ // hack to fix wrapping the queries in a List()
+ case true => {
+ val queries = query.map(q => {'"' + escape(q.toString) + '"'})
+ queries.mkString(" OR ")
+ }
+// case true => {'"' + query.mkString("\" OR \"")
+ case false => '"' + query.mkString(" OR ") + '"'
+ }
+ }
+ /** @inheritdoc */
+ def elasticExtend(qf: List[WeightedField], pf: List[PhraseWeightedField], mm: Option[String]): ElasticQueryBuilder = {
+ val fieldName = qf.head.fieldName
+ val weight = qf.head.weight.toFloat
+ query match {
+ case term::Nil => EQueryBuilders.termQuery(fieldName, term).boost(weight)
+ case terms => {
+ val moarTerms = terms.toSeq.map(_.toString)
+ EQueryBuilders.termsQuery(fieldName, moarTerms: _*).boost(weight)
@holdenk
holdenk added a note

Is this going to work with an empty terms? Or do we need to special case it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ }
+ }
+ }
+ /** @inheritdoc */
+ override def elasticFilter(qf: List[WeightedField]): ElasticFilterBuilder = {
+ val fieldName = qf.head.fieldName
+ query match {
+ case term::Nil => EFilterBuilders.termFilter(fieldName, term).cache(cached)
+ case terms => {
+ val moarTerms = terms.toSeq.map(_.toString)
+ EFilterBuilders.termsFilter(fieldName, moarTerms: _*).cache(cached)
@holdenk
holdenk added a note

Is this going to work with an empty terms? Or do we need to special case it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ }
+ }
+ }
+ }
case class Range[T](q1: Query[T],q2: Query[T]) extends Query[T] {
/** @inheritdoc */
@@ -485,7 +533,7 @@ object Ast {
}
/**
- * Class representing clauses ANDed together
+ * Class representing queries ANDed together
*/
case class And[T](queries: Query[T]*) extends Query[T] {
/** @inheritdoc */
@@ -505,7 +553,7 @@ object Ast {
}
}
/**
- * Case class representing a list of clauses ORed together
+ * Case class representing a list of queries ORed together
*/
case class Or[T](queries: Query[T]*) extends Query[T] {
/** @inheritdoc */
View
67 src/main/scala/com/foursquare/slashem/Schema.scala
@@ -55,6 +55,12 @@ case class SolrResponseException(code: Int, reason: String, solrName: String, qu
}
}
+case class UnimplementedException(reason: String) extends RuntimeException {
+ override def getMessage(): String = {
+ "Not implemented: %s".format(reason)
+ }
+}
+
/** The response header. There are normally more fields in the response header we could extract, but
* we don't at present. */
case class ResponseHeader @JsonCreator()(@JsonProperty("status")status: Int, @JsonProperty("QTime")QTime: Int)
@@ -433,8 +439,8 @@ trait SolrGeoHash {
}
//Default geohash, does nothing.
object NoopSolrGeoHash extends SolrGeoHash {
- def coverString (geoLat: Double, geoLong: Double, radiusInMeters: Int, maxCells: Int ): Seq[String] = List("pleaseUseaRealGeoHash")
- def rectCoverString(topRight: (Double, Double), bottomLeft: (Double, Double), maxCells: Int = 0, minLevel: Int = 0, maxLevel: Int = 0): Seq[String] = List("pleaseUseaRealGeoHash")
+ def coverString (geoLat: Double, geoLong: Double, radiusInMeters: Int, maxCells: Int ): Seq[String] = List("pleaseUseaRealGeoHash", "thisIsForFunctionalityTests")
+ def rectCoverString(topRight: (Double, Double), bottomLeft: (Double, Double), maxCells: Int = 0, minLevel: Int = 0, maxLevel: Int = 0): Seq[String] = List("pleaseUseaRealGeoHash", "thisIsForFunctionalityTests")
}
trait SlashemSchema[M <: Record[M]] extends Record[M] {
@@ -782,28 +788,54 @@ trait SolrSchema[M <: Record[M]] extends SlashemSchema[M] {
}
+/**
+ * A field type for unanalyzed queries. Results in using Term[V] queries.
@holdenk
holdenk added a note

Maybe add available for ES only.

Then do we want separate SlashemGeoFields for Solr / ES? This should behave the same for ES / Solr with the current impl.

@holdenk
holdenk added a note

It does, you're not seeing the latest because of the way this thing does reviews.

@holdenk
holdenk added a note

Fuck github. Sorry for the dumb comment :p

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ */
+trait SlashemUnanalyzedField[V, M <: Record[M]] extends SlashemField[V, M] {
+ self: Field[V, M] =>
+
+ override val unanalyzed = true
+}
trait SlashemField[V, M <: Record[M]] extends OwnedField[M] {
self: Field[V, M] =>
import Helpers._
- //Note eqs and neqs results in phrase queries!
- def eqs(v: V) = Clause[V](self.queryName, Group(Phrase(v)))
- def neqs(v: V) = Clause[V](self.queryName, Phrase(v),false)
+ // Override this value to produce unanalyzed queries!
+ val unanalyzed = false
+
+ def produceQuery(v: V): Query[V] = {
+ unanalyzed match {
+ // use new to use Term's additional non-default constructor
+ case true => new Term(v)
+ case false => Phrase(v)
+ }
+ }
+
+ def produceGroupedQuery(v: Iterable[V]): Query[V] = {
+ unanalyzed match {
+ // we don't want to groupWithOr and instead take advantage of "terms" queries
+ case true => Term(v)
+ case false => groupWithOr(v.map({x: V => produceQuery(x)}))
+ }
+ }
+
+ def eqs(v: V) = Clause[V](self.queryName, Group(produceQuery(v)))
+ def neqs(v: V) = Clause[V](self.queryName, produceQuery(v),false)
//With a boost
- def eqs(v: V, b: Float) = Clause[V](self.queryName, Boost(Group(Phrase(v)),b))
- def neqs(v: V, b:Float) = Clause[V](self.queryName, Boost(Phrase(v),b),false)
+ def eqs(v: V, b: Float) = Clause[V](self.queryName, Boost(Group(produceQuery(v)),b))
+ def neqs(v: V, b:Float) = Clause[V](self.queryName, Boost(produceQuery(v),b),false)
//This allows for bag of words style matching.
def contains(v: V) = Clause[V](self.queryName, Group(BagOfWords(v)))
def contains(v: V, b: Float) = Clause[V](self.queryName, Boost(Group(BagOfWords(v)),b))
- def in(v: Iterable[V]) = Clause[V](self.queryName, groupWithOr(v.map({x: V => Phrase(x)})))
- def nin(v: Iterable[V]) = Clause[V](self.queryName, groupWithOr(v.map({x: V => Phrase(x)})),false)
+ def in(v: Iterable[V]) = Clause[V](self.queryName, produceGroupedQuery(v))
+ def nin(v: Iterable[V]) = Clause[V](self.queryName, produceGroupedQuery(v),false)
- def in(v: Iterable[V], b: Float) = Clause[V](self.queryName, Boost(groupWithOr(v.map({x: V => Phrase(x)})),b))
- def nin(v: Iterable[V], b: Float) = Clause[V](self.queryName, Boost(groupWithOr(v.map({x: V => Phrase(x)})),b),false)
+ def in(v: Iterable[V], b: Float) = Clause[V](self.queryName, Boost(produceGroupedQuery(v),b))
+ def nin(v: Iterable[V], b: Float) = Clause[V](self.queryName, Boost(produceGroupedQuery(v),b),false)
def inRange(v1: V, v2: V) = Clause[V](self.queryName, Group(Range(BagOfWords(v1),BagOfWords(v2))))
def ninRange(v1: V, v2: V) = Clause[V](self.queryName, Group(Range(BagOfWords(v1),BagOfWords(v2))),false)
@@ -843,6 +875,17 @@ trait SlashemField[V, M <: Record[M]] extends OwnedField[M] {
//Slashem field types
class SlashemStringField[T <: Record[T]](owner: T) extends StringField[T](owner, 0) with SlashemField[String, T]
+/**
+ * Field type that can be queried without analyzing.
+ *
+ * Ex: multi-value field or a whitespace tokenized field where
+ * search terms are always for a specific token.
+ *
+ * @see SlashemStringField
+ */
+class SlashemUnanalyzedStringField[T <: Record[T]](owner: T)
+ extends StringField[T](owner, 0) with SlashemUnanalyzedField[String, T]
+
//Allows for querying against the default filed in solr. This field doesn't have a name
class SlashemDefaultStringField[T <: Record[T]](owner: T) extends StringField[T](owner, 0) with SlashemField[String, T] {
override def name = ""
@@ -951,7 +994,7 @@ class SlashemPointField[T <: Record[T]](owner: T) extends PointField[T](owner) w
class SlashemBooleanField[T <: Record[T]](owner: T) extends BooleanField[T](owner) with SlashemField[Boolean, T]
class SlashemDateTimeField[T <: Record[T]](owner: T) extends JodaDateTimeField[T](owner) with SlashemField[DateTime, T]
//More restrictive type so we can access the geohash
-class SlashemGeoField[T <: SlashemSchema[T]](owner: T) extends StringField[T](owner,0) with SlashemField[String, T] {
+class SlashemGeoField[T <: SlashemSchema[T]](owner: T) extends SlashemUnanalyzedStringField[T](owner) {
def inRadius(geoLat: Double, geoLong: Double, radiusInMeters: Int, maxCells: Int = owner.geohash.maxCells) = {
val cellIds = owner.geohash.coverString(geoLat, geoLong, radiusInMeters, maxCells = maxCells)
//If we have an empty cover we default to everything.
View
50 src/test/scala/com/foursquare/slashem/ElasticQueryTest.scala
@@ -79,6 +79,14 @@ class ElasticQueryTest extends SpecsMatchers with ScalaCheckMatchers {
}
@Test
+ def simpleBoostTest {
+ val fullQuery = ESimplePanda.where(_.name contains "lol")
+ .limit(5).boostField(_.followers)
+ val r = fullQuery fetch()
+ }
+
+
+ @Test
def testEmptySearch {
try {
val r = ESimplePanda where (_.name eqs "lolsdonotinsertsomethingwiththisinit") fetch()
@@ -305,10 +313,26 @@ class ElasticQueryTest extends SpecsMatchers with ScalaCheckMatchers {
def testListFieldIn {
val response1 = ESimplePanda where (_.favnums in List(2, 3, 4, 5)) fetch()
val response2 = ESimplePanda where (_.favnums in List(99)) fetch()
- //val response3 = ESimplePanda where (_.favnums in List()) fetch()
+ val response3 = ESimplePanda where (_.termsfield in List("termhit", "lol")) fetch()
Assert.assertEquals(response1.response.results.length, 2)
Assert.assertEquals(response2.response.results.length, 0)
- //Assert.assertEquals(response3.response.results.length, 0)
+ Assert.assertEquals(response3.response.results.length, 1)
+ }
+
+ @Test
+ def testIntListFieldEmptyIn {
+ val response1 = ESimplePanda where (_.favnums in List()) fetch()
+ val response2 = ESimplePanda where (_.termsfield in List()) fetch()
+ Assert.assertEquals(response1.response.results.length, 0)
+ Assert.assertEquals(response2.response.results.length, 0)
+ }
+
+ @Test
+ def testIntListFieldEmptyNin {
+ val response1 = ESimplePanda where (_.favnums nin List()) fetch()
+ val response2 = ESimplePanda where (_.termsfield nin List()) fetch()
+ Assert.assertEquals(response1.response.results.length, 8)
+ Assert.assertEquals(response2.response.results.length, 8)
}
@Test
@@ -326,6 +350,26 @@ class ElasticQueryTest extends SpecsMatchers with ScalaCheckMatchers {
val ids2 = response2.response.oids
// All three docs with favnums should be returned, none contain 99
Assert.assertEquals(ids2.intersect(idsWithFavNums).length, 3)
+
+ val response3 = ESimplePanda where (_.termsfield nin List("termhit")) fetch()
+ val ids3 = response3.response.oids
+ // All three docs with favnums should be returned, none contain 99
+ Assert.assertEquals(ids3.intersect(idsWithFavNums).length, 2)
+ }
+
+ @Test
+ def testTermQueries {
+ val res1 = ESimplePanda where (_.termsfield eqs "termhit") fetch()
+ val res2 = ESimplePanda where (_.termsfield in List("randomterm", "termhit")) fetch()
+ Assert.assertEquals(res1.response.results.length, 1)
+ Assert.assertEquals(res2.response.results.length, 1)
+ }
+
+ @Test
+ def testTermFilters {
+ // grab 2 results, filter to 1
+ val res1 = ESimplePanda where (_.hugenums contains 1L) filter(_.termsfield in List("termhit", "randomterm")) fetch()
+ Assert.assertEquals(res1.response.results.length, 1)
}
@Before
@@ -376,6 +420,7 @@ class ElasticQueryTest extends SpecsMatchers with ScalaCheckMatchers {
val favnums1 = List(1, 2, 3, 4, 5).asJava
val favnums2 = List(1, 2, 3, 4, 5).asJava
val favnums3 = List(6, 7, 8, 9, 10).asJava
+ val terms1 = List("termhit", "nohit").asJava
val nicknames1 = List("jerry", "dawg", "xzibit").asJava
val nicknames2 = List("xzibit", "alvin").asJava
val nicknames3 = List("alvin", "nathaniel", "joiner").asJava
@@ -389,6 +434,7 @@ class ElasticQueryTest extends SpecsMatchers with ScalaCheckMatchers {
.field("favnums", favnums1)
.field("nicknames", nicknames1)
.field("hugenums", hugenums1)
+ .field("termsfield", terms1)
.endObject()
).execute()
.actionGet();
View
1  src/test/scala/com/foursquare/slashem/ElasticTest.scala
@@ -18,6 +18,7 @@ class ESimplePanda extends ElasticSchema[ESimplePanda] {
object favnums extends SlashemIntListField(this)
object nicknames extends SlashemStringListField(this)
object hugenums extends SlashemLongListField(this)
+ object termsfield extends SlashemUnanalyzedStringField(this)
}
object ESimpleGeoPanda extends ESimpleGeoPanda with ElasticMeta[ESimpleGeoPanda] {
View
27 src/test/scala/com/foursquare/slashem/QueryTest.scala
@@ -70,6 +70,27 @@ class QueryTest extends SpecsMatchers with ScalaCheckMatchers {
}
@Test
+ def testProduceCorrectListfieldQueryStringWhenEmpty {
+ val q = SVenueTest where (_.commentList in List())
+ val qp = q.meta.queryParams(q).toList
+ Assert.assertEquals(qp.sortWith(_._1 > _._1),
+ List("q" -> "commentList:(\"\")",
+ "start" -> "0",
+ "rows" -> "10").sortWith(_._1 > _._1))
+ }
+ @Test
+ def testProduceCorrectListfieldQueryStringNinWhenEmpty {
+ val q = SVenueTest where (_.commentList nin List())
+ val qp = q.meta.queryParams(q).toList
+ Assert.assertEquals(qp.sortWith(_._1 > _._1),
+ List("q" -> "(*:* -commentList:(\"\"))",
+ "start" -> "0",
+ "rows" -> "10").sortWith(_._1 > _._1))
+ }
+
+
+
+ @Test
def testProduceCorrectSimpleQueryStringContains {
val q = SUserTest where (_.fullname contains "jon")
val qp = q.meta.queryParams(q).toList
@@ -541,7 +562,7 @@ class QueryTest extends SpecsMatchers with ScalaCheckMatchers {
"qf" -> "text",
"qf" -> "ngram_name^0.2",
"qf" -> "tags^0.01",
- "fq" -> "geo_s2_cell_ids:(\"pleaseUseaRealGeoHash\")",
+ "fq" -> "geo_s2_cell_ids:(\"pleaseUseaRealGeoHash\" OR \"thisIsForFunctionalityTests\")",
"tieBreaker" -> "0.2",
"fl" -> "id,name,userid,mayorid,category_id_0,popularity,decayedPopularity1,lat,lng,checkin_info,score,hasSpecial,address,crossstreet,city,state,zip,country,checkinCount,partitionedPopularity",
"bq" -> "name:(holden's hobohut)^10.0",
@@ -588,7 +609,7 @@ class QueryTest extends SpecsMatchers with ScalaCheckMatchers {
"qf" -> "text",
"qf" -> "ngram_name^0.2",
"qf" -> "tags^0.01",
- "fq" -> "geo_s2_cell_ids:(\"pleaseUseaRealGeoHash\")",
+ "fq" -> "geo_s2_cell_ids:(\"pleaseUseaRealGeoHash\" OR \"thisIsForFunctionalityTests\")",
"tieBreaker" -> "0.2",
"fl" -> "id,name,userid,mayorid,category_id_0,popularity,decayedPopularity1,lat,lng,checkin_info,score,hasSpecial,address,crossstreet,city,state,zip,country,checkinCount,partitionedPopularity",
"bq" -> "name:(holden's hobohut)^10.0",
@@ -609,7 +630,7 @@ class QueryTest extends SpecsMatchers with ScalaCheckMatchers {
"q" -> "(DJ Hixxy)",
"start" -> "0",
"rows" -> "10",
- "fq" -> "geo_s2_cell_ids:(\"pleaseUseaRealGeoHash\")")
+ "fq" -> "geo_s2_cell_ids:(\"pleaseUseaRealGeoHash\" OR \"thisIsForFunctionalityTests\")")
Assert.assertEquals(Nil, ((qp.toSet &~ expected.toSet)).toList)
Assert.assertEquals(Nil, (expected.toSet &~ qp.toSet).toList)
}
Something went wrong with that request. Please try again.