diff --git a/plugins/native-script-example/README.textile b/plugins/native-script-example/README.textile new file mode 100644 index 0000000000000..c119c46012b57 --- /dev/null +++ b/plugins/native-script-example/README.textile @@ -0,0 +1,83 @@ +h1. Example of Native Script Plugin for Elasticsearch + +h2. Introduction + +p. This plugin contains several examples of "native script":http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html for Elasticsearch. + +h2. Creating Elasticsearch Plugin + +p. The simplest way to deploy native script is by wrapping it into standard Elasticsearch plugin infrastructure. An Elasticsearch plugin can be written in java and built using gradle. A typical plugin source directory looks like this: + +bc.. . +|- build.gradle +|- src + |- main + | |- java + | | |- ... source code ... + |- test + |- java + | |- ... source code ... + |- resources + |- ... test resources ... + + +p. An Elasticsearch plugin can be created by following these six steps. + +* Create build.gradle file in the root directory of your plugin. +* Create source code directories: +** @mkdir -p src/main/java@ +** @mkdir -p src/test/java@ +** @mkdir -p src/test/resources@ +* The gradle plugin @esplugin@ provides all needed tasks to assemble the plugin. It is using the following settings that should be modified to match the project's plugin class name and license file definition. + +bc.. esplugin { + description 'ElasticSearch Plugin with Native Script Examples.' + classname 'org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin' + } + +p. + +* Create main Plugin class in the @src/main/java@ directory. This project is using @org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin@ class as an example, so the it has to be saved as @src/main/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesPlugin.java@ + +bc.. package org.elasticsearch.examples.nativescript.plugin; + +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.script.ScriptModule; + +public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin { + /* .... */ +} + +p. + +* The plugin can be built using @gradle build@ command and executed within elasticsearch by using @gradle run@. + +h2. Adding Native Scripts + +p. Now that the plugin infrastructure is complete, it's possible to add a native script. + +h3. Is Prime Native Script + +p. One of the example scripts in this project is the "is_prime" script that can be used to check if a field contains a possible prime number. The script accepts two parameters @field@ and @certainty@. The @field@ parameter contains the name of the field that needs to be checked and the @certainty@ parameter specifies a measure of the uncertainty that the caller is willing to tolerate. The script returns @true@ if the field contains a probable prime number and @false@ otherwise. The probability that the number for which the script returned @true@ is prime exceeds (1 - 0.5^certainty). The script can be used in "Script Filter":http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html as well as a "Script Field":http://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html. The implementation of the "is_prime" native script and it's factory can be found in the "IsPrimeSearchScript":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScript.java class. + +p. In order to enable native script creation the plugin has to contain and register a class that implements "NativeScriptFactory":https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/NativeScriptFactory.java. The NativeScriptFactory interface has only one method @newScript(Map params)@. This method accepts a list of parameters that are passed during script creation and returns an ExecutableScript object with an instance of the script. The factory has to be returned in the @onModule(ScriptModule module)@ method of the Plugin. + +bc.. public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin { + + @Override + public List getNativeScripts() { + return Arrays.asList(new IsPrimeSearchScript.Factory()); + } +} + +p. In general native scripts have to implement the interface "ExecutableScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/ExecutableScript.java, but if they are used in search, they have to also implement the "SearchScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/SearchScript.java interface. The SearchScript interface is quite complex, so in order to simplify implementation, the native script can simply extend the "AbstractSearchScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/AbstractSearchScript.java class instead. The AbstractSearchScript has only one abstract method @run()@. During search Elasticsearch calls this method for every single record in the search result. As in case of non-native script, the content of the current record can be accessed using DocLookup (@doc()@ method), FieldsLookup (@fields()@ method), SourceLookup (@source()@ method). + +h3. Lookup Script + +p. The "lookup script":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/LookupScript.java demonstrates how to make elsticsearch client available within the native script. When script factory is initially created, the node is not fully initialized yet. Because of this client cannot be directory injected into the factory. Instead, the reference to the node is injected and the client is obtained from the node during script creation. A same mechanism can be used to obtain other node components through node injector. + +h3. Random Sort Script + +p. The "random sort script":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptFactory.java demonstrates a slightly different approach to script/factory packaging. In this case the factory is the outer class which creates one inner script or another based on the input parameters. If the parameter @salt@ is present, the script is calculating hash value of @id + salt@ instead of generation random sort values. As a result, for any value of @salt@ the order of the records will appear random, but this order will be repeatable and therefore this approach would be more suitable for paging through result list than a completely random approach. + + diff --git a/plugins/native-script-example/build.gradle b/plugins/native-script-example/build.gradle new file mode 100644 index 0000000000000..e129154efcb55 --- /dev/null +++ b/plugins/native-script-example/build.gradle @@ -0,0 +1,15 @@ +esplugin { + description 'Demonstrates writtting elasticsearch plugins with native scripts' + classname 'org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin' +} + +compileJava.options.compilerArgs << "-Xlint:-deprecation" + +// uncomment to disable 3rd party audit +// thirdPartyAudit.enabled = false + +// uncomment to disable forbidden patterns checks +// forbiddenPatterns.enabled = false + +// uncomment to disable license headers checks +// licenseHeaders.enabled = false diff --git a/plugins/native-script-example/example/popularity.sh b/plugins/native-script-example/example/popularity.sh new file mode 100755 index 0000000000000..5faee142f6b79 --- /dev/null +++ b/plugins/native-script-example/example/popularity.sh @@ -0,0 +1,47 @@ +#!/bin/sh +curl -s -XDELETE "http://localhost:9200/test" +echo +curl -s -XPUT "http://localhost:9200/test/" -d '{ + "settings": { + "index.number_of_shards": 1, + "index.number_of_replicas": 0 + }, + "mappings": { + "type1": { + "properties": { + "name": { + "type": "string" + }, + "number": { + "type": "integer" + } + } + } + } +}' +echo +curl -s -XPUT "localhost:9200/test/type1/1" -d '{"name" : "foo bar baz", "number": 10000 }' +curl -s -XPUT "localhost:9200/test/type1/2" -d '{"name" : "foo foo foo", "number": 1 }' +curl -s -XPOST "http://localhost:9200/test/_refresh" +echo +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "function_score" : { + "boost_mode": "replace", + "query": { + "match": { + "name": "foo" + } + }, + "script_score": { + "script": "popularity", + "lang": "native", + "params": { + "field": "number" + } + } + } + } +} +' +echo \ No newline at end of file diff --git a/plugins/native-script-example/example/primes.sh b/plugins/native-script-example/example/primes.sh new file mode 100755 index 0000000000000..edfecc592c223 --- /dev/null +++ b/plugins/native-script-example/example/primes.sh @@ -0,0 +1,42 @@ +#!/bin/sh +curl -s -XDELETE "http://localhost:9200/test" +echo +curl -s -XPUT "http://localhost:9200/test/" -d '{ + "settings": { + "index.number_of_shards": 1, + "index.number_of_replicas": 0 + }, + "mappings": { + "type1": { + "properties": { + "name": { + "type": "string" + }, + "number": { + "type": "integer" + } + } + } + } +}' +echo +for i in {0..100}; do curl -s -XPUT "localhost:9200/test/type1/$i" -d "{\"name\":\"rec $i\", \"number\":$i}"; done +curl -s -XPOST "http://localhost:9200/test/_refresh" +echo +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "bool": { + "filter": { + "script": { + "script": "is_prime", + "lang": "native", + "params": { + "field": "number" + } + } + } + } + } +} +' + diff --git a/plugins/native-script-example/example/random.sh b/plugins/native-script-example/example/random.sh new file mode 100755 index 0000000000000..ac07dbec5777d --- /dev/null +++ b/plugins/native-script-example/example/random.sh @@ -0,0 +1,105 @@ +#!/bin/sh +curl -s -XDELETE "http://localhost:9200/test" +echo +curl -s -XPUT "http://localhost:9200/test/" -d '{ + "settings": { + "index.number_of_shards": 1, + "index.number_of_replicas": 0 + }, + "mappings": { + "type1": { + "properties": { + "name": { + "type": "string" + }, + "number": { + "type": "integer" + } + } + } + } +}' +echo +for i in {0..100}; do curl -s -XPUT "localhost:9200/test/type1/$i" -d "{\"name\":\"rec $i\", \"number\":$i}"; done +curl -s -XPOST "http://localhost:9200/test/_refresh" +echo +echo "Salt 123" +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "match_all": {} + }, + "sort": { + "_script": { + "script": "random", + "lang": "native", + "type": "number", + "params": { + "salt": "123" + } + } + } +} +' | grep \"_id\" +echo "Salt 123" +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "match_all": {} + }, + "sort": { + "_script": { + "script": "random", + "lang": "native", + "type": "number", + "params": { + "salt": "123" + } + } + } +} +' | grep \"_id\" +echo "Salt 124" +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "match_all": {} + }, + "sort": { + "_script": { + "script": "random", + "lang": "native", + "type": "number", + "params": { + "salt": "124" + } + } + } +} +' | grep \"_id\" +echo "No salt" +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "match_all": {} + }, + "sort": { + "_script": { + "script": "random", + "lang": "native", + "type": "number" + } + } +} +' | grep \"_id\" +echo "No salt" +curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ + "query": { + "match_all": {} + }, + "sort": { + "_script": { + "script": "random", + "lang": "native", + "type": "number" + } + } +} +' | grep \"_id\" +echo diff --git a/plugins/native-script-example/example/stockaggs.sh b/plugins/native-script-example/example/stockaggs.sh new file mode 100755 index 0000000000000..8ade779df580e --- /dev/null +++ b/plugins/native-script-example/example/stockaggs.sh @@ -0,0 +1,47 @@ +#!/bin/sh +curl -s -XDELETE "http://localhost:9200/transactions" +echo +curl -s -XPUT "http://localhost:9200/transactions/" -d '{ + "settings": { + "index.number_of_shards": 1, + "index.number_of_replicas": 0 + }, + "mappings": { + "stock": { + "properties": { + "type": { + "type": "string", + "index": "not_analyzed" + }, + "amount": { + "type": "long" + } + } + } + } +}' +echo +curl -s -XPUT 'http://localhost:9200/transactions/stock/1' -d '{"type": "sale", "amount": 80}' +curl -s -XPUT 'http://localhost:9200/transactions/stock/2' -d '{"type": "cost", "amount": 10}' +curl -s -XPUT 'http://localhost:9200/transactions/stock/3' -d '{"type": "cost", "amount": 30}' +curl -s -XPUT 'http://localhost:9200/transactions/stock/4' -d '{"type": "sale", "amount": 130}' + +curl -s -XPOST "http://localhost:9200/transactions/_refresh" +echo +curl -s -XGET "localhost:9200/transactions/stock/_search?pretty=true" -d '{ + "query" : { + "match_all" : {} + }, + "aggs": { + "profit": { + "scripted_metric": { + "init_script" : "stockaggs_init", + "map_script" : "stockaggs_map", + "combine_script" : "stockaggs_combine", + "reduce_script" : "stockaggs_reduce", + "lang": "native" + } + } + }, + "size": 0 +}' diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesPlugin.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesPlugin.java new file mode 100644 index 0000000000000..6bc29dfea4734 --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesPlugin.java @@ -0,0 +1,43 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.plugin; + +import org.elasticsearch.examples.nativescript.script.IsPrimeSearchScript; +import org.elasticsearch.examples.nativescript.script.PopularityScoreScriptFactory; +import org.elasticsearch.examples.nativescript.script.RandomSortScriptFactory; +import org.elasticsearch.examples.nativescript.script.stockaggs.CombineScriptFactory; +import org.elasticsearch.examples.nativescript.script.stockaggs.InitScriptFactory; +import org.elasticsearch.examples.nativescript.script.stockaggs.MapScriptFactory; +import org.elasticsearch.examples.nativescript.script.stockaggs.ReduceScriptFactory; +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.plugins.ScriptPlugin; +import org.elasticsearch.script.NativeScriptFactory; + +import java.util.Arrays; +import java.util.List; + +/** + * This class is instantiated when Elasticsearch loads the plugin for the + * first time. + */ +public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin { + + @Override + public List getNativeScripts() { + return Arrays.asList(new IsPrimeSearchScript.Factory(), new RandomSortScriptFactory(), new PopularityScoreScriptFactory(), + // Scripted Metric Aggregations Scripts + new InitScriptFactory(), new MapScriptFactory(), new CombineScriptFactory(), new ReduceScriptFactory()); + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScript.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScript.java new file mode 100644 index 0000000000000..7c1573f3d5e7d --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScript.java @@ -0,0 +1,113 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import java.math.BigInteger; +import java.util.Map; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.common.xcontent.support.XContentMapValues; +import org.elasticsearch.index.fielddata.ScriptDocValues; +import org.elasticsearch.index.fielddata.ScriptDocValues.Longs; +import org.elasticsearch.script.AbstractSearchScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +/** + * Implementation of the native script that checks that the field exists and contains a prime number. + *

+ * The native script has to implement {@link org.elasticsearch.script.SearchScript} interface. But the + * {@link org.elasticsearch.script.AbstractSearchScript} class can be used to simplify the implementation. + */ +public class IsPrimeSearchScript extends AbstractSearchScript { + + /** + * Native scripts are build using factories that are registered in the + * {@link org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin#onModule(org.elasticsearch.script.ScriptModule)} + * method when plugin is loaded. + */ + public static class Factory implements NativeScriptFactory { + + /** + * This method is called for every search on every shard. + * + * @param params list of script parameters passed with the query + * @return new native script + */ + @Override + public ExecutableScript newScript(@Nullable Map params) { + // Example of a mandatory string parameter + // The XContentMapValues helper class can be used to simplify parameter parsing + String fieldName = params == null ? null : XContentMapValues.nodeStringValue(params.get("field"), null); + if (fieldName == null) { + throw new IllegalArgumentException("Missing the field parameter"); + } + + // Example of an optional integer parameter + int certainty = params == null ? 10 : XContentMapValues.nodeIntegerValue(params.get("certainty"), 10); + return new IsPrimeSearchScript(fieldName, certainty); + } + + /** + * Indicates if document scores may be needed by the produced scripts. + * + * @return {@code true} if scores are needed. + */ + @Override + public boolean needsScores() { + return false; + } + + @Override + public String getName() { + return "is_prime"; + } + + } + + private final String fieldName; + + private final int certainty; + + /** + * Factory creates this script on every + * + * @param fieldName the name of the field that should be checked + * @param certainty the required certainty for the number to be prime + */ + private IsPrimeSearchScript(String fieldName, int certainty) { + this.fieldName = fieldName; + this.certainty = certainty; + } + + @Override + @SuppressWarnings("unchecked") + public Object run() { + // First we get field using doc lookup + ScriptDocValues docValue = (ScriptDocValues) doc().get(fieldName); + // Check if field exists + if (docValue != null && !docValue.isEmpty()) { + try { + // Try to parse it as an integer + BigInteger bigInteger = BigInteger.valueOf(((Longs) docValue).getValue()); + // Check if it's prime + return bigInteger.isProbablePrime(certainty); + } catch (NumberFormatException ex) { + return false; + } + } + return false; + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/PopularityScoreScriptFactory.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/PopularityScoreScriptFactory.java new file mode 100644 index 0000000000000..712cdb88263a1 --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/PopularityScoreScriptFactory.java @@ -0,0 +1,103 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import org.apache.lucene.search.Scorer; +import org.elasticsearch.common.Nullable; +import org.elasticsearch.common.xcontent.support.XContentMapValues; +import org.elasticsearch.index.fielddata.ScriptDocValues; +import org.elasticsearch.script.AbstractDoubleSearchScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +import java.io.IOException; +import java.util.Map; + +/** + * Factory for the script that boosts score of a record based on a value of the record's field. + *

+ * This native script demonstrates how to write native custom scores scripts. + */ +public class PopularityScoreScriptFactory implements NativeScriptFactory { + + @Override + public ExecutableScript newScript(@Nullable Map params) { + String fieldName = params == null ? null : XContentMapValues.nodeStringValue(params.get("field"), null); + if (fieldName == null) { + throw new IllegalArgumentException("Missing the field parameter"); + } + return new PopularityScoreScript(fieldName); + } + + /** + * Indicates if document scores may be needed by the produced scripts. + * + * @return {@code true} if scores are needed. + */ + @Override + public boolean needsScores() { + return true; + } + + @Override + public String getName() { + return "popularity"; + } + + /** + * This script takes a numeric value from the field specified in the parameter field. And calculates boost + * for the record using the following formula: 1 + log10(field_value + 1). So, records with value 0 in the field + * get no boost. Records with value 9 gets boost of 2.0, records with value 99, gets boost of 3, 999 - 4 and so on. + */ + private static class PopularityScoreScript extends AbstractDoubleSearchScript { + + private final String field; + + private Scorer scorer; + + public PopularityScoreScript(String field) { + this.field = field; + } + + @Override + public void setScorer(Scorer scorer) { + this.scorer = scorer; + } + + @Override + @SuppressWarnings("unchecked") + public double runAsDouble() { + try { + ScriptDocValues docValue = (ScriptDocValues) doc().get(field); + if (docValue != null && !docValue.isEmpty()) { + ScriptDocValues.Longs fieldData = (ScriptDocValues.Longs) docValue; + double boost = 1 + Math.log10(fieldData.getValue() + 1); + // Because this script is used in custom_score script the value of score() is populated. + // In all other cases doc().getScore() should be used instead. + return boost * scorer.score(); + + } + return scorer.score(); + } catch (IOException ex) { + return 0.0; + } + } + + @Override + public Object run() { + return null; + } + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptFactory.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptFactory.java new file mode 100644 index 0000000000000..c4d8b8aa3780f --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptFactory.java @@ -0,0 +1,113 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.common.Randomness; +import org.elasticsearch.common.xcontent.support.XContentMapValues; +import org.elasticsearch.index.fielddata.ScriptDocValues; +import org.elasticsearch.index.mapper.internal.UidFieldMapper; +import org.elasticsearch.script.AbstractLongSearchScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.Map; +import java.util.Random; + +/** + * This script demonstrates how native scripts can be used to create custom sort order. + * Since sort operation is expecting float parameter, the {@link AbstractLongSearchScript} can be used. + *

+ * The script accepts one optional parameter salt. If parameter is specified, a pseudo random sort order is used. + * Otherwise, a random sort order is used. + */ +public class RandomSortScriptFactory implements NativeScriptFactory { + + /** + * This method is called for every search on every shard. + * + * @param params list of script parameters passed with the query + * @return new native script + */ + @Override + public ExecutableScript newScript(@Nullable Map params) { + String salt = params == null ? null : XContentMapValues.nodeStringValue(params.get("salt"), null); + if (salt == null) { + return new RandomSortScript(); + } else { + return new PseudoRandomSortScript(salt); + } + } + + /** + * Indicates if document scores may be needed by the produced scripts. + * + * @return {@code true} if scores are needed. + */ + @Override + public boolean needsScores() { + return false; + } + + @Override + public String getName() { + return "random"; + } + + private static class RandomSortScript extends AbstractLongSearchScript { + private final Random random; + + private RandomSortScript() { + random = Randomness.get(); + } + + @Override + public long runAsLong() { + return random.nextLong(); + } + } + + private static class PseudoRandomSortScript extends AbstractLongSearchScript { + private final String salt; + + private PseudoRandomSortScript(String salt) { + this.salt = salt; + } + + @Override + public long runAsLong() { + ScriptDocValues.Strings fieldData = (ScriptDocValues.Strings) doc().get(UidFieldMapper.NAME); + try { + MessageDigest m = MessageDigest.getInstance("MD5"); + m.reset(); + m.update((fieldData.getValue() + salt).getBytes(StandardCharsets.UTF_8)); + byte[] sort = m.digest(); + return (sort[0] & 0xFFL) << 56 + | (sort[1] & 0xFFL) << 48 + | (sort[2] & 0xFFL) << 40 + | (sort[3] & 0xFFL) << 32 + | (sort[4] & 0xFFL) << 24 + | (sort[5] & 0xFFL) << 16 + | (sort[6] & 0xFFL) << 8 + | (sort[7] & 0xFFL); + } catch (NoSuchAlgorithmException ex) { + return -1; + } + } + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/CombineScriptFactory.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/CombineScriptFactory.java new file mode 100644 index 0000000000000..8d15b4795bfa7 --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/CombineScriptFactory.java @@ -0,0 +1,68 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script.stockaggs; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.script.AbstractExecutableScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +import java.util.ArrayList; +import java.util.Map; + +/** + * Combine script from + * https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html + *

+ * profit = 0; for (t in _agg.transactions) { profit += t }; return profit + */ +public class CombineScriptFactory implements NativeScriptFactory { + + @Override + @SuppressWarnings("unchecked") + public ExecutableScript newScript(@Nullable final Map params) { + Map agg = (Map) params.get("_agg"); + final ArrayList transactions = (ArrayList) agg.get(InitScriptFactory.TRANSACTIONS_FIELD); + return new CombineScript(transactions); + } + + @Override + public boolean needsScores() { + return false; + } + + @Override + public String getName() { + return "stockaggs_combine"; + } + + private static class CombineScript extends AbstractExecutableScript { + + private final ArrayList transactions; + + public CombineScript(ArrayList transactions) { + this.transactions = transactions; + } + + @Override + public Object run() { + long profit = 0; + for (long t : transactions) { + profit += t; + } + return profit; + } + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/InitScriptFactory.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/InitScriptFactory.java new file mode 100644 index 0000000000000..9754bc2096ac2 --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/InitScriptFactory.java @@ -0,0 +1,56 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script.stockaggs; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.script.AbstractExecutableScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +import java.util.ArrayList; +import java.util.Map; + +/** + * Init script from + * https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html + * + * _agg['transactions'] = [] + */ +public class InitScriptFactory implements NativeScriptFactory { + + public static final String TRANSACTIONS_FIELD = "transactions"; + + @Override + @SuppressWarnings("unchecked") + public ExecutableScript newScript(@Nullable final Map params) { + return new AbstractExecutableScript() { + @Override + public Object run() { + ((Map)params.get("_agg")).put(TRANSACTIONS_FIELD, new ArrayList<>()); + return null; + } + }; + } + + @Override + public boolean needsScores() { + return false; + } + + @Override + public String getName() { + return "stockaggs_init"; + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/MapScriptFactory.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/MapScriptFactory.java new file mode 100644 index 0000000000000..9887a9899c203 --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/MapScriptFactory.java @@ -0,0 +1,73 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script.stockaggs; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.index.fielddata.ScriptDocValues; +import org.elasticsearch.script.AbstractSearchScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +import java.util.ArrayList; +import java.util.Map; + +/** + * Map script from + * https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html + *

+ * if (doc['type'].value == \"sale\") { _agg.transactions.add(doc['amount'].value) } else {_agg.transactions.add(-1 * doc['amount'].value)} + */ +public class MapScriptFactory implements NativeScriptFactory { + + @Override + @SuppressWarnings("unchecked") + public ExecutableScript newScript(@Nullable final Map params) { + Map agg = (Map) params.get("_agg"); + ArrayList transactions = (ArrayList) agg.get(InitScriptFactory.TRANSACTIONS_FIELD); + return new MapScript(transactions); + } + + @Override + public boolean needsScores() { + return false; + } + + @Override + public String getName() { + return "stockaggs_map"; + } + + private static class MapScript extends AbstractSearchScript { + + private final ArrayList transactions; + + public MapScript(ArrayList transactions) { + this.transactions = transactions; + + } + + @Override + public Object run() { + ScriptDocValues.Longs amount = (ScriptDocValues.Longs) doc().get("amount"); + ScriptDocValues.Strings type = (ScriptDocValues.Strings) doc().get("type"); + if ("sale".equals(type.getValue())) { + transactions.add(amount.getValue()); + } else { + transactions.add(-amount.getValue()); + } + return null; + } + } +} diff --git a/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/ReduceScriptFactory.java b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/ReduceScriptFactory.java new file mode 100644 index 0000000000000..37ef7111d03af --- /dev/null +++ b/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/stockaggs/ReduceScriptFactory.java @@ -0,0 +1,67 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script.stockaggs; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.script.AbstractExecutableScript; +import org.elasticsearch.script.ExecutableScript; +import org.elasticsearch.script.NativeScriptFactory; + +import java.util.ArrayList; +import java.util.Map; + +/** + * Combine script from + * https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html + *

+ * profit = 0; for (t in _agg.transactions) { profit += t }; return profit + */ +public class ReduceScriptFactory implements NativeScriptFactory { + + @Override + @SuppressWarnings("unchecked") + public ExecutableScript newScript(@Nullable final Map params) { + final ArrayList aggs = (ArrayList) params.get("_aggs"); + return new ReduceScript(aggs); + } + + @Override + public boolean needsScores() { + return false; + } + + @Override + public String getName() { + return "stockaggs_reduce"; + } + + private static class ReduceScript extends AbstractExecutableScript { + + private final ArrayList aggs; + + public ReduceScript(ArrayList aggs) { + this.aggs = aggs; + } + + @Override + public Object run() { + long profit = 0; + for (long t : aggs) { + profit += t; + } + return profit; + } + } +} diff --git a/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesRestIT.java b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesRestIT.java new file mode 100644 index 0000000000000..f9a751ac9ad51 --- /dev/null +++ b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesRestIT.java @@ -0,0 +1,35 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.examples.nativescript.plugin; + +import com.carrotsearch.randomizedtesting.annotations.Name; +import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; +import org.elasticsearch.test.rest.ESRestTestCase; +import org.elasticsearch.test.rest.RestTestCandidate; +import org.elasticsearch.test.rest.parser.RestTestParseException; + +import java.io.IOException; + +public class NativeScriptExamplesRestIT extends ESRestTestCase { + + public NativeScriptExamplesRestIT(@Name("yaml") RestTestCandidate testCandidate) { + super(testCandidate); + } + + @ParametersFactory + public static Iterable parameters() throws IOException, RestTestParseException { + return ESRestTestCase.createParameters(0, 1); + } +} + diff --git a/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/AbstractSearchScriptTestCase.java b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/AbstractSearchScriptTestCase.java new file mode 100644 index 0000000000000..21e10e46e4ad5 --- /dev/null +++ b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/AbstractSearchScriptTestCase.java @@ -0,0 +1,45 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.examples.nativescript.script; + +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin; +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.test.ESIntegTestCase; +import org.elasticsearch.test.ESIntegTestCase.ClusterScope; +import org.elasticsearch.test.ESIntegTestCase.Scope; + +import java.util.Collection; + +import static org.elasticsearch.cluster.metadata.IndexMetaData.SETTING_NUMBER_OF_REPLICAS; +import static org.elasticsearch.cluster.metadata.IndexMetaData.SETTING_NUMBER_OF_SHARDS; + +/** + */ +@ClusterScope(scope = Scope.SUITE, numDataNodes = 1) +public abstract class AbstractSearchScriptTestCase extends ESIntegTestCase { + + @Override + public Settings indexSettings() { + Settings.Builder builder = Settings.builder(); + builder.put(SETTING_NUMBER_OF_SHARDS, 1); + builder.put(SETTING_NUMBER_OF_REPLICAS, 0); + return builder.build(); + } + + @Override + protected Collection> nodePlugins() { + return pluginList(NativeScriptExamplesPlugin.class); + } +} diff --git a/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScriptTests.java b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScriptTests.java new file mode 100644 index 0000000000000..4b899eaa31564 --- /dev/null +++ b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScriptTests.java @@ -0,0 +1,111 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.xcontent.XContentFactory; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptService; +import org.elasticsearch.search.sort.SortOrder; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.elasticsearch.index.query.QueryBuilders.scriptQuery; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures; +import static org.hamcrest.Matchers.equalTo; + +/** + */ +public class IsPrimeSearchScriptTests extends AbstractSearchScriptTestCase { + + public static int[] PRIMES_10 = new int[]{2, 3, 5, 7, 11, 13, 17, 19, 23, 29}; + + public void testIsPrimeScript() throws Exception { + + // Create a new index + String mapping = XContentFactory.jsonBuilder().startObject().startObject("type") + .startObject("properties") + .startObject("name").field("type", "string").endObject() + .startObject("number").field("type", "integer").endObject() + .endObject().endObject().endObject() + .string(); + + assertAcked(prepareCreate("test") + .addMapping("type", mapping)); + + List indexBuilders = new ArrayList(); + // Index 100 records (0..99) + for (int i = 0; i < 100; i++) { + indexBuilders.add( + client().prepareIndex("test", "type", Integer.toString(i)) + .setSource(XContentFactory.jsonBuilder().startObject() + .field("name", "rec " + i) + .field("number", i) + .endObject())); + } + // Index a few records with empty number + for (int i = 100; i < 105; i++) { + indexBuilders.add( + client().prepareIndex("test", "type", Integer.toString(i)) + .setSource(XContentFactory.jsonBuilder().startObject() + .field("name", "rec " + i) + .endObject())); + } + + indexRandom(true, indexBuilders); + + Map params = new HashMap<>(); + params.put("field", "number"); + // Retrieve first 10 prime records + SearchResponse searchResponse = client().prepareSearch("test") + .setQuery(scriptQuery(new Script("is_prime", ScriptService.ScriptType.INLINE, "native", params))) + .setFetchSource("name", null) + .setSize(10) + .addSort("number", SortOrder.ASC) + .execute().actionGet(); + + assertNoFailures(searchResponse); + + // There should be 25 prime numbers between 0 and 100 + assertHitCount(searchResponse, 25); + + // Verify that they are indeed prime numbers + for (int i = 0; i < 10; i++) { + assertThat(searchResponse.getHits().getAt(i).getSource().get("name"), equalTo("rec " + PRIMES_10[i])); + } + + params = new HashMap<>(); + params.put("field", "number"); + params.put("certainty", 0); + // Check certainty parameter - with certainty == 0, it should return all numbers, but only if numbers are present + searchResponse = client().prepareSearch("test") + .setQuery(scriptQuery(new Script("is_prime", ScriptService.ScriptType.INLINE, "native", params))) + .setFetchSource("name", null) + .setSize(10) + .addSort("number", SortOrder.ASC) + .execute().actionGet(); + assertNoFailures(searchResponse); + // With certainty 0 no check is done so it should return all numbers + assertHitCount(searchResponse, 100); + + } + +} diff --git a/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/PopularityScoreScriptTests.java b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/PopularityScoreScriptTests.java new file mode 100644 index 0000000000000..1a2d8d45c68dd --- /dev/null +++ b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/PopularityScoreScriptTests.java @@ -0,0 +1,108 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import static org.elasticsearch.index.query.QueryBuilders.functionScoreQuery; +import static org.elasticsearch.index.query.QueryBuilders.matchQuery; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.greaterThan; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.collect.MapBuilder; +import org.elasticsearch.common.lucene.search.function.CombineFunction; +import org.elasticsearch.common.xcontent.XContentFactory; +import org.elasticsearch.index.query.QueryBuilders; +import org.elasticsearch.index.query.functionscore.FunctionScoreQueryBuilder; +import org.elasticsearch.index.query.functionscore.ScoreFunctionBuilders; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptService; + +/** + */ +public class PopularityScoreScriptTests extends AbstractSearchScriptTestCase { + + public void testPopularityScoring() throws Exception { + + // Create a new index + String mapping = XContentFactory.jsonBuilder().startObject().startObject("type") + .startObject("properties") + .startObject("name").field("type", "string").endObject() + .startObject("number").field("type", "integer").endObject() + .endObject().endObject().endObject() + .string(); + + assertAcked(prepareCreate("test") + .addMapping("type", mapping)); + + List indexBuilders = new ArrayList(); + + // Index 5 records with non-empty number field + for (int i = 0; i < 5; i++) { + indexBuilders.add( + client().prepareIndex("test", "type", Integer.toString(i)) + .setSource(XContentFactory.jsonBuilder().startObject() + .field("name", "rec " + i) + .field("number", i + 1) + .endObject())); + } + // Index a few records with empty number + for (int i = 5; i < 10; i++) { + indexBuilders.add( + client().prepareIndex("test", "type", Integer.toString(i)) + .setSource(XContentFactory.jsonBuilder().startObject() + .field("name", "rec " + i) + .endObject())); + } + + indexRandom(true, indexBuilders); + + Map params = MapBuilder.newMapBuilder().put("field", "number").map(); + // Retrieve first 10 hits + SearchResponse searchResponse = client().prepareSearch("test") + .setQuery(QueryBuilders.functionScoreQuery(matchQuery("name", "rec"), new FunctionScoreQueryBuilder.FilterFunctionBuilder[]{ + new FunctionScoreQueryBuilder.FilterFunctionBuilder(ScoreFunctionBuilders.scriptFunction( + new Script("popularity", ScriptService.ScriptType.INLINE, "native", params)))}) + .boostMode(CombineFunction.REPLACE)) + .setSize(10) + .setFetchSource("name", null) + .execute().actionGet(); + + assertNoFailures(searchResponse); + + // There should be 10 hist + assertHitCount(searchResponse, 10); + + // Verify that first 5 hits are sorted from 4 to 0 + for (int i = 0; i < 5; i++) { + assertThat(searchResponse.getHits().getAt(i).sourceAsMap().get("name"), equalTo("rec " + (4 - i))); + } + + // Verify that hit 5 has non-zero score + assertThat(searchResponse.getHits().getAt(5).score(), greaterThan(0.0f)); + + // Verify that the last 5 hits has the same score + for (int i = 6; i < 10; i++) { + assertThat(searchResponse.getHits().getAt(i).score(), equalTo(searchResponse.getHits().getAt(5).score())); + } + } +} diff --git a/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptTests.java b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptTests.java new file mode 100644 index 0000000000000..76aae74f6fb97 --- /dev/null +++ b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptTests.java @@ -0,0 +1,125 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.not; + +import java.util.ArrayList; +import java.util.List; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.collect.MapBuilder; +import org.elasticsearch.common.xcontent.XContentFactory; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptService; +import org.elasticsearch.search.sort.ScriptSortBuilder; +import org.elasticsearch.search.sort.SortBuilders; + +/** + */ +public class RandomSortScriptTests extends AbstractSearchScriptTestCase { + + public void testPseudoRandomScript() throws Exception { + + // Create a new index + String mapping = XContentFactory.jsonBuilder().startObject().startObject("type") + .startObject("properties") + .startObject("name").field("type", "string").endObject() + .endObject().endObject().endObject() + .string(); + + assertAcked(prepareCreate("test") + .addMapping("type", mapping)); + + List indexBuilders = new ArrayList<>(); + + // Index 100 records (0..99) + for (int i = 0; i < 100; i++) { + indexBuilders.add( + client().prepareIndex("test", "type", Integer.toString(i)) + .setSource(XContentFactory.jsonBuilder().startObject() + .field("name", "rec " + i) + .endObject())); + } + + indexRandom(true, indexBuilders); + + // Retrieve first 10 records + SearchResponse searchResponse = client().prepareSearch("test") + .setQuery(matchAllQuery()) + .setFetchSource("name", null) + .setSize(10) + .addSort(SortBuilders.scriptSort( + new Script("random", ScriptService.ScriptType.INLINE, "native", MapBuilder.newMapBuilder() + .put("salt", "1234").map()), ScriptSortBuilder.ScriptSortType.NUMBER)) + .execute().actionGet(); + + assertNoFailures(searchResponse); + + // Check that random order was applied + assertThat(searchResponse.getHits().getAt(0).getSource().get("name"), not(equalTo("rec0"))); + + String[] records = new String[10]; + + // Store sort order + for (int i = 0; i < 10; i++) { + records[i] = searchResponse.getHits().getAt(i).getSource().get("name").toString(); + } + + // Retrieve first 10 records again + searchResponse = client().prepareSearch("test") + .setQuery(matchAllQuery()) + .setFetchSource("name", null) + .setSize(10) + .addSort(SortBuilders.scriptSort(new Script("random", ScriptService.ScriptType.INLINE, "native", MapBuilder.newMapBuilder() + .put("salt", "1234").map()), ScriptSortBuilder.ScriptSortType.NUMBER)) + .execute().actionGet(); + + assertNoFailures(searchResponse); + + // Verify the same sort order + for (int i = 0; i < 10; i++) { + assertThat(searchResponse.getHits().getAt(i).getSource().get("name"), equalTo(records[i])); + } + + // Retrieve first 10 records without salt + searchResponse = client().prepareSearch("test") + .setQuery(matchAllQuery()) + .setFetchSource("name", null) + .setSize(10) + .addSort(SortBuilders.scriptSort(new Script("random", ScriptService.ScriptType.INLINE, "native", null), + ScriptSortBuilder.ScriptSortType.NUMBER)) + .execute().actionGet(); + + assertNoFailures(searchResponse); + + // Verify different sort order + boolean different = false; + for (int i = 0; i < 10; i++) { + if (!records[i].equals(searchResponse.getHits().getAt(i).getSource().get("name"))) { + different = true; + break; + } + } + assertThat(different, equalTo(true)); + + } +} diff --git a/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/ScriptedMetricAggsScriptTests.java b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/ScriptedMetricAggsScriptTests.java new file mode 100644 index 0000000000000..9905c69abbe6a --- /dev/null +++ b/plugins/native-script-example/src/test/java/org/elasticsearch/examples/nativescript/script/ScriptedMetricAggsScriptTests.java @@ -0,0 +1,82 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.elasticsearch.examples.nativescript.script; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.xcontent.XContentFactory; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptService; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.scriptedMetric; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures; +import static org.hamcrest.Matchers.equalTo; + +/** + */ +public class ScriptedMetricAggsScriptTests extends AbstractSearchScriptTestCase { + + @SuppressWarnings("unchecked") + public void testScriptedMetricAggs() throws Exception { + + // Create a new lookup index + String stockMapping = XContentFactory.jsonBuilder().startObject().startObject("stock") + .startObject("properties") + .startObject("type").field("type", "string").field("index", "not_analyzed").endObject() + .startObject("amount").field("type", "long").endObject() + .endObject().endObject().endObject() + .string(); + + assertAcked(prepareCreate("transactions") + .addMapping("stock", stockMapping)); + + List indexBuilders = new ArrayList(); + // Index stock records: + indexBuilders.add(client().prepareIndex("transactions", "stock", "1").setSource("type", "sale", "amount", 80)); + indexBuilders.add(client().prepareIndex("transactions", "stock", "2").setSource("type", "cost", "amount", 10)); + indexBuilders.add(client().prepareIndex("transactions", "stock", "3").setSource("type", "cost", "amount", 30)); + indexBuilders.add(client().prepareIndex("transactions", "stock", "4").setSource("type", "sale", "amount", 130)); + + indexRandom(true, indexBuilders); + + // Find profit from all transaction + SearchResponse searchResponse = client().prepareSearch("transactions") + .setTypes("stock") + .setQuery(matchAllQuery()) + .setSize(0) + .addAggregation(scriptedMetric("profit") + .initScript(new Script("stockaggs_init", ScriptService.ScriptType.INLINE, "native", null)) + .mapScript(new Script("stockaggs_map", ScriptService.ScriptType.INLINE, "native", null)) + .combineScript(new Script("stockaggs_combine", ScriptService.ScriptType.INLINE, "native", null)) + .reduceScript(new Script("stockaggs_reduce", ScriptService.ScriptType.INLINE, "native", null))) + .execute().actionGet(); + + assertNoFailures(searchResponse); + + // There should be 4 hits - we are running aggs on everything + assertHitCount(searchResponse, 4); + + // The profit should be 170 + assertThat((long) searchResponse.getAggregations().get("profit").getProperty("value"), equalTo(170L)); + } + + +} diff --git a/plugins/native-script-example/src/test/resources/rest-api-spec/test/nativescript/10_primes.yaml b/plugins/native-script-example/src/test/resources/rest-api-spec/test/nativescript/10_primes.yaml new file mode 100644 index 0000000000000..858d11f111f22 --- /dev/null +++ b/plugins/native-script-example/src/test/resources/rest-api-spec/test/nativescript/10_primes.yaml @@ -0,0 +1,50 @@ +# Integration tests for the lookup script +# +setup: + - do: + indices.create: + index: test_index + body: + settings: + index.number_of_shards: 1 + index.number_of_replicas: 0 + mappings: + type1: + properties: + name: {type: text} + number: {type: integer} + + + - do: + index: {index: test_index, type: type1, id: 1, body: {name: "rec 1", number: 1}} + + - do: + index: {index: test_index, type: type1, id: 2, body: {name: "rec 2", number: 2}} + + - do: + index: {index: test_index, type: type1, id: 3, body: {name: "rec 3", number: 3}} + + - do: + index: {index: test_index, type: type1, id: 4, body: {name: "rec 4", number: 4}} + + - do: + index: {index: test_index, type: type1, id: 5, body: {name: "rec 5", number: 5}} + - do: + indices.refresh: {} + +--- + +"Primes Test": + - do: + search: + body: + query: + constant_score: + filter: + script: + script: is_prime + lang: native + params: + field: number + + - match: { hits.total: 3} diff --git a/plugins/native-script-example/src/test/resources/rest-api-spec/test/nativescript/20_aggs.yaml b/plugins/native-script-example/src/test/resources/rest-api-spec/test/nativescript/20_aggs.yaml new file mode 100644 index 0000000000000..a2b79101af73e --- /dev/null +++ b/plugins/native-script-example/src/test/resources/rest-api-spec/test/nativescript/20_aggs.yaml @@ -0,0 +1,45 @@ +# Integration tests for aggregation scripts +# +setup: + - do: + indices.create: + index: transactions + body: + settings: + index.number_of_shards: 1 + index.number_of_replicas: 0 + mappings: + stock: + properties: + type: {type: keyword} + amount: {type: long} + + + - do: + index: {index: transactions, type: stock, id: 1, body: {type: "sale", amount: 80}} + + - do: + index: {index: transactions, type: stock, id: 2, body: {type: "cost", amount: 10}} + + - do: + index: {index: transactions, type: stock, id: 3, body: {type: "cost", amount: 30}} + + - do: + index: {index: transactions, type: stock, id: 4, body: {type: "sale", amount: 130}} + + - do: + indices.refresh: {} + +--- + +"Primes Test": + - do: + search: + index: transactions + body: + aggs: + profit: + scripted_metric: { "init_script" : "stockaggs_init", "map_script" : "stockaggs_map", "combine_script" : "stockaggs_combine", "reduce_script" : "stockaggs_reduce", "lang": "native"} + + - match: { hits.total: 4} + - match: { aggregations.profit.value: 170} diff --git a/settings.gradle b/settings.gradle index 3896082d9d8b4..64cb4551f0323 100644 --- a/settings.gradle +++ b/settings.gradle @@ -42,6 +42,7 @@ List projects = [ 'plugins:mapper-attachments', 'plugins:mapper-murmur3', 'plugins:mapper-size', + 'plugins:native-script-example', 'plugins:repository-azure', 'plugins:repository-gcs', 'plugins:repository-hdfs',