-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate elasticsearch native script examples to the main repo #19334
Changes from all commits
fc57847
4dc7b46
4e8d053
b15f57f
ecb5cc5
418c4c8
9525fa5
c9629a1
25b1565
92c4950
f90c8b0
cb8fcc5
9ab9fb1
211f267
2b184f3
e5d10c7
210aab0
574fd62
803337a
d12bfa7
f280b76
a7f8b4d
1e2da9c
eed5d53
1e3c365
0420b7a
d472879
035dc8f
4d774aa
a6db7f9
5604d3d
3e22997
29975b7
f365998
93d8b81
c5ed8ee
1254c48
9406941
d6a8b87
144e23f
6ccdf8c
54e3080
089f4c9
96f8b93
d12279d
6c4ba4a
4cda2cc
f5feb9e
b95943e
2b87eb9
8881606
a19cdcd
0f4152f
e41024a
089b01c
4a2debe
2ba2cd6
89f2009
b0efbbb
db6c568
548c090
375b7f7
db8281b
6ae94c7
ff3c1aa
5b22950
c418732
8aa2eb4
3a42192
4cdbdf7
7df2d06
3c7ea4c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
h1. Example of Native Script Plugin for Elasticsearch | ||
|
||
h2. Introduction | ||
|
||
p. This plugin contains several examples of "native script":http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html for Elasticsearch. | ||
|
||
h2. Creating Elasticsearch Plugin | ||
|
||
p. The simplest way to deploy native script is by wrapping it into standard Elasticsearch plugin infrastructure. An Elasticsearch plugin can be written in java and built using gradle. A typical plugin source directory looks like this: | ||
|
||
bc.. . | ||
|- build.gradle | ||
|- src | ||
|- main | ||
| |- java | ||
| | |- ... source code ... | ||
|- test | ||
|- java | ||
| |- ... source code ... | ||
|- resources | ||
|- ... test resources ... | ||
|
||
|
||
p. An Elasticsearch plugin can be created by following these six steps. | ||
|
||
* Create build.gradle file in the root directory of your plugin. | ||
* Create source code directories: | ||
** @mkdir -p src/main/java@ | ||
** @mkdir -p src/test/java@ | ||
** @mkdir -p src/test/resources@ | ||
* The gradle plugin @esplugin@ provides all needed tasks to assemble the plugin. It is using the following settings that should be modified to match the project's plugin class name and license file definition. | ||
|
||
bc.. esplugin { | ||
description 'ElasticSearch Plugin with Native Script Examples.' | ||
classname 'org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin' | ||
} | ||
|
||
p. | ||
|
||
* Create main Plugin class in the @src/main/java@ directory. This project is using @org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin@ class as an example, so the it has to be saved as @src/main/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesPlugin.java@ | ||
|
||
bc.. package org.elasticsearch.examples.nativescript.plugin; | ||
|
||
import org.elasticsearch.plugins.Plugin; | ||
import org.elasticsearch.script.ScriptModule; | ||
|
||
public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin { | ||
/* .... */ | ||
} | ||
|
||
p. | ||
|
||
* The plugin can be built using @gradle build@ command and executed within elasticsearch by using @gradle run@. | ||
|
||
h2. Adding Native Scripts | ||
|
||
p. Now that the plugin infrastructure is complete, it's possible to add a native script. | ||
|
||
h3. Is Prime Native Script | ||
|
||
p. One of the example scripts in this project is the "is_prime" script that can be used to check if a field contains a possible prime number. The script accepts two parameters @field@ and @certainty@. The @field@ parameter contains the name of the field that needs to be checked and the @certainty@ parameter specifies a measure of the uncertainty that the caller is willing to tolerate. The script returns @true@ if the field contains a probable prime number and @false@ otherwise. The probability that the number for which the script returned @true@ is prime exceeds (1 - 0.5^certainty). The script can be used in "Script Filter":http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html as well as a "Script Field":http://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html. The implementation of the "is_prime" native script and it's factory can be found in the "IsPrimeSearchScript":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScript.java class. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would be better to have documentation about the examples on the script classes themselves? |
||
|
||
p. In order to enable native script creation the plugin has to contain and register a class that implements "NativeScriptFactory":https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/NativeScriptFactory.java. The NativeScriptFactory interface has only one method @newScript(Map<String, Object> params)@. This method accepts a list of parameters that are passed during script creation and returns an ExecutableScript object with an instance of the script. The factory has to be returned in the @onModule(ScriptModule module)@ method of the Plugin. | ||
|
||
bc.. public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin { | ||
|
||
@Override | ||
public List<NativeScriptFactory> getNativeScripts() { | ||
return Arrays.asList(new IsPrimeSearchScript.Factory()); | ||
} | ||
} | ||
|
||
p. In general native scripts have to implement the interface "ExecutableScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/ExecutableScript.java, but if they are used in search, they have to also implement the "SearchScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/SearchScript.java interface. The SearchScript interface is quite complex, so in order to simplify implementation, the native script can simply extend the "AbstractSearchScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/AbstractSearchScript.java class instead. The AbstractSearchScript has only one abstract method @run()@. During search Elasticsearch calls this method for every single record in the search result. As in case of non-native script, the content of the current record can be accessed using DocLookup (@doc()@ method), FieldsLookup (@fields()@ method), SourceLookup (@source()@ method). | ||
|
||
h3. Lookup Script | ||
|
||
p. The "lookup script":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/LookupScript.java demonstrates how to make elsticsearch client available within the native script. When script factory is initially created, the node is not fully initialized yet. Because of this client cannot be directory injected into the factory. Instead, the reference to the node is injected and the client is obtained from the node during script creation. A same mechanism can be used to obtain other node components through node injector. | ||
|
||
h3. Random Sort Script | ||
|
||
p. The "random sort script":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptFactory.java demonstrates a slightly different approach to script/factory packaging. In this case the factory is the outer class which creates one inner script or another based on the input parameters. If the parameter @salt@ is present, the script is calculating hash value of @id + salt@ instead of generation random sort values. As a result, for any value of @salt@ the order of the records will appear random, but this order will be repeatable and therefore this approach would be more suitable for paging through result list than a completely random approach. | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
esplugin { | ||
description 'Demonstrates writtting elasticsearch plugins with native scripts' | ||
classname 'org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin' | ||
} | ||
|
||
compileJava.options.compilerArgs << "-Xlint:-deprecation" | ||
|
||
// uncomment to disable 3rd party audit | ||
// thirdPartyAudit.enabled = false | ||
|
||
// uncomment to disable forbidden patterns checks | ||
// forbiddenPatterns.enabled = false | ||
|
||
// uncomment to disable license headers checks | ||
// licenseHeaders.enabled = false |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
#!/bin/sh | ||
curl -s -XDELETE "http://localhost:9200/test" | ||
echo | ||
curl -s -XPUT "http://localhost:9200/test/" -d '{ | ||
"settings": { | ||
"index.number_of_shards": 1, | ||
"index.number_of_replicas": 0 | ||
}, | ||
"mappings": { | ||
"type1": { | ||
"properties": { | ||
"name": { | ||
"type": "string" | ||
}, | ||
"number": { | ||
"type": "integer" | ||
} | ||
} | ||
} | ||
} | ||
}' | ||
echo | ||
curl -s -XPUT "localhost:9200/test/type1/1" -d '{"name" : "foo bar baz", "number": 10000 }' | ||
curl -s -XPUT "localhost:9200/test/type1/2" -d '{"name" : "foo foo foo", "number": 1 }' | ||
curl -s -XPOST "http://localhost:9200/test/_refresh" | ||
echo | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"function_score" : { | ||
"boost_mode": "replace", | ||
"query": { | ||
"match": { | ||
"name": "foo" | ||
} | ||
}, | ||
"script_score": { | ||
"script": "popularity", | ||
"lang": "native", | ||
"params": { | ||
"field": "number" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
' | ||
echo |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
#!/bin/sh | ||
curl -s -XDELETE "http://localhost:9200/test" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Instead of having these shell tests, I think we should show how to write unit tests? |
||
echo | ||
curl -s -XPUT "http://localhost:9200/test/" -d '{ | ||
"settings": { | ||
"index.number_of_shards": 1, | ||
"index.number_of_replicas": 0 | ||
}, | ||
"mappings": { | ||
"type1": { | ||
"properties": { | ||
"name": { | ||
"type": "string" | ||
}, | ||
"number": { | ||
"type": "integer" | ||
} | ||
} | ||
} | ||
} | ||
}' | ||
echo | ||
for i in {0..100}; do curl -s -XPUT "localhost:9200/test/type1/$i" -d "{\"name\":\"rec $i\", \"number\":$i}"; done | ||
curl -s -XPOST "http://localhost:9200/test/_refresh" | ||
echo | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"bool": { | ||
"filter": { | ||
"script": { | ||
"script": "is_prime", | ||
"lang": "native", | ||
"params": { | ||
"field": "number" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
' | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
#!/bin/sh | ||
curl -s -XDELETE "http://localhost:9200/test" | ||
echo | ||
curl -s -XPUT "http://localhost:9200/test/" -d '{ | ||
"settings": { | ||
"index.number_of_shards": 1, | ||
"index.number_of_replicas": 0 | ||
}, | ||
"mappings": { | ||
"type1": { | ||
"properties": { | ||
"name": { | ||
"type": "string" | ||
}, | ||
"number": { | ||
"type": "integer" | ||
} | ||
} | ||
} | ||
} | ||
}' | ||
echo | ||
for i in {0..100}; do curl -s -XPUT "localhost:9200/test/type1/$i" -d "{\"name\":\"rec $i\", \"number\":$i}"; done | ||
curl -s -XPOST "http://localhost:9200/test/_refresh" | ||
echo | ||
echo "Salt 123" | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"match_all": {} | ||
}, | ||
"sort": { | ||
"_script": { | ||
"script": "random", | ||
"lang": "native", | ||
"type": "number", | ||
"params": { | ||
"salt": "123" | ||
} | ||
} | ||
} | ||
} | ||
' | grep \"_id\" | ||
echo "Salt 123" | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"match_all": {} | ||
}, | ||
"sort": { | ||
"_script": { | ||
"script": "random", | ||
"lang": "native", | ||
"type": "number", | ||
"params": { | ||
"salt": "123" | ||
} | ||
} | ||
} | ||
} | ||
' | grep \"_id\" | ||
echo "Salt 124" | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"match_all": {} | ||
}, | ||
"sort": { | ||
"_script": { | ||
"script": "random", | ||
"lang": "native", | ||
"type": "number", | ||
"params": { | ||
"salt": "124" | ||
} | ||
} | ||
} | ||
} | ||
' | grep \"_id\" | ||
echo "No salt" | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"match_all": {} | ||
}, | ||
"sort": { | ||
"_script": { | ||
"script": "random", | ||
"lang": "native", | ||
"type": "number" | ||
} | ||
} | ||
} | ||
' | grep \"_id\" | ||
echo "No salt" | ||
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{ | ||
"query": { | ||
"match_all": {} | ||
}, | ||
"sort": { | ||
"_script": { | ||
"script": "random", | ||
"lang": "native", | ||
"type": "number" | ||
} | ||
} | ||
} | ||
' | grep \"_id\" | ||
echo |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
#!/bin/sh | ||
curl -s -XDELETE "http://localhost:9200/transactions" | ||
echo | ||
curl -s -XPUT "http://localhost:9200/transactions/" -d '{ | ||
"settings": { | ||
"index.number_of_shards": 1, | ||
"index.number_of_replicas": 0 | ||
}, | ||
"mappings": { | ||
"stock": { | ||
"properties": { | ||
"type": { | ||
"type": "string", | ||
"index": "not_analyzed" | ||
}, | ||
"amount": { | ||
"type": "long" | ||
} | ||
} | ||
} | ||
} | ||
}' | ||
echo | ||
curl -s -XPUT 'http://localhost:9200/transactions/stock/1' -d '{"type": "sale", "amount": 80}' | ||
curl -s -XPUT 'http://localhost:9200/transactions/stock/2' -d '{"type": "cost", "amount": 10}' | ||
curl -s -XPUT 'http://localhost:9200/transactions/stock/3' -d '{"type": "cost", "amount": 30}' | ||
curl -s -XPUT 'http://localhost:9200/transactions/stock/4' -d '{"type": "sale", "amount": 130}' | ||
|
||
curl -s -XPOST "http://localhost:9200/transactions/_refresh" | ||
echo | ||
curl -s -XGET "localhost:9200/transactions/stock/_search?pretty=true" -d '{ | ||
"query" : { | ||
"match_all" : {} | ||
}, | ||
"aggs": { | ||
"profit": { | ||
"scripted_metric": { | ||
"init_script" : "stockaggs_init", | ||
"map_script" : "stockaggs_map", | ||
"combine_script" : "stockaggs_combine", | ||
"reduce_script" : "stockaggs_reduce", | ||
"lang": "native" | ||
} | ||
} | ||
}, | ||
"size": 0 | ||
}' |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
/* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.elasticsearch.examples.nativescript.plugin; | ||
|
||
import org.elasticsearch.examples.nativescript.script.IsPrimeSearchScript; | ||
import org.elasticsearch.examples.nativescript.script.PopularityScoreScriptFactory; | ||
import org.elasticsearch.examples.nativescript.script.RandomSortScriptFactory; | ||
import org.elasticsearch.examples.nativescript.script.stockaggs.CombineScriptFactory; | ||
import org.elasticsearch.examples.nativescript.script.stockaggs.InitScriptFactory; | ||
import org.elasticsearch.examples.nativescript.script.stockaggs.MapScriptFactory; | ||
import org.elasticsearch.examples.nativescript.script.stockaggs.ReduceScriptFactory; | ||
import org.elasticsearch.plugins.Plugin; | ||
import org.elasticsearch.plugins.ScriptPlugin; | ||
import org.elasticsearch.script.NativeScriptFactory; | ||
|
||
import java.util.Arrays; | ||
import java.util.List; | ||
|
||
/** | ||
* This class is instantiated when Elasticsearch loads the plugin for the | ||
* first time. | ||
*/ | ||
public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin { | ||
|
||
@Override | ||
public List<NativeScriptFactory> getNativeScripts() { | ||
return Arrays.asList(new IsPrimeSearchScript.Factory(), new RandomSortScriptFactory(), new PopularityScoreScriptFactory(), | ||
// Scripted Metric Aggregations Scripts | ||
new InitScriptFactory(), new MapScriptFactory(), new CombineScriptFactory(), new ReduceScriptFactory()); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should probably be general docs we have on plugin authoring?