Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate elasticsearch native script examples to the main repo #19334

Closed
Show file tree
Hide file tree
Changes from 71 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
fc57847
first commit
imotov Feb 12, 2013
4dc7b46
Fix link in README
imotov Feb 12, 2013
4e8d053
Upgrade to elasticsearch 0.20.5
imotov Feb 17, 2013
b15f57f
Add check for null parameters
imotov Feb 17, 2013
ecb5cc5
Add an example of injecting client into native script
imotov Feb 17, 2013
418c4c8
Add license
imotov Feb 17, 2013
9525fa5
Add Lookup Script description
imotov Feb 24, 2013
c9629a1
Make sure that the local shard will be used for the lookup operation …
imotov Feb 24, 2013
25b1565
Add random sort order native script
imotov Feb 25, 2013
92c4950
Add an example of using logger in a native script
imotov Feb 26, 2013
f90c8b0
Add a curl example for lookup script
imotov Feb 26, 2013
cb8fcc5
Add random sort script description
imotov Feb 28, 2013
9ab9fb1
Add a curl example for random sort script
imotov Mar 1, 2013
211f267
Make sure that standard naming convention is used for plugin file
imotov Mar 2, 2013
2b184f3
Release v1.0.0
imotov Mar 2, 2013
e5d10c7
Move to 1.1.0-SNAPSHOT
imotov Mar 2, 2013
210aab0
Release v1.1.0
imotov Mar 2, 2013
574fd62
Move to 1.1.0-SNAPSHOT
imotov Mar 2, 2013
803337a
Change node settings to avoid interference with other clusters
imotov Mar 5, 2013
d12bfa7
Add example of a custom score script
imotov Mar 5, 2013
f280b76
Small cosmetic changes
imotov Mar 14, 2013
a7f8b4d
Fix typo in index settings
imotov Mar 15, 2013
1e2da9c
Add an example of popularity script
imotov Mar 15, 2013
eed5d53
Upgrade to elasticsearch v0.90.5
imotov Oct 25, 2013
1e3c365
Function Score Query
Nov 18, 2013
0420b7a
Fix elasticsearch.yml
imotov Nov 19, 2013
d472879
use elasticsearch version 0.90.8 and move from testNG to junit
brwe Dec 12, 2013
035dc8f
fix doc values lookup
brwe Dec 18, 2013
4d774aa
fix popularity test
brwe Dec 18, 2013
a6db7f9
Add -s flag to curl requests in the sample scripts
imotov Dec 18, 2013
5604d3d
add terms based scoring scripts
brwe Dec 19, 2013
3e22997
Add -s flag to curl requests in the termscoring script
imotov Jan 16, 2014
29975b7
Fix LookupScriptTests mappings
hmalphettes May 24, 2014
f365998
Upgrade to ES-1.2.0
hmalphettes May 24, 2014
93d8b81
fix number of shards settings
brwe May 30, 2014
c5ed8ee
Upgrade to elasticsearch 1.3.4
imotov Oct 16, 2014
1254c48
Upgrade to elasticsearch 1.4.0
imotov Nov 8, 2014
9406941
fix links in README
brwe Mar 26, 2015
d6a8b87
doc: remove not exist import.
ishare Apr 27, 2015
144e23f
Merge pull request #10 from ishare/patch-1
imotov Jun 15, 2015
6ccdf8c
Update to 2.0.0-SNAPSHOT
imotov Jun 14, 2015
54e3080
Use BigInteger.valueOf, not constructor
pickypg Jun 19, 2015
089f4c9
Update to 2.0.0-beta1-SNAPSHOT
imotov Jul 17, 2015
96f8b93
Update the way plugin is loaded in the test
imotov Jul 23, 2015
d12279d
Switch to elasticsearch-plugin as a parent project
imotov Jul 28, 2015
6c4ba4a
Rename test classes
imotov Aug 4, 2015
4cda2cc
Scripts should expose whether they use the `_score` or not
imotov Aug 13, 2015
f5feb9e
Add licenses directory to make license checker happy
imotov Aug 13, 2015
b95943e
Add an example of using scripts in the scripted metrics aggs
imotov Aug 14, 2015
2b87eb9
Changes related to renaming of AbstractPlugin and parent pom
imotov Aug 20, 2015
8881606
Update README to reflect changes in 2.0
imotov Aug 21, 2015
a19cdcd
Fixing glitches in readme
Aug 22, 2015
0f4152f
Merge pull request #14 from YannBrrd/patch-1
imotov Aug 22, 2015
e41024a
Update to 2.1.0-SNAPSHOT
imotov Aug 28, 2015
089b01c
Switch to the direct way of injecting plugins into test nodes
imotov Aug 31, 2015
4a2debe
Update README to include branches that should be used for different v…
imotov Aug 31, 2015
2ba2cd6
Update the test case name to follow the new test naming convention
imotov Sep 3, 2015
89f2009
Update to 3.0.0-SNAPSHOT
imotov Sep 11, 2015
b0efbbb
Remove reference to Guava Charsets
imotov Sep 16, 2015
db6c568
Move rest-api-spec tests to resources dirs
imotov Sep 17, 2015
548c090
Update code to work with the latest 3.0 snapshot
imotov Sep 29, 2015
375b7f7
Remove guava cache
imotov Oct 10, 2015
db8281b
Switch from array to arraylist since the params are no longer re-pars…
imotov Oct 16, 2015
6ae94c7
Add a simple way to start elasticsearch with plugin installed from IDE
imotov Oct 16, 2015
ff3c1aa
Get rid of ArrayList conversion
imotov Oct 16, 2015
5b22950
Get rid of ArrayList conversion in PhraseScoreScript
imotov Oct 17, 2015
c418732
Remove Test annotation
imotov Oct 21, 2015
8aa2eb4
Move to 3.0.0-SNAPSHOT
imotov Jan 5, 2016
3a42192
Move native script examples to the plugins directory
imotov Jul 7, 2016
4cdbdf7
Merge branch 'migrate-elasticsearch-native-script-example-to-main-repo'
imotov Jul 7, 2016
7df2d06
Update to work with the latest master
imotov Jul 7, 2016
3c7ea4c
Remove all scripts that demonstrate use of indexLookup()
imotov Jul 18, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
83 changes: 83 additions & 0 deletions plugins/native-script-example/README.textile
@@ -0,0 +1,83 @@
h1. Example of Native Script Plugin for Elasticsearch

h2. Introduction

p. This plugin contains several examples of "native script":http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html for Elasticsearch.

h2. Creating Elasticsearch Plugin

p. The simplest way to deploy native script is by wrapping it into standard Elasticsearch plugin infrastructure. An Elasticsearch plugin can be written in java and built using gradle. A typical plugin source directory looks like this:

bc.. .
|- build.gradle
|- src
|- main
| |- java
| | |- ... source code ...
|- test
|- java
| |- ... source code ...
|- resources
|- ... test resources ...


p. An Elasticsearch plugin can be created by following these six steps.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be general docs we have on plugin authoring?


* Create build.gradle file in the root directory of your plugin.
* Create source code directories:
** @mkdir -p src/main/java@
** @mkdir -p src/test/java@
** @mkdir -p src/test/resources@
* The gradle plugin @esplugin@ provides all needed tasks to assemble the plugin. It is using the following settings that should be modified to match the project's plugin class name and license file definition.

bc.. esplugin {
description 'ElasticSearch Plugin with Native Script Examples.'
classname 'org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin'
}

p.

* Create main Plugin class in the @src/main/java@ directory. This project is using @org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin@ class as an example, so the it has to be saved as @src/main/java/org/elasticsearch/examples/nativescript/plugin/NativeScriptExamplesPlugin.java@

bc.. package org.elasticsearch.examples.nativescript.plugin;

import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.script.ScriptModule;

public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin {
/* .... */
}

p.

* The plugin can be built using @gradle build@ command and executed within elasticsearch by using @gradle run@.

h2. Adding Native Scripts

p. Now that the plugin infrastructure is complete, it's possible to add a native script.

h3. Is Prime Native Script

p. One of the example scripts in this project is the "is_prime" script that can be used to check if a field contains a possible prime number. The script accepts two parameters @field@ and @certainty@. The @field@ parameter contains the name of the field that needs to be checked and the @certainty@ parameter specifies a measure of the uncertainty that the caller is willing to tolerate. The script returns @true@ if the field contains a probable prime number and @false@ otherwise. The probability that the number for which the script returned @true@ is prime exceeds (1 - 0.5^certainty). The script can be used in "Script Filter":http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html as well as a "Script Field":http://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html. The implementation of the "is_prime" native script and it's factory can be found in the "IsPrimeSearchScript":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/IsPrimeSearchScript.java class.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to have documentation about the examples on the script classes themselves?


p. In order to enable native script creation the plugin has to contain and register a class that implements "NativeScriptFactory":https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/NativeScriptFactory.java. The NativeScriptFactory interface has only one method @newScript(Map<String, Object> params)@. This method accepts a list of parameters that are passed during script creation and returns an ExecutableScript object with an instance of the script. The factory has to be returned in the @onModule(ScriptModule module)@ method of the Plugin.

bc.. public class NativeScriptExamplesPlugin extends Plugin implements ScriptPlugin {

@Override
public List<NativeScriptFactory> getNativeScripts() {
return Arrays.asList(new IsPrimeSearchScript.Factory());
}
}

p. In general native scripts have to implement the interface "ExecutableScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/ExecutableScript.java, but if they are used in search, they have to also implement the "SearchScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/SearchScript.java interface. The SearchScript interface is quite complex, so in order to simplify implementation, the native script can simply extend the "AbstractSearchScript":https://github.com/elasticsearch/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/script/AbstractSearchScript.java class instead. The AbstractSearchScript has only one abstract method @run()@. During search Elasticsearch calls this method for every single record in the search result. As in case of non-native script, the content of the current record can be accessed using DocLookup (@doc()@ method), FieldsLookup (@fields()@ method), SourceLookup (@source()@ method).

h3. Lookup Script

p. The "lookup script":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/LookupScript.java demonstrates how to make elsticsearch client available within the native script. When script factory is initially created, the node is not fully initialized yet. Because of this client cannot be directory injected into the factory. Instead, the reference to the node is injected and the client is obtained from the node during script creation. A same mechanism can be used to obtain other node components through node injector.

h3. Random Sort Script

p. The "random sort script":https://github.com/elastic/elasticsearch/blob/master/plugins/native-script-example/src/main/java/org/elasticsearch/examples/nativescript/script/RandomSortScriptFactory.java demonstrates a slightly different approach to script/factory packaging. In this case the factory is the outer class which creates one inner script or another based on the input parameters. If the parameter @salt@ is present, the script is calculating hash value of @id + salt@ instead of generation random sort values. As a result, for any value of @salt@ the order of the records will appear random, but this order will be repeatable and therefore this approach would be more suitable for paging through result list than a completely random approach.


15 changes: 15 additions & 0 deletions plugins/native-script-example/build.gradle
@@ -0,0 +1,15 @@
esplugin {
description 'Demonstrates writtting elasticsearch plugins with native scripts'
classname 'org.elasticsearch.examples.nativescript.plugin.NativeScriptExamplesPlugin'
}

compileJava.options.compilerArgs << "-Xlint:-deprecation"

// uncomment to disable 3rd party audit
// thirdPartyAudit.enabled = false

// uncomment to disable forbidden patterns checks
// forbiddenPatterns.enabled = false

// uncomment to disable license headers checks
// licenseHeaders.enabled = false
47 changes: 47 additions & 0 deletions plugins/native-script-example/example/popularity.sh
@@ -0,0 +1,47 @@
#!/bin/sh
curl -s -XDELETE "http://localhost:9200/test"
echo
curl -s -XPUT "http://localhost:9200/test/" -d '{
"settings": {
"index.number_of_shards": 1,
"index.number_of_replicas": 0
},
"mappings": {
"type1": {
"properties": {
"name": {
"type": "string"
},
"number": {
"type": "integer"
}
}
}
}
}'
echo
curl -s -XPUT "localhost:9200/test/type1/1" -d '{"name" : "foo bar baz", "number": 10000 }'
curl -s -XPUT "localhost:9200/test/type1/2" -d '{"name" : "foo foo foo", "number": 1 }'
curl -s -XPOST "http://localhost:9200/test/_refresh"
echo
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"function_score" : {
"boost_mode": "replace",
"query": {
"match": {
"name": "foo"
}
},
"script_score": {
"script": "popularity",
"lang": "native",
"params": {
"field": "number"
}
}
}
}
}
'
echo
42 changes: 42 additions & 0 deletions plugins/native-script-example/example/primes.sh
@@ -0,0 +1,42 @@
#!/bin/sh
curl -s -XDELETE "http://localhost:9200/test"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of having these shell tests, I think we should show how to write unit tests?

echo
curl -s -XPUT "http://localhost:9200/test/" -d '{
"settings": {
"index.number_of_shards": 1,
"index.number_of_replicas": 0
},
"mappings": {
"type1": {
"properties": {
"name": {
"type": "string"
},
"number": {
"type": "integer"
}
}
}
}
}'
echo
for i in {0..100}; do curl -s -XPUT "localhost:9200/test/type1/$i" -d "{\"name\":\"rec $i\", \"number\":$i}"; done
curl -s -XPOST "http://localhost:9200/test/_refresh"
echo
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"bool": {
"filter": {
"script": {
"script": "is_prime",
"lang": "native",
"params": {
"field": "number"
}
}
}
}
}
}
'

105 changes: 105 additions & 0 deletions plugins/native-script-example/example/random.sh
@@ -0,0 +1,105 @@
#!/bin/sh
curl -s -XDELETE "http://localhost:9200/test"
echo
curl -s -XPUT "http://localhost:9200/test/" -d '{
"settings": {
"index.number_of_shards": 1,
"index.number_of_replicas": 0
},
"mappings": {
"type1": {
"properties": {
"name": {
"type": "string"
},
"number": {
"type": "integer"
}
}
}
}
}'
echo
for i in {0..100}; do curl -s -XPUT "localhost:9200/test/type1/$i" -d "{\"name\":\"rec $i\", \"number\":$i}"; done
curl -s -XPOST "http://localhost:9200/test/_refresh"
echo
echo "Salt 123"
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"script": "random",
"lang": "native",
"type": "number",
"params": {
"salt": "123"
}
}
}
}
' | grep \"_id\"
echo "Salt 123"
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"script": "random",
"lang": "native",
"type": "number",
"params": {
"salt": "123"
}
}
}
}
' | grep \"_id\"
echo "Salt 124"
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"script": "random",
"lang": "native",
"type": "number",
"params": {
"salt": "124"
}
}
}
}
' | grep \"_id\"
echo "No salt"
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"script": "random",
"lang": "native",
"type": "number"
}
}
}
' | grep \"_id\"
echo "No salt"
curl -s "localhost:9200/test/type1/_search?pretty=true" -d '{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"script": "random",
"lang": "native",
"type": "number"
}
}
}
' | grep \"_id\"
echo
47 changes: 47 additions & 0 deletions plugins/native-script-example/example/stockaggs.sh
@@ -0,0 +1,47 @@
#!/bin/sh
curl -s -XDELETE "http://localhost:9200/transactions"
echo
curl -s -XPUT "http://localhost:9200/transactions/" -d '{
"settings": {
"index.number_of_shards": 1,
"index.number_of_replicas": 0
},
"mappings": {
"stock": {
"properties": {
"type": {
"type": "string",
"index": "not_analyzed"
},
"amount": {
"type": "long"
}
}
}
}
}'
echo
curl -s -XPUT 'http://localhost:9200/transactions/stock/1' -d '{"type": "sale", "amount": 80}'
curl -s -XPUT 'http://localhost:9200/transactions/stock/2' -d '{"type": "cost", "amount": 10}'
curl -s -XPUT 'http://localhost:9200/transactions/stock/3' -d '{"type": "cost", "amount": 30}'
curl -s -XPUT 'http://localhost:9200/transactions/stock/4' -d '{"type": "sale", "amount": 130}'

curl -s -XPOST "http://localhost:9200/transactions/_refresh"
echo
curl -s -XGET "localhost:9200/transactions/stock/_search?pretty=true" -d '{
"query" : {
"match_all" : {}
},
"aggs": {
"profit": {
"scripted_metric": {
"init_script" : "stockaggs_init",
"map_script" : "stockaggs_map",
"combine_script" : "stockaggs_combine",
"reduce_script" : "stockaggs_reduce",
"lang": "native"
}
}
},
"size": 0
}'