Use of Groovy scripting in ElasticSearch indexing backend limits indexed property size #154
Description
Groovy's maximum string size is 65535 characters. Because a mutation to an ES-indexed property sets that property via a Groovy script, the effective limit on ES-indexed properties is 65535 characters.
ERROR 2017-03-07T15:08:39.485-08:00 [Grizzly(5)] com.thinkaurelius.titan.diskstorage.es.ElasticSearchIndex - Failed to execute ES query ElasticsearchIllegalArgumentException[failed to execute script]; nested: GroovyScriptCompilationException[MultipleCompilationErrorsException[startup failed:
69b79da3909d74223752de333112de783b510858: 1: String too long. The given string is 994823 Unicode code units long, but only a maximum of 65535 is allowed.
@ line 1, column 27.
ctx._source["Document"] = "The Project Gutenberg EBook of Dracula...
This is especially unfortunate because no ES script is involved in creating a new property. So I have no problem indexing entire Project Gutenberg books in ES, as long as it's a new property and not an existing property.
I tried the workaround of deleting the property and re-adding it but this is still (apparently) a mutation to ElasticSearch.
FWIW, ElasticSearch dynamic scripting is also considered by some to be a security issue and is now off by default. It would be nice if this wasn't a requirement for JanusGraph. (As far as I can tell from the code, it is important for SET and LIST data types but is only used for SINGLE cardinality properties as a matter of convenience.)
(I observed this problem in Titan 1.0 but as far as I can tell it still applies to JanusGraph, including #79 .)