Unable to replace standard actions via Java plugin

[andreasch]

The plugin module is loaded before the actions module and therefore plugins are loaded before registering the default actions. If a custom plugin registers a custom action in place of the default ones , the custom action is removed when the action module is loaded.

For more details see:

https://groups.google.com/d/topic/elasticsearch/OFNkDUcnJGA/discussion

[wires]

This can probably be fixed by adding a check to ActionModule.registerAction(...) in the ES source

        // don't override already existing actions
        if(actions.containsKey(action.name()))
            return;

but I wonder if this is actually what you want; thought I needed this, but then I found the org.elasticsearch.index.indexing.IndexingOperationListener that allows you to hook into indexing operations.

[andreasch]

That was my solution. I introduced a property for allowing/disallowing standard action overrides. If overriding is enabled, the action is added only if it does not already exist. The IndexingOperationListener would get me most of what I needed but I also needed to override another action (PutMappingAction) and there is no listener for that one.

[kimchy]

The idea is not to allow one to override the default set of actions in elasticsearch, this is not what the extension aspect was meant to do.

[andreasch]

Is there any other way to intercept a request and change it or execute something else before executing the default action?

[wires]

I've intercepted the calls at shard level using a IndexingOperationListener, see this commit. I'm not sure if the document can be modified here, but it seems so, Kimchy?

Furthermore, Jörg commented on the forum

Thanks for sharing this. Afaik the indexing operation listener's job is to catch write/update events inside of a Lucene index within an ES shard (which is not the scope of a whole index, only a part of it), just as the translog works. Pretty much low-level stuff. I'm not sure if shard level operations are the best place for adding denormalized data to documents, I would prefer the node client level, where you have access to the field mapping and the cluster state.

So maybe he misunderstood me, because I don't want to add the attributes there, I just want to drop some attributes. I am interested in catching every document going into ES and then later, asynchronously, update the document using a regular get/put (and use MVCC to make sure it stays consistent).

But I would certainly be interested how to achieve what Jörg is talking about, having access to the field mappings, but still catch all documents (in particular ones that are pushed using rivers). Any hints?

thnx,
Jelle

[kimchy]

I am not a fan of extensions into elasticsearch that munges the data, or change "default" behavior outside of the features that are provided. You don't extend MySQL to drop some attributes, you do that on your application layer.

[wires]

I can certainly follow that reasoning, but unlike DBMS, in ES there are plugins like rivers which allow elasticsearch to "autonomously" fill itself (which is great!).

Yet is is not possible to place a function in-between a river and the indexing. Such processing is thus moved into the river: the couch river, for instance, has script filters. But, at least from my perspective, it makes sense to use this by other rivers as well. Can this not be generalised into "document modification" plugins that sit just before the indexing?

[kimchy]

@0x01 to be honest, if you end up needing to have additional functionality that the rivers does not produce, personally, I would write it outside of ES and just do what the river does. For example, with couch, its pretty simple to listen on the changes feed writing it in your favorite language, and munging the data however you want before indexing it to ES. I personally find this type of solution more manageable, then having to write a plugin "into" elasticsearch.

[wires]

I see.

Well, we get our data from various sources:

• couchdb, activemq, rss
• things pushing to solr (using solr-mock plugin)
• things pushing to ES api

For us the rivers and the mock-solr plugin provide all the functionality we need, in terms of fetching/receiving the documents.

But what I'm looking for is a central place to process these documents.

What we've already build comes down to two things:

1. extract graph information from incoming documents, preferably dropping that information before it hits indexing
2. define special "denormalized" fields, which are (asynchronously) added to the document, based on a traversal of the graph.

All configuration is pushed by the user, in a similar way as River configuration is send to /_river/, so there is little magic going on inside the plugin (it's all in de configuration).

As an alternative we could use something like the proposed changes feed (#1242) but then we cannot modify the documents. (We want to modify the documents to remove some fields that will be stored in the graph).

So at least for our situation it makes a lot of sense to run this "inside" ES, as we get a lot of functionality, essentially for free.

Some others have also expressed interest in such a plugin (mrvisser, jprante), so maybe we should move this discussion to the mailing list?

bye,
Jelle.