logstash 1.5.4: mongodb output plugin support for update with $set operator

[stevecalvert]

No description provided.

[stevecalvert]

Hi all, I'm using the HTTP input plugin to receive HTTP POST/PATCH requests, and would like to map the PATCH in the mongodb output plugin to an update on an existing document based on the $set operator. I'm not familiar with Ruby, but checking the latest mongodb.rb, it looks like only insert_one() is used. Are there any plans to support update_one() or other mechanisms to update an existing Mongo document?

[guyboertje]

@stevecalvert - In general the LS outputs are append only. Consider this blog article from LinkedIn. You turn your collection into a journal showing how the document changed over time.

[jordansissel]

@guyboertje on the other hand, we support action of delete, insert, create, update --- for the elasticsearch output. I'm open to doing so for the mongodb output; in which case we'll need to add another setting to let users set whatever the equivalent to the 'document_id' setting would be.

[guyboertje]

@stevecalvert @jordansissel - in Mongo we could use update with the upsert option. https://docs.mongodb.org/manual/reference/method/db.collection.update/#upsert-parameter
Obviously driven by config options, to allow the user to specify which field contains the query key for the upsert. e.g.

config :upsert_query_key, :validate => :string, :default => "_id"
config :upsert, :validate => :boolean, :default => false.

[sdcharle]

Has this been ruled out as something that will be added? I'm needing upsert functionality for a project I'm working on so I'm evaluating options. This plugin works great but I need some way to handle a record getting sent thru with an update....

[joelpermanas]

Hi, Is there any update on this issue ?

[jsvd]

@joelpermanas any update will be publicly visible here. PR contributions are welcome :)

[devchristianschroeder]

I would like to have this feature also, please.

Thank you for the great product.

[colinsurprenant]

relates to #26 and #54

[nilskuhn]

Did anybody start with an implementation or plans to do so?

[nilskuhn]

I am implementing this and I plan to use these config flags:

config :action, :validate => :string, :required => true
config :query_key, :validate => :string, :required => false, :default => "_id"
config :query_value, :validate => :string, :required => false
config :upsert, :validate => :boolean, :required => false, :default => false

Where action may be one of insert (the default, behaves like plugin does now), update or replace. For update and replace, query to find document to update/replace is built like this:

:filter => {@query_key => @query_value}

Where _id would be the default value for @query_key config setting. By setting upsert to true one could enable upsert mode for update/replace. For insert this setting wouldn't have any effect.
I tried to align naming of config settings to elasticsearch output but I think, at least for me, upsert is much more clear than doc_as_upsert.

Any comments? Suggestions?

[nilskuhn]

I opened a PR for this (see above). Could anybody please take a look and let me know whether my changes make sense for you, too?
It's working for us so far.

[21stcaveman]

has there been any traction on this? 7+ years open, PR request pending almost 3 years now...

[AndreaFranchini]

Up!
I'm also interested in this, it's very limiting the absence of the mongo upsert in the output plugin.