Need INSERT IGNORE support

[webigorkiev]

Hello
Is it possible to add support for INSERT IGNORE?

In my case, the index is constantly updated in real time, but sometimes full sync is needed.

The problem is that at the time of bulk inserting (After TRUNCATE) - another processes insert records and I get a duplicate id error

Thanks

[sanikolaev]

Are you sure you want INSERT IGNORE and not REPLACE?

[webigorkiev]

Replace is the first thing I thought of.
But, when sync, I already have all the index data in my memory, and new data arrives in real time.
Replace will overwrite new data with stale data.

The possibility of a bulk insertion of data is needed - without changing existing ones.

[sanikolaev]

Got it. We'll discuss it. Perhaps it's a low hanging fruit that we can easily implement, then we'll include it into the next release.

[githubmanticore]

➤ Sergey Nikolaev commented:

Perhaps it's a low hanging fruit that we can easily implement

After discussion with the core team it turned out to be quite complex task, so I'm afraid we can't prioritize this.

[scruel]

➤ Sergey Nikolaev commented:

Perhaps it's a low hanging fruit that we can easily implement

After discussion with the core team it turned out to be quite complex task, so I'm afraid we can't prioritize this.

How about have it in bulk?

[sanikolaev]

Things have changed since 2022 and now we have a tool to solve this task easily - Manticore Buddy. PRs are welcome. Here's a blog post about it https://manticoresearch.com/blog/manticoresearch-buddy-pluggable-design/

[djklim87]

After our discussion, we have decided that this feature cannot be implemented in Buddy. Potentially, we could encounter large batches of inserts, and parsing that query within Buddy would be a very bad idea as it could significantly slow down the plugin and the entire Buddy extension. Instead, it should be implemented in the searchd daemon.