Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deleted documents show up in completion suggester #7761

Closed
ankitkr opened this issue Sep 17, 2014 · 15 comments
Closed

Deleted documents show up in completion suggester #7761

ankitkr opened this issue Sep 17, 2014 · 15 comments

Comments

@ankitkr
Copy link

@ankitkr ankitkr commented Sep 17, 2014

Indexed the docs.
Used suggester
Deleted one doc
Used suggester (deleted doc still shows)
Used optimize
Used suggester (deleted doc still shows)
Used refresh
Used suggester (deleted doc still shows)

If the number of docs is less then everything works fine but when the number increases, the completion suggester shows false result. The test case can be found at
https://gist.github.com/ankitkr/e7776419b0e8ff5334af

https://gist.github.com/missinglink/01c7db8bb09f2f935574

@areek
Copy link
Contributor

@areek areek commented Sep 18, 2014

This is a known issue for the current completion suggester. There has been progress made to fix this, along with other enhancements/imporvements, but the support for the new completion suggester implementation in ES is still not in place. (see #7353, #7650)

@s1monw
Copy link
Contributor

@s1monw s1monw commented Sep 18, 2014

it's not a bug it's a feature at this point. We are working on fixes - closing.

@s1monw s1monw closed this Sep 18, 2014
@tvdavid
Copy link

@tvdavid tvdavid commented Oct 3, 2014

Should the status of this bug perhaps be reflected more clearly in the completion suggester's documentation? It currently reads The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that, while in reality it's very easy to get deleted documents stuck in the index with no way at all to purge them out.

@cari-cm
Copy link

@cari-cm cari-cm commented Apr 14, 2015

How can you say it's a feature and then say you are working on fixes? Admit it's a bug. It would be really nice if there was a fix to this BUG prior to 2.0!

@gitazem
Copy link

@gitazem gitazem commented Sep 23, 2015

What situation is on this bug ? How can i solve it ? I am trying many workarounds -- and nothing. this is still don't work "_optimize?only_expunge_deletes=true&max_num_segments=1&force=true&refresh=true"
any suggestions ?

@clintongormley
Copy link
Contributor

@clintongormley clintongormley commented Sep 23, 2015

@gitazem the new completion suggester coming in 2.1 solves this problem

@chape
Copy link

@chape chape commented Jan 22, 2016

@clintongormley Have 2.1.1 fixed it problem yet? I don't see it in 2.1.0&2.1.1 release notes.

@clintongormley
Copy link
Contributor

@clintongormley clintongormley commented Jan 22, 2016

It was bumped to 2.2.0, but now we've just bumped it to 3.0 because it has breaking changes, unfortunately.

@chape
Copy link

@chape chape commented Jan 25, 2016

@clintongormley thank you for response

@drush
Copy link

@drush drush commented Feb 19, 2016

@clintongormley Can you re-open this issue as a bug, since it is in fact OPEN, and a BUG? kthxbye

@majksner
Copy link

@majksner majksner commented Mar 17, 2016

This issue is still actual. I'm using 2.2.1

@s1monw
Copy link
Contributor

@s1monw s1monw commented Mar 17, 2016

@drush it's not a bug always advertised as such
@majksner it will not come until 5.0

@drush
Copy link

@drush drush commented Mar 19, 2016

It is absolutely a bug @s1monw Per your docs on completion suggesters:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
"NOTE: The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that. You can call optimize with the only_expunge_deletes=true to only target deletions for merging."

DELETES do NOT work on suggesters even with this optimize operation. This suggestion does not in-fact solve the problem, so the bug is that it is NOT possible to delete documents without destroying the entire index.

If you are suggesting that you support a DELETE operation on suggester indexes but that the DELETE operation is never expected to work, then you need to demote this entire feature to 'Experimental' or 'Alpha' rather than a production capability.

The documentation is lacking at best and deceiving at worst. I would suggest re-evaluating this issue for proper prioritization.

@s1monw
Copy link
Contributor

@s1monw s1monw commented Mar 19, 2016

@drush I take these complaints serious so I added a test for your that makes sure the feature works as documented. (5fa5438)

The downside of the implementation (which me and @mikemccand basically wrote years ago) is that it's a very optimized data-structure and the enhancements that we are shipping in the next major version will resolve this issue.

Believe me I see a lot of people that try to be loud on issues like this just like you try to. That's fine, I can't change that. What I can change is to let you know that it doesn't help, it is counter productive. You don't like the progress over perfection approach we are practicing! This feature has already as it is provided huge improvements do large scale suggest applications, it's not perfect as documented.
We are shipping the improvement that you may need in the next major, back-porting is not an option.

Again, you may be heard in your environment when you are loud, in this environment you may just be shown wrong and we move on.

@s1monw
Copy link
Contributor

@s1monw s1monw commented Mar 21, 2016

@drush Just FYI I tried to clarify the documentation and added a possible caveat to the documentation.

s1monw added a commit to s1monw/elasticsearch that referenced this issue Mar 21, 2016
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to elastic#7761
s1monw added a commit that referenced this issue Mar 21, 2016
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761
s1monw added a commit that referenced this issue Mar 21, 2016
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761
s1monw added a commit that referenced this issue Mar 21, 2016
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761
s1monw added a commit that referenced this issue Mar 21, 2016
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761
s1monw added a commit that referenced this issue Mar 21, 2016
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants