New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deleted documents show up in completion suggester #7761

Closed
ankitkr opened this Issue Sep 17, 2014 · 15 comments

Comments

Projects
None yet
10 participants
@ankitkr
Copy link

ankitkr commented Sep 17, 2014

Indexed the docs.
Used suggester
Deleted one doc
Used suggester (deleted doc still shows)
Used optimize
Used suggester (deleted doc still shows)
Used refresh
Used suggester (deleted doc still shows)

If the number of docs is less then everything works fine but when the number increases, the completion suggester shows false result. The test case can be found at
https://gist.github.com/ankitkr/e7776419b0e8ff5334af

https://gist.github.com/missinglink/01c7db8bb09f2f935574

@areek

This comment has been minimized.

Copy link
Contributor

areek commented Sep 18, 2014

This is a known issue for the current completion suggester. There has been progress made to fix this, along with other enhancements/imporvements, but the support for the new completion suggester implementation in ES is still not in place. (see #7353, #7650)

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Sep 18, 2014

it's not a bug it's a feature at this point. We are working on fixes - closing.

@s1monw s1monw closed this Sep 18, 2014

@tvdavid

This comment has been minimized.

Copy link

tvdavid commented Oct 3, 2014

Should the status of this bug perhaps be reflected more clearly in the completion suggester's documentation? It currently reads The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that, while in reality it's very easy to get deleted documents stuck in the index with no way at all to purge them out.

@cari-cm

This comment has been minimized.

Copy link

cari-cm commented Apr 14, 2015

How can you say it's a feature and then say you are working on fixes? Admit it's a bug. It would be really nice if there was a fix to this BUG prior to 2.0!

@gitazem

This comment has been minimized.

Copy link

gitazem commented Sep 23, 2015

What situation is on this bug ? How can i solve it ? I am trying many workarounds -- and nothing. this is still don't work "_optimize?only_expunge_deletes=true&max_num_segments=1&force=true&refresh=true"
any suggestions ?

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Sep 23, 2015

@gitazem the new completion suggester coming in 2.1 solves this problem

@chape

This comment has been minimized.

Copy link

chape commented Jan 22, 2016

@clintongormley Have 2.1.1 fixed it problem yet? I don't see it in 2.1.0&2.1.1 release notes.

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Jan 22, 2016

It was bumped to 2.2.0, but now we've just bumped it to 3.0 because it has breaking changes, unfortunately.

@chape

This comment has been minimized.

Copy link

chape commented Jan 25, 2016

@clintongormley thank you for response

@drush

This comment has been minimized.

Copy link

drush commented Feb 19, 2016

@clintongormley Can you re-open this issue as a bug, since it is in fact OPEN, and a BUG? kthxbye

@majksner

This comment has been minimized.

Copy link

majksner commented Mar 17, 2016

This issue is still actual. I'm using 2.2.1

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Mar 17, 2016

@drush it's not a bug always advertised as such
@majksner it will not come until 5.0

@drush

This comment has been minimized.

Copy link

drush commented Mar 19, 2016

It is absolutely a bug @s1monw Per your docs on completion suggesters:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
"NOTE: The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that. You can call optimize with the only_expunge_deletes=true to only target deletions for merging."

DELETES do NOT work on suggesters even with this optimize operation. This suggestion does not in-fact solve the problem, so the bug is that it is NOT possible to delete documents without destroying the entire index.

If you are suggesting that you support a DELETE operation on suggester indexes but that the DELETE operation is never expected to work, then you need to demote this entire feature to 'Experimental' or 'Alpha' rather than a production capability.

The documentation is lacking at best and deceiving at worst. I would suggest re-evaluating this issue for proper prioritization.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Mar 19, 2016

@drush I take these complaints serious so I added a test for your that makes sure the feature works as documented. (5fa5438)

The downside of the implementation (which me and @mikemccand basically wrote years ago) is that it's a very optimized data-structure and the enhancements that we are shipping in the next major version will resolve this issue.

Believe me I see a lot of people that try to be loud on issues like this just like you try to. That's fine, I can't change that. What I can change is to let you know that it doesn't help, it is counter productive. You don't like the progress over perfection approach we are practicing! This feature has already as it is provided huge improvements do large scale suggest applications, it's not perfect as documented.
We are shipping the improvement that you may need in the next major, back-porting is not an option.

Again, you may be heard in your environment when you are loud, in this environment you may just be shown wrong and we move on.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Mar 21, 2016

@drush Just FYI I tried to clarify the documentation and added a possible caveat to the documentation.

s1monw added a commit to s1monw/elasticsearch that referenced this issue Mar 21, 2016

Clarify expunge deletes option on completion suggester
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to elastic#7761

s1monw added a commit that referenced this issue Mar 21, 2016

Clarify expunge deletes option on completion suggester
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761

s1monw added a commit that referenced this issue Mar 21, 2016

Clarify expunge deletes option on completion suggester
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761

s1monw added a commit that referenced this issue Mar 21, 2016

Clarify expunge deletes option on completion suggester
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761

s1monw added a commit that referenced this issue Mar 21, 2016

Clarify expunge deletes option on completion suggester
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761

s1monw added a commit that referenced this issue Mar 21, 2016

Clarify expunge deletes option on completion suggester
It's confusing that we advertise `expunge deletes` as an option
without clarifying it's relationship to the merge policy. By default
segments with less than `10%` deleted docs will not be merged / selected
for merge. This means, deletes don't _go away_ in the suggester.

Relates to #7761
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment