FieldIndex value of None should be unindexed, not just ignored

[ewohnlich]

Reported here first plone/Products.CMFPlone#3064

There is a note here from @hannosch that a value of None should not be indexed because of limitations in BTrees 4.0+. That makes sense, but if the value has changed from something to None I think it would make sense to also remove the old value if one exists.

My original case was on a Plone site:

1. Plone object "pub1" with attribute "pmid" set to "1"
2. index with FieldIndex for "pmid"
3. Assign value of None to pub1.pmid, reindex

The result is pub1 is still found for a query on pmid="1"

[jamadden]

That comment is no longer valid with BTrees >= 4.4.0 (2017-01-11). Those versions again allow None.

[ewohnlich]

I was about to submit a PR but given your comment, perhaps this condition should just be removed entirely?

[d-maurer]

Eric Wohnlich wrote at 2020-5-5 13:44 -0700:

Reported here first plone/Products.CMFPlone#3064 There is a note [here](https://github.com/zopefoundation/Products.ZCatalog/blob/5.1/src/Products/PluginIndexes/unindex.py#L253) from @hannosch that a value of None should not be indexed because of limitations in BTrees 4.0+.

This `BTrees` restriction has meanwhile be removed. Nevertheless, I think it a good idea to interpret "None" as "no value" (and do not index it in the normal way; @andbag has worked on indexing it as a "special value").

That makes sense, but if the value has changed from _something_ to None I think it would make sense to also remove the old value if one exists.

I agree with you. This is a bug.

[ewohnlich]

I will fix up that branch as submit as a PR along those lines then.

Also, I would like to hear more about @andbag's work on that if someone could direct me to it. I have a use case where users need to be able to search for content that have a None value (or some other python False value like an empty string). To do that I am currently using Plone indexers that assign some magic value in those cases. It would be much better if the Index itself could handle this natively. But this need to be connected with a fix here by any means.

[d-maurer]

Eric Wohnlich wrote at 2020-5-5 14:52 -0700:

Also, I would like to hear more about @andbag's work on that if someone could direct me to it.

"#74"

[wesleybl]

Fixed in: #103