PLIP: assimilate collective.indexing

[MrTango]

Proposer : Gil Forcada

Seconder : Maik Derstappen

Abstract

Move all functionality on existing packages (Products.ZCatalog, Products.CMFCore and Products.CMFPlone) and apply all monkey patches directly to the packages they belong to, like Products.CMFCore or Products.Archetypes.

Motivation

Using collective.indexing has two main benefits:

• performance: indexing operations are queued, optimized and done at the transaction boundary. Only one indexing operation is done per object on any transaction. This means that plone is much faster, specially if lots of event handlers modify objects and keep reindexing the same object over and over again.
• pluggable: it creates a utility that enables third-party search indexers to be notified on any indexing operation (that's how collective.solr is plugged-in)

We think they are important and transparent enough that any Plone installation will benefit from it, specially for the performance part, although the pluggable makes even more easier to scale Plone as the site grows.

Assumptions

Some tests in Plone core will need to be updated to work around the indexing optimizations (as tests expect that objects are indexed/reindexed/unindexed right away).

No refactorings will be done on either CMFPlone CatalogTool nor CMFCore CatalogTool nor ZCatalog ZCatalog base class (which both CMFPlone and CMFCore override), this cleanup/refactoring is out of scope.

We deemed more importantly to get the functionality first in rather than merging the catalog tools together. At the same time we encourage any individual/company to step up and propose a PLIP that goes in that direction.

Proposal & Implementation

• plip config file on buildout.coredev to track the implementation and branches
• jenkins job to test the plip config
• move all functionality provided by collective.indexing to currently existing packages (Products.CMFCore and Products.CMFPlone)
• the code of collective.indexing is now GPL and should be re-licensed by the PF-Board to ZPL before getting into CMFCore which are ZPL (@plone/plone-foundation-board)
• add tests where the functionality moved
• fix tests due to the way indexing is done with collective.indexing
• documentation (describe the new feature, how to update tests, how to migrate from collective.indexing)
• collective.solr branch
• deprecation warning on collective.indexing

Deliverables

• CMFCore branch
• CMFPlone branch
• documentation on how to fix your tests and how the new functionality benefits Plone
• collective.solr branch (as is collective.indexing main consumer)
• new release of collective.indexing with deprecation warnings

Risks

Some tests of 3th party packages can fail, but are easy to fix, with the processing_queue function which will be provided.

Participants

Gil Forcada
Maik Derstappen

[tomgross]

I'm +1 for this but propose a slightly different focus:

Provide a catalog / indexing API for which ZCatalog is the default implementation and Plone still ships with it but it can be easily repleced by another indexing solution (solr, pylucene, elasticsearch, ...) following the index / query / etc. API. It reads like batteries included but easily changeable.

Additional Deliverables:

Documentation of the API and example how to pull in a custom index

[gforcada]

@tomgross our main idea is to drop the code in without major/any refactorings of any code. A follow up PLIP will be the one that can/should merge ZCatalog/CMFCore/CMFPlone code to make that a single code base without one patching the other in both directions.

Still, collective.indexing already allows external indexers (collective.solr at least) to be used.

[gforcada]

@MrTango @tomgross I updated the proposal

[gforcada]

@tomgross read this post from @hannosch back in 2010 to understand why is not possible/desirable to have a completely external catalog.

Full text searches (with faceted, and la la la), special indexes, etc etc can/should be of course made external to benefit from enterprise level search indexers (Solr, Elastic Search...).

At the same time, reducing the amount of metadata and indexes would help make things faster, see #1279 for that.