Merge branch 'master' into plone5-kitconcept

Conflicts:
	src/collective/solr/dispatcher.py
	src/collective/solr/exportimport.py
	src/collective/solr/tests/test_exportimport.py

CHANGES.rst

@@ -1,10 +1,47 @@
 Changelog
 =========
 
-4.1.1 (unreleased)
+5.0.4 (unreleased)
 ------------------
 
-Note: This release requires you to to update your Solr config and do a full reindex.
+- Nothing changed yet.
+
+
+5.0.3 (2016-06-05)
+------------------
+
+- Fix Pypi page.
+  [timo]
+
+
+5.0.2 (2016-06-04)
+------------------
+
+- Fix README formatting.
+  [timo]
+
+
+5.0.1 (2016-06-04)
+------------------
+
+- Fix BlobError occuring when indexing new files (fixes #120)
+  [tomgross]
+
+- Make extracting text from binary content and indexing 2 steps (#65)
+  [tomgross]
+
+- Make suggest search work when entering multiple search terms.
+  [jcharra]
+
+- Fix field-list export.
+  [gforcada]
+
+
+5.0 (2016-04-13)
+----------------
+
+Note: This release requires you to to update your Solr config and do a full reindex. Make sure you add "updateLog = true" to your "solr-instance"
+buildout section. See https://github.com/collective/collective.solr/blob/master/solr.cfg for a working example.
 
 - Ported atomic updates from ftw.solr.
   This requires you to update your solr config, load the new solr config and

README.rst

@@ -62,17 +62,13 @@ Search Enhancements
 * Exclude from search
 * Elevation
 
-ZCTextIndex Replacement
-***********************
-
-
 
 Detailed Documentation
 ======================
 
-A full Documentation of the Solr integration of Plone could be found in `docs.plone.org collective.solr`_.
+A full Documentation of the Solr integration of Plone could be found on `collectivesolr.readthedocs.org`_.
 
-.. _`docs.plone.org collective.solr`: http://docs.plone.org/external/collective.solr/docs/index.html
+.. _`collectivesolr.readthedocs.org`: http://collectivesolr.readthedocs.org/en/latest/
 
 
 Installation & Configuration
@@ -86,16 +82,26 @@ Download the latest default Solr configuration from github::
 .. note: Please do not extend your buildout directly with those files since they are likely to change over time. Always fetch the files via wget to have a stable local copy. In general you should never rely on extending buildout config files from servers that aren't under your control.
 
 Extend your buildout to use those files and make sure collective.solr is added
-to the eggs in your instance section::
+to the eggs in your instance section. Your full buildout file should look
+something like this::
 
   [buildout]
+  parts += instance
   extends =
-    solr.cfg
-    solr-4.10.x.cfg
+      http://dist.plone.org/release/4.3.8/versions.cfg
+      solr.cfg
+      solr-4.10.x.cfg
 
   [instance]
-  eggs += collective.solr
-
+  recipe = plone.recipe.zope2instance
+  http-address = 8080
+  user = admin:admin
+  eggs =
+      Plone
+      collective.solr
+
+  [versions]
+  collective.recipe.solrinstance = 5.3.2
 
 After saving this to let's say ``buildout.cfg`` the buildout can be run and the `Solr`_ server and `Plone`_ instance started::
 

base.cfg

@@ -9,6 +9,7 @@ parts +=
     code-analysis
     releaser
     sphinxbuilder
+    check-readme
 develop = .
 
 auto-checkout = *
@@ -76,3 +77,11 @@ eggs = zest.releaser
 recipe = collective.recipe.sphinxbuilder
 source = ${buildout:directory}/docs
 build = ${buildout:directory}/docs
+
+[check-readme]
+recipe = collective.recipe.template
+input = inline:
+    #!/bin/sh
+    python ${buildout:directory}/setup.py check -r -s
+output = ${buildout:directory}/bin/check-readme
+mode = 755

docs/base/dependencies.rst

@@ -2,6 +2,9 @@ Dependencies
 ------------
 
 Currently we depend on `collective.indexing` as a means to hook into the normal catalog machinery of Plone to detect content changes.
-`c.indexing` before version two had some persistent data structures that frequently caused problems when removing the add-on. These problems have been fixed in version two.
-Unfortunately `c.indexing` still has to hook the catalog machinery in various evil ways, as the machinery lacks the required hooks for its use-case.
-Going forward it is expected for `c.indexing` to be merged into the underlying `ZCatalog` implementation, at which point `collective.solr` can use those hooks directly.
+`c.indexing` before version two had some persistent data structures that frequently caused problems when removing the add-on.
+These problems have been fixed in version two.
+Unfortunately `c.indexing` still has to hook the catalog machinery in various evil ways,
+as the machinery lacks the required hooks for its use-case.
+Going forward it is expected for `c.indexing` to be merged into the underlying `ZCatalog` implementation,
+at which point `collective.solr` can use those hooks directly.

docs/base/index.rst

@@ -1,8 +1,6 @@
 Base Information how Solr and the Integration of Solr and Plone work
 ====================================================================
 
-
-
 Architecture
 ------------
 

docs/base/indexing.rst

@@ -3,37 +3,55 @@ Indexing
 
 Solr is not transactional aware or supports any kind of rollback or undo.
 We therefor only sent data to Solr at the end of any successful request.
-This is done via collective.indexing, a transaction manager and an end request transaction hook.
+This is done via collective.indexing,
+a transaction manager and an end request transaction hook.
 This means you won't see any changes done to content inside a request when doing Solr searches later on in the same request.
 Inside tests you need to either commit real transactions or otherwise flush the Solr connection.
-There's no transaction concept, so one request doing a search might get some results in its beginning, than a different request might add new information to Solr.
+There's no transaction concept,
+so one request doing a search might get some results in its beginning,
+than a different request might add new information to Solr.
 If the first request is still running and does the same search again it might get different results taking the changes from the second request into account.
 
 Solr is not a real time search engine.
-While there's work under way to make Solr capable of delivering real time results, there's currently always a certain delay up to some minutes from the time data is sent to Solr to when it is available in searches.
+While there's work under way to make Solr capable of delivering real time results,
+there's currently always a certain delay up to some minutes from the time data is sent to Solr to when it is available in searches.
 
 Search results are returned in Solr by distinct search threads.
 These search threads hold a great number of caches which are crucial for Solr to perform.
-When index or unindex operations are sent to Solr, it will keep those in memory until a commit is executed on its own search index.
+When index or unindex operations are sent to Solr,
+it will keep those in memory until a commit is executed on its own search index.
 When a commit occurs, all search threads and thus all caches are thrown away and new threads are created reflecting the data after the commit.
-While there's a certain amount of cache data that is copied to the new search threads, this data has to be validated against the new index which takes some time.
+While there's a certain amount of cache data that is copied to the new search threads,
+this data has to be validated against the new index which takes some time.
 The `useColdSearcher` and `maxWarmingSearchers` options of the Solr recipe relate to this aspect.
 While cache data is copied over and validated for a new search thread, the searcher is `warming up`.
 If the warming up is not yet completed the searcher is considered to be `cold`.
 
-In order to get real good performance out of Solr, we need to minimize the number of commits against the Solr index.
+In order to get real good performance out of Solr,
+we need to minimize the number of commits against the Solr index.
 We can achieve this by turning off `auto-commit` and instead use `commitWithin`.
 So we don't sent a `commit` to Solr at the end of each index/unindex request on the Plone side.
 Instead we tell Solr to commit the data to its index at most after a certain time interval.
 Values of 15 minutes to 1 minute work well for this interval.
-The larger you can make this interval, the better the performance of Solr will be, at the cost of search results lagging behind a bit.
-In this setup we also need to configure the `autoCommitMaxTime` option of the Solr server, as `commitWithin` only works for index but not unindex operations.
+The larger you can make this interval,
+the better the performance of Solr will be,
+at the cost of search results lagging behind a bit.
+In this setup we also need to configure the `autoCommitMaxTime` option of the Solr server,
+as `commitWithin` only works for index but not unindex operations.
 Otherwise a large number of unindex operations without any index operations occurring could not be reflected in the index for a long time.
 
-As a result of all the above, the Solr index and the Plone site will always have slightly diverging contents.
-If you use Solr to do searches you need to be aware of this, as you might get results for objects that no longer exist.
+As a result of all the above,
+the Solr index and the Plone site will always have slightly diverging contents.
+If you use Solr to do searches you need to be aware of this,
+as you might get results for objects that no longer exist.
 So any `brain/getObject` call on the Plone side needs to have error handling code around it as the object might not be there anymore and traversing to it can throw an exception.
 
-When adding new or deleting old content or changing the workflow state of it, you will also not see those actions reflected in searches right away, but only after a delay of at most the `commitWithin` interval.
-After a `commitWithin` operation is sent to Solr, any other operations happening during that time window will be executed after the first interval is over.
-So with a 15 minute interval, if document A is indexed at 5:15, B at 5:20 and C at 5:35, both A & B will be committed at 5:30 and C at 5:50.
+When adding new or deleting old content or changing the workflow state of it,
+you will also not see those actions reflected in searches right away,
+but only after a delay of at most the `commitWithin` interval.
+After a `commitWithin` operation is sent to Solr,
+any other operations happening during that time window will be executed after the first interval is over.
+So with a 15 minute interval,
+if document A is indexed at 5:15,
+B at 5:20 and C at 5:35,
+both A & B will be committed at 5:30 and C at 5:50.

docs/base/searching.rst

@@ -2,10 +2,13 @@ Searching
 *********
 
 Information retrieval is a complex science.
-We try to give a very brief explanation here, refer to the literature and documentation of Lucene/Solr for much more detailed information.
+We try to give a very brief explanation here,
+refer to the literature and documentation of Lucene/Solr for much more detailed information.
 
-If you do searches in normal Plone, you have a search term and query the SearchableText index with it.
-The SearchableText is a simple concatenation of all searchable fields, by default title, description and the body text.
+If you do searches in normal Plone,
+you have a search term and query the SearchableText index with it.
+The SearchableText is a simple concatenation of all searchable fields,
+by default title, description and the body text.
 
 The default ZCTextIndex in Plone uses a simplified version of the Okapi BM25 algorithm described in papers in 1998.
 It uses two metrics to score documents:
@@ -15,40 +18,60 @@ It uses two metrics to score documents:
   Terms only occurring in a few documents are scored higher than those occurring in many documents.
 
 It calculates the sum of all scores, for every term common to the query and any document.
-So for a query with two terms, a document is likely to score higher if it contains both terms, except if one of them is a very common term and the other document contains the non-common term more often.
+So for a query with two terms,
+a document is likely to score higher if it contains both terms,
+except if one of them is a very common term and the other document contains the non-common term more often.
 
-The similarity function used in Solr/Lucene uses a different algorithm, based on a combination of a boolean and vector space model, but taking the same underlying metrics into account.
+The similarity function used in Solr/Lucene uses a different algorithm,
+based on a combination of a boolean and vector space model,
+but taking the same underlying metrics into account.
 In addition to the term frequency and inverse document frequency Solr respects some more metrics:
 
 - length normalization: The number of all terms in a field.
   Shorter fields contribute higher scores compared to long fields.
 - boost values: There's a variety of boost values that can be applied, both index-time document boost values as well as boost values per search field or search term
 
-In its pre 2.0 versions, collective.solr used a naive approach and mirrored the approach taken by ZCTextIndex.
+In its pre 2.0 versions,
+collective.solr used a naive approach and mirrored the approach taken by ZCTextIndex.
 So it sent each search query as one query and matched it against the full SearchableText field inside Solr.
 By doing that Solr basically used the same algorithm as ZCTextIndex as it only had one field to match with the entire text in it.
-The only difference was the use of the length normalization, so shorter documents ranked higher than those with longer texts.
-This actually caused search quality to be worse, as you'd frequently find folders, links or otherwise rather empty documents.
+The only difference was the use of the length normalization,
+so shorter documents ranked higher than those with longer texts.
+This actually caused search quality to be worse,
+as you'd frequently find folders, links or otherwise rather empty documents.
 The Okapi BM25 implementation in ZCTextIndex deliberately ignores the document length for that reason.
 
-In order to get good or better search quality from Solr, we have to query it in a different way.
-Instead of concatenating all fields into one big text, we need to preserve the individual fields and use their intrinsic importance. 
+In order to get good or better search quality from Solr,
+we have to query it in a different way.
+Instead of concatenating all fields into one big text,
+we need to preserve the individual fields and use their intrinsic importance.
 We get the main benefit be realizing that matches on the title and description are more important than matches on the body text or other fields in a document.
-collective.solr 2.0+ does exactly that by introducing a `search-pattern` to be used for text searches. In its default form it causes each query to work against the title, description and full searchable text fields and boosts the title by a high and the description by a medium value.
-The length normalization already provides an improvement for these fields, as the title is likely short, the description a bit longer and the full text even longer.
+collective.solr 2.0+ does exactly that by introducing a `search-pattern` to be used for text searches.
+In its default form it causes each query to work against the title,
+description and full searchable text fields and boosts the title by a high and the description by a medium value.
+The length normalization already provides an improvement for these fields,
+as the title is likely short,
+the description a bit longer and the full text even longer.
 By using explicit boost values the effect gets to be more pronounced.
 
 If you do custom searches or want to include more fields into the full text search you need to keep the above in mind.
 Simply setting the `searchable` attribute on the schema of a field to `True` will only include it in the big searchable text stream.
-If you for example include a field containing tags, the simple tag names will likely 'drown' in the full body text.
+If you for example include a field containing tags,
+the simple tag names will likely 'drown' in the full body text.
 You might want to instead change the search pattern to include the field and potentially put a boost value on it - though it will be more important as it's likely to be extremely short.
 Similarly extracting the full text of binary files and simply appending them into the search stream might not be the best approach.
 You should rather index those in a separate field and then maybe use a boost value of less than one to make the field less important.
-Given two documents with the same content, one as a normal page and one as a binary file, you'll likely want to find the page first, as it's faster to access and read than the file.
-
+Given two documents with the same content,
+one as a normal page and one as a binary file,
+you'll likely want to find the page first,
+as it's faster to access and read than the file.
 
 There's a good number of other improvements you can do using query time and index time boost values.
-To provide index time boost values, you can provide a skin script called `solr_boost_index_values` which gets the object to be indexed and the data sent to Solr as arguments and returns a dictionary of field names to boost values for each document.
-The safest is to return a boost value for the empty string, which results in a document boost value.
-Field level boost values don't work with all searches, especially wildcard searches as done by most simple web searches.
-The index time boost allows you to implement policies like boosting certain content types over others, taking into account ratings or number of comments as a measure of user feedback or anything else that can be derived from each content item.
+To provide index time boost values,
+you can provide a skin script called `solr_boost_index_values` which gets the object to be indexed and the data sent to Solr as arguments and returns a dictionary of field names to boost values for each document.
+The safest is to return a boost value for the empty string,
+which results in a document boost value.
+Field level boost values don't work with all searches,
+especially wildcard searches as done by most simple web searches.
+The index time boost allows you to implement policies like boosting certain content types over others,
+taking into account ratings or number of comments as a measure of user feedback or anything else that can be derived from each content item.

docs/development/TODO.rst

@@ -14,8 +14,8 @@ TODOs:
 * evaluate http://www.gnuenterprise.org/~jcater/solr.py as a replacement
   (also see http://tinyurl.com/2zcogf)
 * evaluate sunburnet as a replacement https://pypi.python.org/pypi/sunburnt
-* evaluat mysolr as backend https://pypi.python.org/pypi/mysolr
+* evaluate mysolr as backend https://pypi.python.org/pypi/mysolr
 * implement LocalParams to have a nicer facet view http://wiki.apache.org/solr/SimpleFacetParameters#Multi-Select_Faceting_and_LocalParams
-* Use current search view and get rid of anicient search override
+* Use current search view and get rid of ancient search override
 * Implement a push only and read only mode
 * Play nice with eea.facetednavigation