docs improvements #779 #751 #749; ref 2.8.2

docs/command_line_tools_reference/searchd_command_reference.rst

@@ -119,8 +119,8 @@ The options available to ``searchd`` on all builds are:
    ``query_log`` will need to have been activated in ``sphinx.conf``) to
    provide more detailed information on a per-query basis as to the
    input/output operations carried out in the course of that query, with
-   a slight performance hit and of course bigger logs. Further details
-   are available under the `query log
+   a slight performance hit and of course bigger logs.  The IO statistics don't include information about IO operations for attributes,as these are loaded with mmap().
+   Further details are available under the `query log
    format <README>` section. You might
    start ``searchd`` thus:
 

docs/conf_options_reference/index_configuration_options.rst

@@ -1676,6 +1676,11 @@ For dict=keywords word infixing and prefixing cannot be both enabled at the same
 For dict=crc it is possible to specify only some fields to have infixes  declared with :ref:`infix_fields <infix_fields>`  and
 other fields to have prefixes declared with :ref:`prefix_fields <prefix_fields>`, but it's forbidden to declare same field in both lists.
 
+In case of dict=keywords, beside the wildcard ``*`` two other wildcard characters can be used:
+
+* ``?`` can match any(one) character: ``t?st`` will match ``test``, but not ``teast``
+* ``%`` can match zero or one character : ``tes%`` will match ``tes`` or ``test``, but not ``testing``
+
 Example:
 
 

docs/indexing/indexes.rst

@@ -84,14 +84,18 @@ Index files
 
 Each index consists of a number of files. 
 
+Small index components are fully loaded into memory. 
+Big index components  are read from disk as needed.  Currently these use seek+read to retrieve data from disk.
+Attribute components are opened and mapped with mmap(). They can be loaded fully in memory or left on disk and read when needed.
+
 Plain indexes and RealTime indexes chunks:
 
 +-----------+------------------------------+-----------------------------------------+
 | Extension |  Stores                      | Memory management                       |
 +===========+==============================+=========================================+
-| spa       | scalar attrs                 | see :ref:`ondisk_attrs`                 |
+| spa       | scalar attrs                 | mmap(), also see :ref:`ondisk_attrs`    |
 +-----------+------------------------------+-----------------------------------------+
-| spd       | document lists               | on disk, gets cached by OS              |
+| spd       | document lists               | read, on disk, gets cached by OS        |
 +-----------+------------------------------+-----------------------------------------+
 | spi       | dictionary                   | always loaded in memory                 |
 +-----------+------------------------------+-----------------------------------------+
@@ -101,11 +105,11 @@ Plain indexes and RealTime indexes chunks:
 +-----------+------------------------------+-----------------------------------------+
 | spl       | index lock file              | on disk only                            |
 +-----------+------------------------------+-----------------------------------------+
-| spm       | MVA attrs                    | see :ref:`ondisk_attrs`                 |
+| spm       | MVA attrs                    | mmap(), also see :ref:`ondisk_attrs`    |
 +-----------+------------------------------+-----------------------------------------+
-| spp       | keyword positions            | on disk, gets cached by OS              |
+| spp       | keyword positions            | read, on disk, gets cached by OS        |
 +-----------+------------------------------+-----------------------------------------+
-| sps       | string/json attrs            | see :ref:`ondisk_attrs`                 |
+| sps       | string/json attrs            | mmap(), also see :ref:`ondisk_attrs`    |
 +-----------+------------------------------+-----------------------------------------+
 | mvp       | MVA attrs updates :sup:`[1]` | always loaded in memory                 |
 +-----------+------------------------------+-----------------------------------------+

docs/installation.rst

@@ -281,8 +281,8 @@ Required libraries/packages on Linux
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 * Development version of 'ssl' lib. Usually comes in package named like `libssl-dev` or `openssl-devel`.
-* Development version of 'boost'. On debian packages `libboost-system-dev` and `libboost-program-options-dev`
-are enough; on redhat it is `boost-devel`.
+* Development version of 'boost'. On debian packages `libboost-system-dev` and `libboost-program-options-dev` are enough; on redhat it is `boost-devel`.
+
 
 Optional dependencies
 ~~~~~~~~~~~~~~~~~~~~~

docs/searching/percolate_query.rst

@@ -9,17 +9,22 @@ it works opposite to a regular search where documents are stored in an index and
 Queries are stored in special kind index and they can be added, deleted and listed using INSERT/DELETE/SELECT statements
 similar way as it's done for a regular index.
 
-Checking if a document matches any of the predefined criterias (queries) performed via sphinxql ``CALL PQ`` function, or via http
+Checking if a document matches any of the predefined criterias (queries) performed via sphinxql :ref:`CALL PQ<call_pq_syntax>`  function, or via http
 ``/json/pq/<index>/_search`` endpoint. They returns list of matched queries and may be additional info as matching clause, filters, and tags.
 
+The workflow of using percolate queries is the following:
+
+* a percolate index is defined in the configuration
+* queries are inserted in the percolate index in same way as documents for a Real-Time index
+* documents can be tested against the stored queries with CALL PQ statement
+
 .. _percolate_query_index:
 
 The percolate index
 ~~~~~~~~~~~~~~~~~~~
 
-A percolate query works only for ``percolate`` index :ref:`type <type>`.  The percolate index internaly is a modified  Real-Time index
-and shares a similar configuration. The schema provided in config (list of fields and attributes) describes documents which
-you'll provide for matching. If the schema is omited, index will imply default field ``text`` and an integer attribute ``gid``.
+A percolate query works only for ``percolate`` index :ref:`type <type>`.  
+The percolate index internaly is a modified  Real-Time index and shares a similar configuration. 
 
 .. code-block:: ini
 
@@ -28,17 +33,27 @@ you'll provide for matching. If the schema is omited, index will imply default f
         type = percolate
         path = path/index_name
         min_infix_len   = 4
+        rt_field = title
+        rt_field = body
+        rt_attr_uint = author_id
     }
 
-The percolate index has several particularities over regular Real-Time indexes:
 
-The doc `id` has autoincrement functionality.
 
-The declared fields and attributes are not used for storing data, they define the document schema used by percolate queries
-and incoming documents.
+The fields and attributes  declared in the configuration  define the document schema used by percolate queries
+and documents that will be tested against stored queries using :ref:`CALL PQ <call_pq_syntax>`  command. The schema can be viewed with :ref:`DESCRIBE table_name TABLE<describe_syntax>` statement.
+If the schema is omited, index will imply default field ``text`` and an integer attribute ``gid``.
+The default field and attribute are removed  when defining explicit a schema, but their name can be reused if needed.
+
+The stored queries and the tested documents must respect the defined schema in the percolate index configuration. Trying to use fields/attributes not declared in the schema will give results with no matches.
 
-Besided regular fields and attributes, the percolate index has 3 specific properties that are enabled by default and these are
-used in INSERT statements:
+The schema of the percolate index itself which can be viewed with :ref:`DESCRIBE table_name<describe_syntax>`  and which will be used in INSERT statements to add queries in the percolate index  uses a fixed structure 
+and contain the following elements:
+
+ID
+^^
+
+The "id" is an unsigned 64-bit integer with **autoincrement** functionality therefor it can be ommited in INSERT statements.
 
 .. _percolate_query_query:
 
@@ -73,6 +88,7 @@ The tags  can be returned in the CALL PQ result set.
 Index schemas
 ~~~~~~~~~~~~~
 
+
 Usual sphinxql command :ref:`DESCRIBE<describe_syntax>` will reveal you both internal (schema for call pq) and external (schema for select).
 
 Store queries