Merge PR #134 into 16.0

Signed-off-by lmignon

.pre-commit-config.yaml

@@ -3,7 +3,6 @@ exclude: |
   # NOT INSTALLABLE ADDONS
   ^connector_algolia/|
   ^connector_elasticsearch/|
-  ^connector_search_engine/|
   # END NOT INSTALLABLE ADDONS
   # Files and folders generated by bots, to avoid loops
   ^setup/|/static/description/index\.html$|

connector_search_engine/README.rst

@@ -14,24 +14,146 @@ Connector Search Engine
     :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
     :alt: License: AGPL-3
 .. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fsearch--engine-lightgray.png?logo=github
-    :target: https://github.com/OCA/search-engine/tree/14.0/connector_search_engine
+    :target: https://github.com/OCA/search-engine/tree/16.0/connector_search_engine
     :alt: OCA/search-engine
 .. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
-    :target: https://translation.odoo-community.org/projects/search-engine-14-0/search-engine-14-0-connector_search_engine
+    :target: https://translation.odoo-community.org/projects/search-engine-16-0/search-engine-16-0-connector_search_engine
     :alt: Translate me on Weblate
 .. |badge5| image:: https://img.shields.io/badge/runbot-Try%20me-875A7B.png
-    :target: https://runbot.odoo-community.org/runbot/276/14.0
+    :target: https://runbot.odoo-community.org/runbot/276/16.0
     :alt: Try me on Runbot
 
 |badge1| |badge2| |badge3| |badge4| |badge5| 
 
-Base module for connecting Odoo with external search engines.
+Base module for connecting Odoo with external search engines. This addon is
+intended to be used as a base for other addons that implement specific search
+engines. It's designed to be easily extensible and modular.
 
 **Table of contents**
 
 .. contents::
    :local:
 
+Installation
+============
+
+This addon uses the native json python package provided by python. When
+a json for a record is recomputed, the new value is compared to the original
+one to see if an export to the search engine index is needed.  This is
+done by comparing the md5 of the two json strings. This process when done on
+a large number of records can be slow when the json is large and complex. To speed
+up this process you can install the orjson package.
+
+.. code-block:: bash
+
+    pip install orjson
+
+Usage
+=====
+
+Overview
+~~~~~~~~
+
+A search engine is a system designed to store information in a way that makes
+it easy to find through search and analytics queries. The main difference
+between a search engine and a database is that a search engine is optimized
+for search and analytics queries, while a database is optimized for
+transactional and relational queries.
+
+This addons is designed around 4 main concepts:
+
+* **The search engine backend** is used to define into Odoo the kind
+  of search engine that will be used to index the data. It's main responsibility
+  is to provide an instance of `odoo.addons.search_engine.tools.adapter.SearchEngineAdapter`
+  that will be used to communicate with the search engine.
+
+* **The search engine index** is used to define into Odoo the index where
+  the data will be indexed. An index is always linked to a search engine backend.
+  The index provides methods to use to manage the lifecycle of the data put into
+  the index for the records of a given model. To do so, it uses:
+
+  * **The SearchEngineAdapter** provided by the backend to communicate with the
+    search engine.
+  * **A ModelSerializer** that is used to transform an odoo record into
+    a dictionary that can be indexed into the search engine.
+  * **A JsonValidator** that is used to validate the data that is to be
+    indexed into the search engine.
+
+  The RecordSerializer and IndexDataValidator are defined on the index itself.
+  The current addon provides a default implementation only for the IndexDataValidator.
+  You can find into the github repository `search-engine <https://github.com:
+  OCA/search-engine/tree/16.0>`_ An implementation of the RecordSerializer based
+  on the jsonifier addon `connector_search_engine_jsonifier`.
+
+* **The search engine indexable record** is a mixin that is used to define
+  the records that can be indexed into a search engine index. The mixin
+  provides methods:
+
+  * To add a record to an index.
+  * To remove a record from an index.
+  * To mark the record into an index (*the search engine bindings*) as to be
+    recomputed (This method should be called when modifications are made on
+    the record that could impact the data that are indexed into the search
+    engine. It will instruct the index that the record must be recomputed and
+    re-indexed).
+
+  It also ensures that when the record is unlinked, it is removed from the indexes
+  it was indexed into.
+
+* **The search engine binding** is a model that represents the link between
+  an index and an indexable odoo record. It give you access to the data
+  that are indexed into the search engine for the record. It's also used to
+  manage the lifecycle of the data into the search engine. When a binding is
+  created, it's marked as to be computed. Once the data are computed, the
+  binding is marked as to be indexed. Once the data are indexed, the binding
+  is marked as indexed. If the linked record is unlinked, the binding is
+  marked as to be removed. Once the data are removed from the search engine,
+  the binding is deleted.
+
+Indexing lifecycle
+~~~~~~~~~~~~~~~~~~
+
+The indexing lifecycle is based on the following steps:
+
+* When a record is added to an index, a binding is created and marked as to be
+  computed.
+* A cron job scheduled every 5 minutes will look for bindings that are to be
+  computed and for each of them will schedule a job to re compute the json data.
+* When the json data is computed, the binding is marked as to be exported if the
+  json is valid and is different from the one that has been computed last time.
+* A cron job scheduled every 5 minutes will ensure the syncing with the search
+  engine. It will:
+
+  * look for bindings that are to be exported and for each of them will schedule
+    a job to export the json data into the search engine. Once exported, the
+    binding is marked as 'done'.
+  * look for bindings that are to be removed and for each of them will schedule
+    a job to remove the data from the search engine. Once removed, the binding
+    is deleted.
+
+To keep in sync the data from your model instance and the data that are indexed
+into the search engine, you should call the method `_se_mark_to_update` on the
+mode instance when you make modifications that could impact the data that are
+indexed into the search engine.
+
+* When the method `_se_mark_to_update` is called, the binding is marked as to be
+  computed.
+* From there, the same process as described above will be used to recompute the
+  data and reindex them into the search engine.
+
+When a model instance is unlinked, the binding is marked as to be removed. From
+there if will be processed by the job syncing the data with the search engine.
+
+.. note::
+
+  In previous versions of this addon, there was no method to mark a record as
+  to be recomputed. As a consequence, all the records were re-computed every day
+  to ensure that the data in the search engine were up to date. This was a
+  performance issue and consumed a lot of resources. If despite this, you want
+  to recompute all the records every day, you can activate the cron jon
+  `Search engine: recompute all index` and deactivate the one named
+  `earch engine: Generate job for recompute binding to recompute per index`.
+
 Known issues / Roadmap
 ======================
 
@@ -54,7 +176,7 @@ Bug Tracker
 Bugs are tracked on `GitHub Issues <https://github.com/OCA/search-engine/issues>`_.
 In case of trouble, please check there if your issue has already been reported.
 If you spotted it first, help us smashing it by providing a detailed and welcomed
-`feedback <https://github.com/OCA/search-engine/issues/new?body=module:%20connector_search_engine%0Aversion:%2014.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+`feedback <https://github.com/OCA/search-engine/issues/new?body=module:%20connector_search_engine%0Aversion:%2016.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
 
 Do not contact contributors directly about support or help with technical issues.
 
@@ -89,6 +211,6 @@ OCA, or the Odoo Community Association, is a nonprofit organization whose
 mission is to support the collaborative development of Odoo features and
 promote its widespread use.
 
-This module is part of the `OCA/search-engine <https://github.com/OCA/search-engine/tree/14.0/connector_search_engine>`_ project on GitHub.
+This module is part of the `OCA/search-engine <https://github.com/OCA/search-engine/tree/16.0/connector_search_engine>`_ project on GitHub.
 
 You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

connector_search_engine/__init__.py

@@ -3,4 +3,4 @@
 # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
 
 from . import models  # pragma: no cover
-from . import components  # pragma: no cover
+from . import tools

connector_search_engine/__manifest__.py

@@ -4,27 +4,29 @@
 
 {
     "name": "Connector Search Engine",
-    "version": "14.0.2.6.2",
+    "version": "16.0.0.0.0",
     "author": "Akretion,"
     "ACSONE SA/NV,"
     "Camptocamp,"
     "Odoo Community Association (OCA)",
     "website": "https://github.com/OCA/search-engine",
     "license": "AGPL-3",
     "category": "Generic Modules",
-    "depends": ["connector", "jsonifier", "server_environment"],
+    "depends": ["queue_job", "mail", "server_environment"],
     "external_dependencies": {"python": ["unidecode"]},
     "data": [
         "security/connector_search_engine_security.xml",
         "security/se_index_config.xml",
         "security/ir.model.access.csv",
         "views/se_backend.xml",
         "views/se_index.xml",
+        "views/se_binding_view.xml",
         "views/se_index_config.xml",
         "views/se_menu.xml",
         "data/ir_cron.xml",
         "data/queue_job_channel_data.xml",
         "data/queue_job_function_data.xml",
+        "data/ir_action_data.xml",
     ],
-    "installable": False,
+    "installable": True,
 }

connector_search_engine/data/ir_action_data.xml

@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<odoo>
+
+    <record model="ir.actions.server" id="action_recompute_se_index">
+        <field name="name">Recompute Search Index</field>
+        <field name="model_id" ref="connector_search_engine.model_se_binding" />
+        <field name="state">code</field>
+        <field name="code">model.recompute_from_owner()</field>
+    </record>
+
+</odoo>