Added multiple database support.

.coveragerc

@@ -3,3 +3,5 @@ source = example/article,parler
 omit =
     */south_migrations/*
     */migrations/*
+    */tests/*
+    */example/*

.github/workflows/tests.yaml

@@ -1,5 +1,6 @@
 name: CI Testing
 on:
+  workflow_dispatch:
   pull_request:
     branches:
       - master
@@ -10,9 +11,11 @@ on:
 jobs:
   test:
     name: "Python ${{ matrix.python }} Django ${{ matrix.django }}"
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
+      # AB, 09/2023 Changed from ubuntu-latest to ubuntu-20.04 because there is no py3.6 build for ubuntu-22.04
+      #             Ref: https://github.com/actions/setup-python/issues/544
     strategy:
-      # max-parallel: 8  # default is max available
+      # max-parallel: 8  # default is max available.
       fail-fast: false
       matrix:
         include:
@@ -46,18 +49,30 @@ jobs:
           - django: "3.2"
             python: "3.10"
           # Django 4.0
-          - django: "4.0b1"
+          - django: "4.0"
+            python: "3.9"
+          - django: "4.0"
+            python: "3.10"
+          # Django 4.1
+          - django: "4.1"
+            python: "3.9"
+          - django: "4.1"
+            python: "3.10"
+          # Django 4.2
+          - django: "4.2"
+            python: "3.9"
+          - django: "4.2"
             python: "3.10"
 
     steps:
       - name: Install gettext
         run: sudo apt-get install -y gettext
 
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Setup Python ${{ matrix.python }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python }}
 

CHANGES.rst

@@ -1,6 +1,16 @@
 Changelog
 =========
 
+Changes in 2.x (date TBD)
+-------------------------
+
+Added support for multiple databases:
+
+- Fixed cache overlap issue when using multiple database: cache key now includes the database alias.
+- Now properly support multiple databases and translatable models duplication (fully documented on page `Duplicating instances, using multiple databases and more...`).
+- Now clearly identify some marginal case django-parler does not support, and raise an exception instead of silently failing and possibly corrupting the database.
+
+
 Changes in 2.3 (2021-11-18)
 ---------------------------
 

docs/_ext/djangodummy/requirements.txt

@@ -1,5 +1,6 @@
 # for readthedocs
 # Remaining requirements are picked up from setup.py
-Django==2.2.24
+django==4.2
+sphinx==7.2.5
 djangorestframework==3.12.4
-sphinxcontrib-django == 0.5.1
+sphinxcontrib-django==2.4

docs/_ext/djangodummy/settings.py

@@ -13,7 +13,7 @@
 SITE_ID = 1
 
 INSTALLED_APPS = [
-    "parser",
+    "parler",
     "django.contrib.admin",
     "django.contrib.auth",
     "django.contrib.contenttypes",

docs/_static/dummy_static_file.txt

@@ -0,0 +1,3 @@
+This file is just present to make sure docs/_static is part of the repository.
+If this folder is missing, recent versions of Sphinx (e.g. 7.2.5) issue a warning,
+which becomes an error when the documentation build is run by tox.

docs/advanced/cache_design.rst

@@ -0,0 +1,24 @@
+:orphan:
+
+..
+    NB :orphan: tag required because this document is not part of any toctree
+                but just included from another page. Without the tag, sphinx issues a warning
+
+Some technical considerations about caching
+===========================================
+
+.. versionadded 2.x
+
+When fixing the cache overlap issue in Parler <=2.3, several solutions have been considered:
+
+    a) Include the database name in the cache key, to have separate cache for each model in each DB. This requires an additional parameter to ``get_translation_cache_key``, to be provided by every user of this function (5 of them, all in cache.py module, + a couple of tests).
+
+    b) Leave the key unchanged, but use a separate cache for each database. This requires:
+
+        - Configuring a CACHE for each DATABASE (in the settings)
+        - Adding a check at startup to make sure configuration fulfills the above constraint.
+        - Adapting the 5 methods actually interacting with the cache: ``cache.xxx`` becomes ``caches[db_alias].xxx``.
+
+Both solutions make it necessary to use the db alias associated to each model, which is actually available in any ``<model_instance>._state.db``.
+
+Solution a) was selected since it is 100% transparent to the users while solution b) would force an update on the cache configuration in the settings of every multi-db project.

docs/advanced/caching.rst

@@ -1,3 +1,8 @@
+Caching in Django-parler
+========================
+
+Django-parler provides a transparent caching feature, using the ``default cache`` configured by django setting ``CACHES``.
+
 Disabling caching
 -----------------
 
@@ -11,3 +16,14 @@ Parler on more sites with same cache
 If Parler runs on multiple sites that share the same cache, it is necessary
 to set a different prefix for each site
 by adding :ref:`PARLER_CACHE_PREFIX = 'mysite' <PARLER_CACHE_PREFIX>` to the settings.
+
+
+Parler caching with multiple database
+-------------------------------------
+
+.. versionchanged 2.x:: The use of the cache by Django-parler <= 2.3 caused cache overlap across databases with problematic side-effects.
+
+Parler caching can be used safely when using multiple databases: models retrieved from different databases are now use distinct cache keys.
+
+Technical design information can be found on :doc:`this page <cache_design>`.
+

docs/advanced/index.rst

@@ -14,5 +14,6 @@ Advanced usage patterns
    rest-framework
    multisite
    caching
+   multiple_db
    manual-models
    custom-settings

docs/advanced/multiple_db.rst

@@ -0,0 +1,105 @@
+Duplicating instances, using multiple databases and more...
+===========================================================
+
+.. versionchanged:: 2.x
+    In previous versions, the caching mechanism was common to all databases with nasty side-effects when managing instances of the same Model in several databases. Once fixed, tests revealed that a number of operations commonly performed on plain Models where either silently failing or resulting in unintuitive effects (compared to the effects described by Django documentation about plain Models). Hence this summary to clarify how typical Django operation are supported (or not) for translatable models.
+
+Overview
+--------
+
+When using single or multiple databases, we expect the behaviour of translatable models to be as close as possible to the behaviour of plain models (as described in Django documentation), when performing similar operations. This page reviews the main operations we can perform on Models, and clarifies how Django-parler provides the same behaviour for translatable models.
+
+.. note:: Django documentation (https://docs.djangoproject.com/en/4.2/topics/db/multi-db/) states that
+
+        "If you don’t specify using, the save() method will save into the default database allocated by the routers."
+
+    and
+
+        "By default, a call to delete an existing object will be executed on the same database that was used to retrieve the object in the first place."
+
+    After careful checking and testing: delete() and save() BOTH determine the database to use the same way,
+    and both use as default db, the database from which the object was retrieved. The statement above about ``save()`` is only valid if the object was never saved into a database before.
+
+
+Creating a new model
+--------------------
+
+The typical creation of a regular model is:
+
+.. code-block:: python
+
+    obj = MyModel(...)
+    # set here any number of attribute and/or relations
+    obj.save()  # or obj.save(using="my_db")
+
+The corresponding sequence for a translatable model is:
+
+.. code-block:: python
+
+    obj = MyTranslatableModel(...)
+    # set here any number of attribute and/or relations in any number of languages
+    obj.save()  # or obj.save(using="my_db")
+
+The sequence is strictly similar, and Django-parler saves all translations, just as Django ORM saves all fields. In both cases, if the instance is saved to a non-default database ``my_db`` (with ``save(using="my_db")``), any further operation will by default be performed on ``my_db``.
+
+Retrieving a model from a non-default database
+----------------------------------------------
+
+When retrieving a model from a non-default database (e.g with ``MyModel.objects.using("my_db").filter(...)``), we expect a subsequent database operation (e.g ``my_instance.save()`` or ``my_instance.delete()``, without an explicit ``using="my_db"`` argument) to be applied to the database the instance was retrieved from, and we expect *all* translations to be saved or deleted, together with the master model instance.  This is the standard behaviour for regular models, and is similarly implemented for translatable models:
+
+.. code-block:: python
+
+    obj = MyTranslatableModel.objects.using("my_db").get(....)
+    # modify instance and translations in any language
+    obj.save()   # Saves all translations in my_db implicitly.
+
+Duplicating a model retrieved from a database into the same or another database
+-------------------------------------------------------------------------------
+
+.. versionchanged 2.x :: Version <= 2.3 did not save non-prefetched translations nor unchanged translations when duplicating a translatable model.
+
+When working with a model created in ``'my_db'`` or retrieved from ``'my_db'``, a plain model allows to
+easily insert it in another database or duplicate it in the same database:
+
+.. code-block:: python
+
+    obj = MyModel.objects.get(....)
+    # possibly modify obj
+    obj.pk = None
+    obj.save()  #  or obj.save(using="another_db")
+
+Translatable models behave the same way, saving all translations, including possible unsaved edits into the destination database. This enforces the principle that saving a translatable models saves **all** translations.
+
+.. code-block:: python
+
+    obj = MyTranslatableModel.objects.get(....)
+    # possibly modify obj and translations in any language
+    obj.pk = None
+    obj.save()  #  or obj.save(using="another_db") This saves all translations, including edits.
+
+.. warning:: Unsaved changes to the original model are saved in the duplicate obj, in the target database, but are **NOT** saved in the original model in the original database.
+
+.. warning:: As for any Model, when duplicating to a new database, relations to other Models must be carefully considered. Django-parler takes care of transparently duplicating the translations as required, but any other foreign key in your model must be carefully managed to avoid inadvertently referencing models using foreign keys which only make sense in the original database.
+
+Overwriting a model in the same or another database by setting the primary key before saving
+--------------------------------------------------------------------------------------------
+
+Regular models allow this possibly dangerous (and mostly not advisable operation): if ``my_other_db`` includes a model with pk=123, we can force the pk of any model (previously saved in another database or not) to this value, and save it to ``my_other_db`` in order to **overwrite** the existing model (Django will in this case perform and ``UPDATE`` instead of an ``INSERT``). This operation is OK with a model without any relation to other models, but becomes very tricky if relations to other models must be managed.
+
+.. code-block:: python
+
+    obj = MyModel(...)  # or obj = MyModel.objects.using("my_db").get(....)
+    # Possibly update obj
+    obj.pk = 123    # set pk to the pk of an existing model in destination db to update the
+                    # model with this pk in my_other_db
+    obj.save(using="my_other_db")   # OK with a plain (simple) Model, NOT supported for translatable models.
+
+Although possible, this operation requires some precautions to properly overwrite a translatable model: in the most general case, some translations must be overwritten (either with unsaved data or data from the database), some must be created (either with unsaved data or data from the database) and some must be deleted. This is currently NOT supported by django-parler. Attempting it raises a ``NomImplementedError``.
+
+The construct is nevertheless accepted if no model with the provided primary key exists in the target database (and is then just a way to control the primary key of a newly created master model).
+
+.. warning:: Forcing the PK when duplicating an object can result in corrupted data due to race conditions (for translatable models just as for plain models): would the PK be used in the target database between the moment parler checked it was not in use, and the moment parler actually saved the model with this PK, it would result in an unwanted overwrite, and a possibly inconsistent object (e.g. with a mix of translations from both instances using the PK). It is the user's responsibility to use transactions to guarantee this cannot happen, and this is consistent with the general design of Parler, which puts the responsibility for managing transactions to the user.
+
+.. note:: Overwriting an existing model can usually as easily be achieved by retrieving the model from the database, updating it and saving it back.
+
+Technical design information can be found on :doc:`this page <multiple_db_design>`.

docs/advanced/multiple_db_design.rst

@@ -0,0 +1,42 @@
+:orphan:
+
+..
+    NB :orphan: tag required because this document is not part of any toctree
+                but just included from another page. Without the tag, sphinx issues a warning
+
+About the routing of ``save()`` calls
+=====================================
+
+.. versionadded 2.x:
+
+This table summarizes how :func:`~TranslatableModelMixin.save` discriminates between the various actions to be taken when called:
+
+First, we need to know whether the model is (i) new and being saved for the first time, (ii) being saved again in the same database, or (iii) being saved in a new database:
+
+* If ``_state.db`` is None, we are saving **for the 1st time**.
+
+    The destination database is either the one defined y the database router, or the one provided by the user in the ``using="xxx"`` parameter, if any.
+
+* If ``_state.db`` is not None:
+    - If the ``using="xxx"`` parameter is absent or has the same value as ``_state.db``, we are saving **in the same DB**.
+    - Else we are saving **in another DB**.
+
+Next, the action is defined according to this table, based on the previous conclusion, the value of the model's primary key (which may have been tampered with by the caller), and the value of the primary key after the last save (``_last_saved_pk``, which is internal and reliable) :
+
++--------------------------------------+--------------------+--------------------------+--------------------------+
+|                                      | For the 1st time   | In same DB               | In another DB            |
++=============+========================+====================+==========================+==========================+
+| pk = None   | last_saved_pk = None   | Regular first save | *IMPOSSIBLE*             | *IMPOSSIBLE*             |
+|             +------------------------+--------------------+--------------------------+--------------------------+
+|             | last_saved_pk not None | *IMPOSSIBLE*       | Duplication in same DB   | Duplication in new DB    |
++-------------+------------------------+--------------------+--------------------------+--------------------------+
+| pk not None | last_saved_pk = None   | Regular first save | *IMPOSSIBLE*             | *IMPOSSIBLE*             |
+|             +------------------------+--------------------+--------------------------+--------------------------+
+|             | last_saved_pk = pk     | *IMPOSSIBLE*       | Regular update           | Overwrite/Duplicate [1]_ |
+|             +------------------------+--------------------+--------------------------+--------------------------+
+|             | last_saved_pk != pk    | *IMPOSSIBLE*       | Overwrite/Duplicate [1]_ | Overwrite/Duplicate [1]_ |
++-------------+------------------------+--------------------+--------------------------+--------------------------+
+
+.. [1] This is an overwrite if the pk is already in use in the destination database (and this is **not** supported by Parler) and a duplication with forced primary key otherwise (which is accepted by Parler).
+
+

docs/advanced/multisite.rst

@@ -1,6 +1,6 @@
 Multi-site support
 ==================
 
-When using the sites framework (:mod:`django.contrib.sites`) and the :django:setting:`SITE_ID`
+When using the sites framework (:mod:`django.contrib.sites`) and the :setting:`SITE_ID`
 setting, the dict can contain entries for every site ID.
 See the :ref:`configuration <multisite-configuration>` for more details.

docs/conf.py

@@ -61,9 +61,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = "2.3"
+version = "2.x"
 # The full version, including alpha/beta/rc tags.
-release = "2.3"
+release = "2.x"
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -284,7 +284,7 @@
 
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {
-    "https://docs.python.org/3/": None,
-    "https://docs.djangoproject.com/en/dev": "https://docs.djangoproject.com/en/dev/_objects/",
+    'python': ('https://docs.python.org/3', None),
     "polymorphic": ("https://django-polymorphic.readthedocs.io/en/latest/", None),
+    'django': ('http://docs.djangoproject.com/en/stable/', 'http://docs.djangoproject.com/en/stable/_objects/'),
 }

docs/configuration.rst

@@ -10,7 +10,7 @@ PARLER_DEFAULT_LANGUAGE_CODE
 The language code for the fallback language.
 This language is used when a translation for the currently selected language does not exist.
 
-By default, it's the same as :django:setting:`LANGUAGE_CODE`.
+By default, it's the same as :setting:`LANGUAGE_CODE`.
 
 This value is used as input for ``PARLER_LANGUAGES['default']['fallback']``.
 
@@ -66,7 +66,7 @@ The following entries are available:
 Multi-site support
 ~~~~~~~~~~~~~~~~~~
 
-When using the sites framework (:mod:`django.contrib.sites`) and the :django:setting:`SITE_ID`
+When using the sites framework (:mod:`django.contrib.sites`) and the ::setting:`SITE_ID`
 setting, the dict can contain entries for every site ID. The special ``None`` key is no longer used::
 
     PARLER_LANGUAGES = {
@@ -134,7 +134,7 @@ PARLER_SHOW_EXCLUDED_LANGUAGE_TABS
 
     PARLER_SHOW_EXCLUDED_LANGUAGE_TABS = False
 
-By default, the admin tabs are limited to the language codes found in :django:setting:`LANGUAGES`.
+By default, the admin tabs are limited to the language codes found in ::setting:`LANGUAGES`.
 If the models have other translations, they can be displayed by setting this value to ``True``.
 
 

docs/quickstart.rst

@@ -130,5 +130,5 @@ Optionally, the admin tabs can be configured too::
         }
     }
 
-Replace ``None`` with the :django:setting:`SITE_ID` when you run a multi-site project with the sites framework.
-Each :django:setting:`SITE_ID` can be added as additional entry in the dictionary.
+Replace ``None`` with the ::setting:`SITE_ID` when you run a multi-site project with the sites framework.
+Each ::setting:`SITE_ID` can be added as additional entry in the dictionary.

docs/templatetags.rst

@@ -10,7 +10,7 @@ All translated fields can be read like normal fields, just using like:
     {{ object.fieldname }}
 
 When a translation is not available for the field,
-an empty string (or :django:setting:`TEMPLATE_STRING_IF_INVALID`) will be outputted.
+an empty string (or ::setting:`TEMPLATE_STRING_IF_INVALID`) will be outputted.
 The Django template system safely ignores the :class:`~parler.models.TranslationDoesNotExist`
 exception that would normally be emitted in code;
 that's because that exception inherits from :class:`~exceptions.AttributeError`.