dumpdata --natural management command missing parent fields to restore polymorphic child models

[one-geek]

My project settings (my_django_project):
...
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'polymorphic',
'my_app',
)
...

My application models (my_app):

from __future__ import unicode_literals

from django.contrib import admin
from django.db import models
from polymorphic import PolymorphicModel
from polymorphic.admin import PolymorphicParentModelAdmin, PolymorphicChildModelAdmin


class Project(PolymorphicModel):
    topic = models.CharField(max_length=30)


class ArtProject(Project):
    artist = models.CharField(max_length=30)


class ResearchProject(Project):
    supervisor = models.CharField(max_length=30)

class ArtProjectChildAdmin(PolymorphicChildModelAdmin):
    base_model = ArtProject
    base_form = ()
    base_fieldsets = ()


class ResearchProjectAdmin(ArtProjectChildAdmin):
    base_model = ResearchProject


class ProjectParentAdmin(PolymorphicParentModelAdmin):
    base_model = Project
    child_models = (
        (ArtProject, ArtProjectChildAdmin),
        (ResearchProject, ArtProjectChildAdmin),
    )

# Only the parent needs to be registered:
admin.site.register(Project, ProjectParentAdmin)

The issue:
In the django admin, if i created two row of an ArtProject and ResearchProject.
Then, i deleted everything, and try to restore the database with "dumpdata/loaddata" management command. The json file is clear missing some field to refer to the parent object. Like topic field from Project.

• Manage.py dumpdata my_app.project --indent = 4 --natural > data.json

[
{
    "fields": {
        "artist": "guerson"
    },
    "model": "my_app.artproject",
    "pk": 1
},
{
    "fields": {
        "supervisor": "max"
    },
    "model": "my_app.researchproject",
    "pk": 2
}
]

Also, the json file can not be restored because of a missing pointer project_ptr_id, Which breaks the restoration later with the command loaddata

• Manage.py loaddata data.json

referenced_table_name, referenced_column_name)) django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'my_app_artproject' with primary key '1' has an invalid foreign key: my_app_artproject.project_ptr_id contains a value '1' that does not have a corresponding value in my_app_project.id.

My environment :
$ pip freeze

• Django==1.8.6
• django-polymorphic==0.7.2
• mysql-python==1.2.3
• virtualenv==13.1.2

[sheepeatingtaz]

This is affecting me too. Is there any workaround? It is also evident when using --natural-foreign

[sheepeatingtaz]

Workaround for me was to

1. temporarily change the model from PolymorphicModel to models.Model
2. run the dumpdata command (with --natural-foreign)
3. change the model back to PolymorphicModel
4. loaddata and tests (the scenario I was working on) load the fixture as expected.

Granted, this won't work if you want to do it in code, but I just wanted some fixtures to use in tests.

[KyeRussell]

I have this issue too (latest version of polymorphic + latest version of Django 1.8)

The above fix isn't working for me - changing the base class to model removes the polymorphic-related fields from the serialisation, which - whilst still allowing import - means there is data loss.

[oden3000]

+1

[andgein]

I think you use Windows as me. Django-Polymorphic has special hack for this case in polymorphic/base.py:

if len(sys.argv) > 1 and sys.argv[1] == 'dumpdata':
        # manage.py dumpdata is running

        def __getattribute__(self, name):
            if name == '_default_manager':
                frm = inspect.stack()[1]  # frm[1] is caller file name, frm[3] is caller function name
                if 'django/core/management/commands/dumpdata.py' in frm[1]:
                    return self.base_objects

But it doesn't work on Windows because it has backslashes instead of slashes in path. The correct way will be replace
if 'django/core/management/commands/dumpdata.py' in frm[1]:
to
if os.path.join('django', 'core', 'management', 'commands', 'dumpdata.py') in frm[1]:
(and import os.path of course).

@chrisglass If it's not difficult please fix this bug.

[vdboor]

Thanks @andgein , that explains something! I've added a fix in e76259f

[Mimino666]

Still fails when -a flag is used, i.e.: python manage.py dumpdata -a.
In that case, _base_manager is used, instead of _default_manager.

[tony]

Off-topic: As of django 1.11, there is no --natural

There is: --natural-foreign and --natural-primary. Which one(s) would be used if doing it correctly now?

[tony]

The other thing is as of django-polymorphic 1.2, django.1.11.3

If I do a dump-data on a parent model, I get all the data, except the _ptr_id. So polymorphic queries don't populate.

Also, -a won't seem to populate the queries on sub-models with dumpdata either.

[tony]

When using loaddata:

If I'm using post_save signals, I'll get an error the model isn't found (which is expected, no models are found if only the table data for the parent class is populated - django-polymorphic's manager needs to child table's _ptr_id to pull any data, even if using the root model).

But if I just hook into save on the parent model (after I do super(Model, self).save(**kwargs)), no error shows. It just populates the parent table. But no model information shows through ORM at all. Parent table populates data, but sub tables have no relational information.

Hope this helps.

[tony]

This is only with fixtures. By the way. If I create models on the fly through python or through the normal website, relationships are correctly created and show fine.

[tony]

Here's the good news: I do get the "polymorphic_ctype": data in dumpdata.

The bad news is loaddata doesn't actually help populate the relation. For instance, assume the core.Node parent and ask.Question subclassing it.

I'll get a dump with

    "polymorphic_ctype": [
      "ask",
      "question"
    ],

(yay). But upon loaddata, ask.Question's SQL table doesn't create the relation node_ptr_id.

Node.objects.all() and Question.objects.all() both return nothing.

Any ideas @vdboor ?

[ablakey]

Just wanted to call out a related issue for anyone in the future that runs into it. The dumpdata hack that Polymorphic uses explicitly looks at sys.argv[1] for "dumpdata", which is what broke my code. I have modified the manage script to add a "track" to the commands. Eg. python manage local dumpdata python manage devel dumpdata which changed the argument order. This ended up being my problem, so popping the track name off argv after I loaded my desired track's settings fixed everything for me.

[Federico-Comesana]

Still having this issue in 2020. Is there any fix for that? I've published a question in StackOverflow and nothing.
https://stackoverflow.com/questions/60728200/django-polymorphic-dumpdata-command-fails-django-db-utils-integrityerror-prob

[olivierdalang]

Hitting this issue when calling the management command through call_command:

from django.core.management import call_command
call_command("dumpdata", "my_app", "-o", path)

IMO the previous approach (i.e. providing a specific polymorphic_dumpdata command) was more robust. Alternatively an override of dumpdata would also do the trick. It would even open the door to actually making a polymorphic dump (just one object of the subclass, including inherited fields).