Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`dumpdata --natural` management command missing parent fields to restore polymorphic child models #175

Open
guerson opened this issue Dec 14, 2015 · 13 comments

Comments

@guerson
Copy link

@guerson guerson commented Dec 14, 2015

My project settings (my_django_project):
...
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'polymorphic',
'my_app',
)
...

My application models (my_app):

from __future__ import unicode_literals

from django.contrib import admin
from django.db import models
from polymorphic import PolymorphicModel
from polymorphic.admin import PolymorphicParentModelAdmin, PolymorphicChildModelAdmin


class Project(PolymorphicModel):
    topic = models.CharField(max_length=30)


class ArtProject(Project):
    artist = models.CharField(max_length=30)


class ResearchProject(Project):
    supervisor = models.CharField(max_length=30)

class ArtProjectChildAdmin(PolymorphicChildModelAdmin):
    base_model = ArtProject
    base_form = ()
    base_fieldsets = ()


class ResearchProjectAdmin(ArtProjectChildAdmin):
    base_model = ResearchProject


class ProjectParentAdmin(PolymorphicParentModelAdmin):
    base_model = Project
    child_models = (
        (ArtProject, ArtProjectChildAdmin),
        (ResearchProject, ArtProjectChildAdmin),
    )

# Only the parent needs to be registered:
admin.site.register(Project, ProjectParentAdmin)

The issue:
In the django admin, if i created two row of an ArtProject and ResearchProject.
Then, i deleted everything, and try to restore the database with "dumpdata/loaddata" management command. The json file is clear missing some field to refer to the parent object. Like topic field from Project.

  • Manage.py dumpdata my_app.project --indent = 4 --natural > data.json
[
{
    "fields": {
        "artist": "guerson"
    },
    "model": "my_app.artproject",
    "pk": 1
},
{
    "fields": {
        "supervisor": "max"
    },
    "model": "my_app.researchproject",
    "pk": 2
}
]

Also, the json file can not be restored because of a missing pointer project_ptr_id, Which breaks the restoration later with the command loaddata

  • Manage.py loaddata data.json

referenced_table_name, referenced_column_name)) django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'my_app_artproject' with primary key '1' has an invalid foreign key: my_app_artproject.project_ptr_id contains a value '1' that does not have a corresponding value in my_app_project.id.

My environment :
$ pip freeze

  • Django==1.8.6
  • django-polymorphic==0.7.2
  • mysql-python==1.2.3
  • virtualenv==13.1.2
@sheepeatingtaz

This comment has been minimized.

Copy link

@sheepeatingtaz sheepeatingtaz commented Feb 4, 2016

This is affecting me too. Is there any workaround? It is also evident when using --natural-foreign

@sheepeatingtaz

This comment has been minimized.

Copy link

@sheepeatingtaz sheepeatingtaz commented Feb 4, 2016

Workaround for me was to

  1. temporarily change the model from PolymorphicModel to models.Model
  2. run the dumpdata command (with --natural-foreign)
  3. change the model back to PolymorphicModel
  4. loaddata and tests (the scenario I was working on) load the fixture as expected.

Granted, this won't work if you want to do it in code, but I just wanted some fixtures to use in tests.

@KyeRussell

This comment has been minimized.

Copy link

@KyeRussell KyeRussell commented Mar 8, 2016

I have this issue too (latest version of polymorphic + latest version of Django 1.8)

The above fix isn't working for me - changing the base class to model removes the polymorphic-related fields from the serialisation, which - whilst still allowing import - means there is data loss.

@nevros

This comment has been minimized.

Copy link

@nevros nevros commented Mar 18, 2016

+1

@andgein

This comment has been minimized.

Copy link

@andgein andgein commented May 30, 2016

I think you use Windows as me. Django-Polymorphic has special hack for this case in polymorphic/base.py:

if len(sys.argv) > 1 and sys.argv[1] == 'dumpdata':
        # manage.py dumpdata is running

        def __getattribute__(self, name):
            if name == '_default_manager':
                frm = inspect.stack()[1]  # frm[1] is caller file name, frm[3] is caller function name
                if 'django/core/management/commands/dumpdata.py' in frm[1]:
                    return self.base_objects

But it doesn't work on Windows because it has backslashes instead of slashes in path. The correct way will be replace
if 'django/core/management/commands/dumpdata.py' in frm[1]:
to
if os.path.join('django', 'core', 'management', 'commands', 'dumpdata.py') in frm[1]:
(and import os.path of course).

@chrisglass If it's not difficult please fix this bug.

@vdboor

This comment has been minimized.

Copy link
Member

@vdboor vdboor commented Sep 12, 2016

Thanks @andgein , that explains something! I've added a fix in e76259f

@Mimino666

This comment has been minimized.

Copy link

@Mimino666 Mimino666 commented Oct 14, 2016

Still fails when -a flag is used, i.e.: python manage.py dumpdata -a.
In that case, _base_manager is used, instead of _default_manager.

@tony

This comment has been minimized.

Copy link
Contributor

@tony tony commented Jul 11, 2017

Off-topic: As of django 1.11, there is no --natural

There is: --natural-foreign and --natural-primary. Which one(s) would be used if doing it correctly now?

@tony

This comment has been minimized.

Copy link
Contributor

@tony tony commented Jul 11, 2017

The other thing is as of django-polymorphic 1.2, django.1.11.3

If I do a dump-data on a parent model, I get all the data, except the _ptr_id. So polymorphic queries don't populate.

Also, -a won't seem to populate the queries on sub-models with dumpdata either.

@tony

This comment has been minimized.

Copy link
Contributor

@tony tony commented Jul 11, 2017

When using loaddata:

If I'm using post_save signals, I'll get an error the model isn't found (which is expected, no models are found if only the table data for the parent class is populated - django-polymorphic's manager needs to child table's _ptr_id to pull any data, even if using the root model).

But if I just hook into save on the parent model (after I do super(Model, self).save(**kwargs)), no error shows. It just populates the parent table. But no model information shows through ORM at all. Parent table populates data, but sub tables have no relational information.

Hope this helps.

@tony

This comment has been minimized.

Copy link
Contributor

@tony tony commented Jul 11, 2017

This is only with fixtures. By the way. If I create models on the fly through python or through the normal website, relationships are correctly created and show fine.

@tony

This comment has been minimized.

Copy link
Contributor

@tony tony commented Jul 11, 2017

Here's the good news: I do get the "polymorphic_ctype": data in dumpdata.

The bad news is loaddata doesn't actually help populate the relation. For instance, assume the core.Node parent and ask.Question subclassing it.

I'll get a dump with

    "polymorphic_ctype": [
      "ask",
      "question"
    ],

(yay). But upon loaddata, ask.Question's SQL table doesn't create the relation node_ptr_id.

Node.objects.all() and Question.objects.all() both return nothing.

Any ideas @vdboor ?

@ablakey

This comment has been minimized.

Copy link

@ablakey ablakey commented Sep 7, 2018

Just wanted to call out a related issue for anyone in the future that runs into it. The dumpdata hack that Polymorphic uses explicitly looks at sys.argv[1] for "dumpdata", which is what broke my code. I have modified the manage script to add a "track" to the commands. Eg. python manage local dumpdata python manage devel dumpdata which changed the argument order. This ended up being my problem, so popping the track name off argv after I loaded my desired track's settings fixed everything for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.