exportdata / loaddata cycle doesn't work, possibly due to abstract models?

[duaneking]

So I did something that seems to be pretty simple to me, but maybe I'm just doing it wrong since its been a few versions since I worked with django, as what I think should be should be simple is failing in a way that suggests ether I have no idea what I am doing, or something is really wrong.

Problem

Once a manage.py cities_light has been done, I cant do a exportdata/loaddata cycle without the load cycle side failing to import anything due to a "Country matching query does not exist." error.

Software Used

python 2.7
django 1.8 - yesterdays release of the new version
django-cities-light - pulled this form pip yesterday

Repro steps

(1). Create new empty dkango project.
(2). Add models to the first app of the project. Make sure to add Country, Region, and City models that literally only look like this for each of them:

from cities_light.abstract_models import AbstractCountry
from cities_light.receivers import connect_default_signals


class Country(AbstractCountry):
    pass
connect_default_signals(Country)

(3). Do the 2+ hour import of the city-light data.
(4). Do an exportdata to export all the data in the db a city_light.json file:

python manage.py dumpdata --all > city_light.json

(5). Delete the sqlite db to trick django into thinking it needs to be created again.

 C:\Users\Duane\Dropbox\code\github\mouser>del db.sqlite3

(6). I then do a full migration of my app so that it re-initalizes a new empty database with the correct schema that I had before.

C:\Users\Duane\Dropbox\code\github\mouser>python manage.py migrate
C:\Python27\lib\site-packages\south\modelsinspector.py:20: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes.
  from django.contrib.contenttypes import generic

Operations to perform:
  Synchronize unmigrated apps: staticfiles, messages
  Apply all migrations: sessions, admin, auth, launcher, contenttypes, cities_light
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying cities_light.0001_initial... OK
  Applying cities_light.0002_city... OK
  Applying cities_light.0003_auto_20141120_0342... OK
  Applying cities_light.0004_auto_20150403_2001... OK
  Applying launcher.0001_initial... OK
  Applying launcher.0002_auto_20150403_2001... OK
  Applying sessions.0001_initial... OK

(7). Now at this point the database should be EXACTLY the same schema as it was when I did the data export. Exactly,just with no data. Right?

(8). Now attempt to load the data from the saved city_light.json file into the database with the hope of never having to do another 2 hour long import of just city data again:

C:\Users\Duane\Dropbox\code\github\mouser>python manage.py loaddata -v 3 city_light.json
C:\Python27\lib\site-packages\south\modelsinspector.py:20: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes.
  from django.contrib.contenttypes import generic

Loading 'city_light' fixtures...
Checking 'C:\Users\Duane\Dropbox\code\github\mouser\launcher\fixtures' for fixtures...
No fixture 'city_light' in 'C:\Users\Duane\Dropbox\code\github\mouser\launcher\fixtures'.
Checking 'C:\Users\Duane\Dropbox\code\github\mouser' for fixtures...
Installing json fixture 'city_light' from 'C:\Users\Duane\Dropbox\code\github\mouser'.
Traceback (most recent call last):
  File "manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 338, in execute_from_command_line
        utility.execute()
      File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 330, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "C:\Python27\lib\site-packages\django\core\management\base.py", line 390, in run_from_argv
        self.execute(*args, **cmd_options)
      File "C:\Python27\lib\site-packages\django\core\management\base.py", line 441, in execute
    output = self.handle(*args, **options)
      File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 60, in handle
        self.loaddata(fixture_labels)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 90, in loaddata
    self.load_label(fixture_label)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 147, in load_label
    obj.save(using=self.using)
  File "C:\Python27\lib\site-packages\django\core\serializers\base.py", line 173, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "C:\Python27\lib\site-packages\django\db\models\base.py", line 734, in save_base
    update_fields=update_fields)
  File "C:\Python27\lib\site-packages\django\dispatch\dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "C:\Python27\lib\site-packages\cities_light\receivers.py", line 28, in set_display_name
    instance.display_name = instance.get_display_name()
  File "C:\Python27\lib\site-packages\cities_light\abstract_models.py", line 114, in get_display_name    
    return '%s, %s' % (self.name, self.country.name)
  File "C:\Python27\lib\site-packages\django\db\models\fields\related.py", line 602, in __get__
    rel_obj = qs.get()
  File "C:\Python27\lib\site-packages\django\db\models\query.py", line 334, in get
    self.model._meta.object_name
launcher.models.country.DoesNotExist: Problem installing fixture     'C:\Users\Duane\Dropbox\code\github\mouser\city_light.json': Country matching query does not exist.

As you can see it seems to be failing to detect the models after loading the abstract ones.

I'm totally blocked, and I have no idea what to do. I have done everything according to the docs that I am aware of, and I expect to be able to export/import data at will, that does not work in my experience so far.

What I Am trying to do

I want my app to be initialized from migrations, as that is the new django way and I want to embrace the future. I don't want to have to download a large file form the internet and suffer though a huge deployment every time I deploy; I want to do it at most once, then have all the data in a django migration i just apply and am done with, that I can also check into our source control (git, in this case). That is what I am trying to do

[st4lk]

Hi Duane!
Precise, have you specified CITIES_LIGHT_APP_NAME = 'yourapp' in your settings?
'yourapp' here must be the name of the app, where you define your custom cities_light models (check docs for details).
By the way, if you don't need to customise the cities_light models, you can just ignore abstract_models, i.e. don't inherit them.

[duaneking]

Yes I have. I have set CITIES_LIGHT_APP_NAME to the app name in question in the settings file.

I'm using my own version of the models so that can support editing them later with migrations.

[jpic]

@st4lk could you reproduce it ?

I'm working on a new app (django-referendum) and i'm testing django-swappable-models, it seems pretty well supported by migrations. Should we try that ?

[st4lk]

@jpic i'll try to reproduce in next couple of days
As for django-swappable-models, it is interesting, will be good to try. Currently can't say pros and cons for this, but it looks good.

[st4lk]

Ok, i have reproduced the problem. It is linked to usage of abstract models, but looks like it is not a problem of abstract models itself. It is caused by the order of custom models.
So, here what i tried. My custom main/models.py:

from cities_light.abstract_models import AbstractCountry, AbstractRegion, AbstractCity
from cities_light.receivers import connect_default_signals

class City(AbstractCity):
    pass
connect_default_signals(City)


class Country(AbstractCountry):
    pass
connect_default_signals(Country)


class Region(AbstractRegion):
    pass
connect_default_signals(Region)

Pay attention, models are declared in this order: City, Country, Region.
Now, fill database:

python manage.py cities_light --force-import-all

Create fixture (--indent 4 just for readability):

python manage.py dumpdata --all --indent 4 > city_light.json

Everything is ok for now.
Lets look at the beginning of city_light.json. It will contain something like this:

[
// some contenttypes
{
    "fields": {
        "model": "city",
        "app_label": "main"
    },
    "model": "contenttypes.contenttype",
    "pk": 7
},
{
    "fields": {
        "model": "country",
        "app_label": "main"
    },
    "model": "contenttypes.contenttype",
    "pk": 8
},
{
    "fields": {
        "model": "region",
        "app_label": "main"
    },
    "model": "contenttypes.contenttype",
    "pk": 9
},
{
    "fields": {
        "display_name": "Yerres, \u00cele-de-France, France",
        "name": "Yerres",
        "country": 1,
        "region": 11,
        "alternate_names": "",
        "search_names": "yerresfrance yerresiledefrancefrance",
        "longitude": "2.49338",
        "geoname_id": 2967245,
        "feature_code": "PPL",
        "name_ascii": "Yerres",
        "latitude": "48.71785",
        "slug": "yerres",
        "population": 28897
    },
    "model": "main.city",
    "pk": 1
},
// other cities ...
{
    "fields": {
        "code2": "FR",
        "code3": "FRA",
        "name": "France",
        "tld": "fr",
        "alternate_names": "",
        "continent": "EU",
        "geoname_id": 3017382,
        "phone": "33",
        "name_ascii": "France",
        "slug": "france"
    },
    "model": "main.country",
    "pk": 1
},
// other countries ...
{
    "fields": {
        "display_name": "Rh\u00f4ne-Alpes, France",
        "name": "Rh\u00f4ne-Alpes",
        "country": 1,
        "alternate_names": "Rh\u00f4ne-Alpes Region",
        "geoname_id": 2983751,
        "geoname_code": "B9",
        "name_ascii": "Rhone-Alpes",
        "slug": "rhone-alpes"
    },
    "model": "main.region",
    "pk": 1
},
// other regions ...
// and some other django models, not interesting here ...
]

As we can see, fixture reflect the order of our declared models: first goes City then Country and finally Region.

Now, delete db.sqlite3, create new by applying migrations, and try to loaddata. We'll get the same error, as in description of this issue.

Why?

Because some pre_save signals are connected to your models. And one of them, (exactly set_display_name) will cause access to model by ForeignKey. Before saving the city models this signal receiver will try to get city.country by foreignkey. But, as we can see for fixture, City will be loaded first, whereas Country is not in the database yet. And we get DoesNotExist error.

How to fix?

1. Simple but not beautiful solution. Just comment the signals connection to your models:

   # connect_default_signals(Country)
   # connect_default_signals(City)
   # connect_default_signals(Region)
   

   when you do loaddata and uncomment after (all fields already was populated and saved in fixtures).

2. Good, but complicated solution. Declare you custom cities_light models in the same order, as cities_light package does:

   class Country(AbstractCountry): pass
   class Region(AbstractRegion): pass
   class City(AbstractCity): pass
   

   Now, it is needed to create fixture again (so models will be saved in correct order).
   Keep in mind, that your database is not changed, you just change the order of models declaration and create new fixture. In that case, when you'll delete the database and create new one (empty), database will be filled with contenttypes objects automatically (matters, because flag --all was used).
   But, as order of our models is changed, those contenttypes objects will also have different order and consequently other ids (different from our fixture, that was created from another database).

   If we will try to loaddata now, following error probably gonna to happen:

   django.db.utils.IntegrityError:
       Problem installing fixture '.../city_light.json':
       Could not load contenttypes.ContentType(pk=7):
       UNIQUE constraint failed: django_content_type.app_label, django_content_type.model
   

   To avoid it, we must manually set correct ids in our city_light.json for contenttypes.
   Or, after changing order of custom cities_light models, delete database, do
   python manage.py cities_light --force-import-all again and make fixture from this database. In that case no manual work with contenttypes is needed.

So, as i see, nothing to do with cities_light package according to this issue.
Maybe we just need to mention in the docs, that order of custom models matters in case of fixtures and signals.

[st4lk]

By the way, even when custom models are used, original cities_light tables will be created because of migrations (in django >= 1.7):

Applying cities_light.0001_initial... OK
Applying cities_light.0002_city... OK
Applying cities_light.0003_auto_20141120_0342... OK
Applying cities_light.0004_auto_20150403_2001... OK

They don't break anything, just empty tables, but it is not very good. I think swappable models is the better choice because of this.
I'll work at this direction soon.

[duaneking]

The problem is you are making incorrect assumptions about how developers use this library, or indeed, most libraries for django from my personal experience.

I'm actually not using an order for models at all; each model is a separate file as its own object inside a 'models' package directory so I have no way to control the order they get declared.

Now django supports this and its even considered best programming practices to do anyway for keeping things DRY, For Separation of Concerns, To keep Loose Coupling, etc. It also guards against circular dependencies and other issues. In short I tend to keep to SOLID and thats a good thing. In fact I have never worked on a project that didn't REQUIRE that all models be in separate files. Its just good design, and no professional developer wants multiple models in the same file anyway as its harder to do multi-developer development on teams when everybody is trying to edit the same file.

So I have limited ability to order the loading via the packages init.py file or in the models that use them, but that is it. And if I reorder them for declaration as you suggest, it does not help.

So if cities-light doesn't support this django standard setup, then I would look to be a bug in in cities-light. A pretty big one, honestly.

It should also be noted that google is making this bug the top result for any errors I Google for cities-light; including a "Failed to populate slug Country.slug from name_ascii" error I'm now getting after attempting your fix by commenting out the signals.

This is of course after upgrading EVERYTHING and trying again. Latest django as of today, latest released cities-light as of today, etc.

If I comment out the default signals, my attempt to migrate also fails and the data is never added to the DB; the resulting db is only about a 168k in size and not the 10+ MB file I expect. As a result, the data is not loaded and my database is left in a corrupt state and I am forced to do everything over... meaning that I'm still blocked, months after I started this, because I'm not able to assure that if I add a model that uses data in cities light, that I will reference the same object that I intend to since updates could change primary ID's.

Right now, cities light doesn't support migrations in django and that is both a project and good emotion killer.; every time I have tried to work on it this library has given me problems, given me errors that take me back to this same exact thread in Google, and made me so angry, that I stop working on it for day or even weeks at a time.

I would love to work on this and stop that cycle, but I'm not sure what to do as every single attempt I have made to make progress on this has created a corrupt, unusable database that either does not have the data in it, or contains corrupted data that ultimately requires I restart from scratch. cities-light just doesn't seem to support the standard migrations life cycle. :(

I have been restarting ever and over and over and over and over on this, and I'm emotionally tired of doing so. I just want a fix, please. Anything I can do to help, just ask, but giving the models an explicit order or disabling signals and needing to modify code as part of deployment is not an option as we need to be able to do a "zero touch" automated deployments, and editing source code is antithetical to that sort of continuous integration.

[st4lk]

I'm actually not using an order for models at all; each model is a separate file as its own object inside a > 'models' package directory so I have no way to control the order they get declared.

Order matters, when you install objects from fixtures, as some pre_save signals are connected to cities_light models. These signals can access related objects (that could not be loaded yet). You can check the order by looking in your fixtures files.

If I comment out the default signals, my attempt to migrate also fails and the data is never added to the DB

What do you mean by "fails"? What traceback, what error message?

By the way, for simplicity you can just not override the default models. Such customisation surely can involve some issues, as you modify the code of library in fact.
So by choosing the customisation you should keep in mind, that this is an advanced topic.

[jpic]

Could you maybe reproduce your issue in a minimal test project ?

http://docs.yourlabs.org/en/latest/guidelines.html#yourlabs-community-guidelines

Thanks !

[amalsalimcode]

i found another post that said use -e flag to avoid this unique constraint exception

eg: python manage.py dumpdata -e --all

[masajid390]

@st4lk, Thank you, you're right. Order matters and it should be:
Country, Region and City

[duaneking]

Cities are in Regions, Regions are in a Country... That makes perfect sense. But it was not how the system handled it.

[rubenanapu]

Today (24th, April, 2020) I had a similar problem with Django 2.2 (nothing related to django-cities-light)

My fixtures file was as simple as this:

[
{
    "model": "contenttypes.contenttype",
    "pk": 1,
    "fields": {
        "app_label": "admin",
        "model": "logentry"
    }
},
{
    "model": "contenttypes.contenttype",
    "pk": 2,
    "fields": {
        "app_label": "auth",
        "model": "permission"
    }
}]

When I ran ./manage.py loaddata initial_data.json, I got:

django.db.utils.IntegrityError: Problem installing fixture '/home/user/reponame/projectname/initial_data.json': Could not load contenttypes.ContentType(pk=2): UNIQUE constraint failed: django_content_type.app_label, django_content_type.model

What I did to make it work was just rename the pk to id for the contenttypes.contenttype model. After that, the migration worked as expected.

./manage.py loaddata initial_data.json 
Installed 2 object(s) from 1 fixture(s)

After the change, my initial_data.json file was:

[
{
    "model": "contenttypes.contenttype",
    "id": 1,
    "fields": {
        "app_label": "admin",
        "model": "logentry"
    }
},
{
    "model": "contenttypes.contenttype",
    "id": 2,
    "fields": {
        "app_label": "auth",
        "model": "permission"
    }
}]

It is worth mentioning that my original initial_dataj.json has many other models, but renaming pk to id only for the contenttypes.contenttype solved my problem.