Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
exportdata / loaddata cycle doesn't work, possibly due to abstract models? #89
So I did something that seems to be pretty simple to me, but maybe I'm just doing it wrong since its been a few versions since I worked with django, as what I think should be should be simple is failing in a way that suggests ether I have no idea what I am doing, or something is really wrong.
Once a manage.py cities_light has been done, I cant do a exportdata/loaddata cycle without the load cycle side failing to import anything due to a "Country matching query does not exist." error.
(1). Create new empty dkango project.
(3). Do the 2+ hour import of the city-light data.
(5). Delete the sqlite db to trick django into thinking it needs to be created again.
(6). I then do a full migration of my app so that it re-initalizes a new empty database with the correct schema that I had before.
(7). Now at this point the database should be EXACTLY the same schema as it was when I did the data export. Exactly,just with no data. Right?
(8). Now attempt to load the data from the saved city_light.json file into the database with the hope of never having to do another 2 hour long import of just city data again:
As you can see it seems to be failing to detect the models after loading the abstract ones.
I'm totally blocked, and I have no idea what to do. I have done everything according to the docs that I am aware of, and I expect to be able to export/import data at will, that does not work in my experience so far.
What I Am trying to do
I want my app to be initialized from migrations, as that is the new django way and I want to embrace the future. I don't want to have to download a large file form the internet and suffer though a huge deployment every time I deploy; I want to do it at most once, then have all the data in a django migration i just apply and am done with, that I can also check into our source control (git, in this case). That is what I am trying to do
Ok, i have reproduced the problem. It is linked to usage of abstract models, but looks like it is not a problem of abstract models itself. It is caused by the order of custom models.
Pay attention, models are declared in this order:
Create fixture (
Everything is ok for now.
As we can see, fixture reflect the order of our declared models: first goes City then Country and finally Region.
Now, delete db.sqlite3, create new by applying migrations, and try to loaddata. We'll get the same error, as in description of this issue.
Because some pre_save signals are connected to your models. And one of them, (exactly
How to fix?
So, as i see, nothing to do with cities_light package according to this issue.
By the way, even when custom models are used, original cities_light tables will be created because of migrations (in django >= 1.7):
They don't break anything, just empty tables, but it is not very good. I think swappable models is the better choice because of this.
The problem is you are making incorrect assumptions about how developers use this library, or indeed, most libraries for django from my personal experience.
I'm actually not using an order for models at all; each model is a separate file as its own object inside a 'models' package directory so I have no way to control the order they get declared.
Now django supports this and its even considered best programming practices to do anyway for keeping things DRY, For Separation of Concerns, To keep Loose Coupling, etc. It also guards against circular dependencies and other issues. In short I tend to keep to SOLID and thats a good thing. In fact I have never worked on a project that didn't REQUIRE that all models be in separate files. Its just good design, and no professional developer wants multiple models in the same file anyway as its harder to do multi-developer development on teams when everybody is trying to edit the same file.
So I have limited ability to order the loading via the packages init.py file or in the models that use them, but that is it. And if I reorder them for declaration as you suggest, it does not help.
So if cities-light doesn't support this django standard setup, then I would look to be a bug in in cities-light. A pretty big one, honestly.
It should also be noted that google is making this bug the top result for any errors I Google for cities-light; including a "Failed to populate slug Country.slug from name_ascii" error I'm now getting after attempting your fix by commenting out the signals.
This is of course after upgrading EVERYTHING and trying again. Latest django as of today, latest released cities-light as of today, etc.
If I comment out the default signals, my attempt to migrate also fails and the data is never added to the DB; the resulting db is only about a 168k in size and not the 10+ MB file I expect. As a result, the data is not loaded and my database is left in a corrupt state and I am forced to do everything over... meaning that I'm still blocked, months after I started this, because I'm not able to assure that if I add a model that uses data in cities light, that I will reference the same object that I intend to since updates could change primary ID's.
Right now, cities light doesn't support migrations in django and that is both a project and good emotion killer.; every time I have tried to work on it this library has given me problems, given me errors that take me back to this same exact thread in Google, and made me so angry, that I stop working on it for day or even weeks at a time.
I would love to work on this and stop that cycle, but I'm not sure what to do as every single attempt I have made to make progress on this has created a corrupt, unusable database that either does not have the data in it, or contains corrupted data that ultimately requires I restart from scratch. cities-light just doesn't seem to support the standard migrations life cycle. :(
I have been restarting ever and over and over and over and over on this, and I'm emotionally tired of doing so. I just want a fix, please. Anything I can do to help, just ask, but giving the models an explicit order or disabling signals and needing to modify code as part of deployment is not an option as we need to be able to do a "zero touch" automated deployments, and editing source code is antithetical to that sort of continuous integration.
Order matters, when you install objects from fixtures, as some pre_save signals are connected to cities_light models. These signals can access related objects (that could not be loaded yet). You can check the order by looking in your fixtures files.
What do you mean by "fails"? What traceback, what error message?
By the way, for simplicity you can just not override the default models. Such customisation surely can involve some issues, as you modify the code of library in fact.
Could you maybe reproduce your issue in a minimal test project ?