Error importing uk data #48

Closed
gjstockham opened this Issue Nov 28, 2012 · 8 comments

Projects

None yet

3 participants

@gjstockham

I'm getting an error importing the uk data. I've tried the AMI instance, and a scripted install and get the same error on the command

./manage.py mapit_UK_import_boundary_line \
    --control=mapit.controls.first-gss --commit \
    `ls ../data/Boundary-Line/Data/*.shp|grep -v high_water`

The error happens at the same point every time:

  New area: CED None 1373 Wisbech North ED
  New area: CED None 42985 Petersfield ED
  New area: CED None 1281 Romsey ED
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 322, in handle
    label_output = self.handle_label(label, **options)
  File "/var/www/mapit/mapit/mapit/management/commands/mapit_UK_import_boundary_line.py", line 74, in handle_label
    m_name = m.names.get(type=name_type).name
  File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 349, in get
    % self.model._meta.object_name)
mapit.models.DoesNotExist: Name matching query does not exist.
@mhl
mySociety member

This is odd - I'll try to reproduce the problem. Could you let me know what version of Boundary-Line you are importing, and where you downloaded it from?

@gjstockham

I followed the instructions on http://code.mapit.mysociety.org/import/uk/ and downloaded from http://parlvid.mysociety.org:81/os/ with the live May2012 data

@mhl
mySociety member

I think you must be running the command without the --commit - it looks as if there's a bug when doing a "dry run" import of Boundary-Line with that dataset. (At least, I can only reproduce this without --commit, and it makes sense from the code.) If you just want to move forward, it should work with --commit - in the mean time, we'll work on a fix for the problem you found with a dry-run. Thank for reporting this - it's much appreciated!

@gjstockham

Ah, thanks. So much for playing it safe :) At least with the new AMI/ setup scripts etc it's so much easier to recreate from scratch!

@gjstockham gjstockham closed this Nov 28, 2012
@mhl mhl added a commit that referenced this issue Nov 28, 2012
@mhl mhl Fix an error with an initial Boundary-Line import without --commit
Thanks to Geoff Stockham for pointing out this problem.  This
fixes issue #48.

When we see a second polygon with the same unit_id and this
is a new import, a dry-run import will fail because the name
can't be found in the database from the unit_id - it was never
saved the first time around.  As with finding the name from
ons_code we should ignore this error if we're not saving to the
database.

(The example in the bug report is the second polygon in
Waterbeach ED in county_electoral_division_region.shp.)
22383f3
@gjstockham gjstockham reopened this Nov 28, 2012
@gjstockham

Sorry to reopen this, but with --commit, I get this error now:

  New area: CED None 42682 Bideford South and Hartland ED
  New area: CED None 42685 Bickleigh and Wembury ED
.Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 322, in handle
    label_output = self.handle_label(label, **options)
  File "/var/www/mapit/mapit/mapit/management/commands/mapit_UK_import_boundary_line.py", line 148, in handle_label
    save_polygons(self.unit_id_to_shape)
  File "/var/www/mapit/mapit/mapit/management/command_utils.py", line 56, in save_polygons
    m.polygons.create(polygon=must_be_two_d)
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 436, in create
    return super(RelatedManager, self.db_manager(db)).create(**kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 138, in create
    return self.get_query_set().create(**kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 360, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 553, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 1436, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 34, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
django.db.utils.IntegrityError: new row for relation "mapit_geometry" violates check constraint "enforce_srid_polygon"

This is on a fresh AMI instance

@mhl
mySociety member

I think this is a problem that @dracos discovered this morning - the AMI has an instance of MapIt that uses the SRID 4326, while for the UK it should be 27700. Could you try the following?

Edit /var/www/mapit/mapit/conf/general.yml and change the AREA_SRID line to read:

AREA_SRID: 27700

The schema migrations unfortunately depend on this setting, so you'll also need to roll back your database completely, and re-run the migrations:

cd /var/www/mapit/mapit/project
./manage.py migrate myapp zero
./manage.py migrate

Then you can go back to step 4 of the instructions here: http://code.mapit.mysociety.org/import/uk/

Sorry about this problem...

@dracos
mySociety member

Sounds likely. I am actually installing a new UK mapit as I write, so will be able to commit documentation fixes for this later on today.

@gjstockham

I had already set the SRID in general.yml correctly, but it appears that just rolling back the migrations using

./manage.py migrate mapit zero
./manage.py migrate

and starting again has solved the issue. Cheers

@gjstockham gjstockham closed this Nov 29, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment