During import, table id will not be auto-incremented

[raratiru]

Running on:

• Django-1.6
• django-import-export-0.1.5
• postgresql.x86_64-9.2.5-1
• tablib-0.9.11
• python-2.7.5-9

I have successfully imported data from a csv file.

When I try to add more data manually, I receive the error:

duplicate key value violates unique constraint "mytable_pkey"
DETAIL:  Key (id)=(1) already exists.

From psql:

select max(id) from mytable;
  75

SELECT nextval('mytable_id_seq');
  2

Does django-import-export support the table's id auto-incrementing?

[bmihelac]

django-import-export uses standard Django ORM, I believe it auto-increment should be handled without any special work.

One question - did you specify id values in CSV or you leaved them empty?

[raratiru]

Thank you for the quick answer.

At the beginning, I did not specify the id values, having in my mind django ORM.

It raised an error, this is why I included the id values without evaluating the error type.

Should I give it a try again without including the id, when I return home at night, and I will report back.

[bmihelac]

Yes, please try without specifying id, and let me know which error was raised.

[raratiru]

resources.py

class RegionalUnitsResource(resources.ModelResource):

    class Meta(object):
        model = RegionalUnits

admin.py

class RegionalUnitsResourceAdmin(ImportExportModelAdmin):
    resource_class = RegionalUnitsResource

admin.site.register(RegionalUnits, RegionalUnitsResourceAdmin)

The error I get for each line, if I do not specify the id value is the following:

Errors

    Line number: 1 - KeyError(u'id',)

    Traceback (most recent call last):
    File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/resources.py", line 292, in import_data
    instance, new = self.get_or_init_instance(instance_loader, row)
    File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/resources.py", line 141, in get_or_init_instance
    instance = self.get_instance(instance_loader, row)
    File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/resources.py", line 138, in get_instance
    return instance_loader.get_instance(row)
    File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/instance_loaders.py", line 29, in get_instance
    params[field.attribute] = field.clean(row)
    File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/fields.py", line 47, in clean
    value = data[self.column_name]
    KeyError: u'id'

I thought of modifying the resources.py to the following:

class RegionalUnitsResource(resources.ModelResource):

    class Meta(object):
        model = RegionalUnits
        exclude = ('id', )

In which case, nothing happened. Silence and no result.

[bmihelac]

You should have id in a csv file, but it should also be empty for new
records.

let me know if this does not help.

regards,
B

On Fri, Dec 13, 2013 at 1:00 AM, raratiru notifications@github.com wrote:

resources.py

class RegionalUnitsResource(resources.ModelResource):

class Meta(object):
    model = RegionalUnits

admin.py

class RegionalUnitsResourceAdmin(ImportExportModelAdmin):
resource_class = RegionalUnitsResource

admin.site.register(RegionalUnits, RegionalUnitsResourceAdmin)

The error I get for each line, if I do not specify the id value is the
following:

Errors

Line number: 1 - KeyError(u'id',)

Traceback (most recent call last):
File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/resources.py", line 292, in import_data
instance, new = self.get_or_init_instance(instance_loader, row)
File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/resources.py", line 141, in get_or_init_instance
instance = self.get_instance(instance_loader, row)
File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/resources.py", line 138, in get_instance
return instance_loader.get_instance(row)
File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/instance_loaders.py", line 29, in get_instance
params[field.attribute] = field.clean(row)
File "/home/flyer/.virtualenvs/daea/lib/python2.7/site-packages/import_export/fields.py", line 47, in clean
value = data[self.column_name]
KeyError: u'id'

I thought of modifying the resources.py to the following:

class RegionalUnitsResource(resources.ModelResource):

class Meta(object):
    model = RegionalUnits
    exclude = ('id', )

In which case, nothing happened. Silence and no result.

—
Reply to this email directly or view it on GitHubhttps://github.com/bmihelac/django-import-export/issues/59#issuecomment-30474815
.

Bojan Mihelac
bmihelac@mihelac.org | T: +386(0)1 2321798
www.informatikamihelac.com

[raratiru]

Thank you, it works great!

[diptipsawant]

Hello, Django is new for me.
I am getting same error. "id" is my auto increment field I am getting above same error.
I used exclude(), import_id_fields but still getting same error. also added "id" in header with other header like email,first_name etc
Suggest solution. I created "UserResource"

Error is as follows:
Traceback (most recent call last):
File "/var/www/shaylex-root/env/local/lib/python2.7/site-packages/import_export/resources.py", line 342, in import_data
instance, new = self.get_or_init_instance(instance_loader, row)
File "/var/www/shaylex-root/env/local/lib/python2.7/site-packages/import_export/resources.py", line 174, in get_or_init_instance
instance = self.get_instance(instance_loader, row)
KeyError: 'id'

Code is as follows
from import_export.admin import ImportExportMixin, ImportExportModelAdmin
from import_export import resources
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from django.contrib.auth.models import Group
from django.contrib.sites.models import Site
from django.utils.translation import ugettext as _
from django.db import models
from django.forms import TextInput, Textarea
from rest_framework.authtoken.models import Token
from fx1_user.forms import UserChangeForm, UserRegistrationForm
from fx1_user.models import CustomUser

def move_to_paid(modeladmin, request, queryset):
for user in queryset:
CustomUser.objects.move_user_to_paid(user)
move_to_paid.short_description = _("Move selected users to paid")
def move_to_free(modeladmin, request, queryset):
for user in queryset:
CustomUser.objects.move_user_to_free(user)
move_to_free.short_description = _("Move selected users to free")

class UserResource(resources.ModelResource):
class Meta:
model = CustomUser
import_id_fields = ['id',]
#exclude = ('id', ) //onces I try this and onces import_id_field
fields = ('email','first_name','last_name','contact_number','is_subscribed')
def get_instance(self, instance_loader, row):
return False
def save_instance(self, instance, real_dry_run):
if not real_dry_run:
try:
obj = CustomUser.objects.get(some_val=instance.first_name)
# extra logic if object already exist
except NFCTag.DoesNotExist:
# create new object
obj = CustomUser(some_val=instance.first_name)
obj.save()
def before_import(self, dataset, dry_run):
if dataset.headers:
dataset.headers = [str(header).lower().strip() for header in dataset.headers]

class CustomUserAdmin(ImportExportMixin,UserAdmin):
resource_class = UserResource
# The forms to add and change user instances
form = UserChangeForm
add_form = UserRegistrationForm
actions = [move_to_free, move_to_paid]

# The fields to be used in displaying the User model.
# These override the definitions on the base UserAdmin
# that reference specific fields on auth.User.
list_display = ('email', 'created', 'is_admin', 'is_subscribed','special_user')
list_filter = ('is_admin',)
readonly_fields = (
    'created',
    'last_login',
    'owner',
)
fieldsets = (
    (None, {'fields': ('email', 'password',)}),
    (_('Permissions'), {'fields': ('is_admin', 'is_superuser', 'is_subscribed','special_user')}),
    (_('Important dates'), {'fields': ('last_login', 'created',)}),
    (_('Profile'), {'fields': ('profile_picture',
                               'first_name',
                               'last_name',
                               'contact_number',
                               'referral',
                               'owner',)}),
    (_('Subscription'), {'fields': ('subscription_type',
                                    'subscription_start',
                                    'subscription_end',
                                    'subscription_ongoing',)}),
)
formfield_overrides = {
    models.CharField: {'widget': TextInput(attrs={'size': '20'})},
    models.TextField: {'widget': Textarea(attrs={'rows': 4, 'cols': 40})},
    }
add_fieldsets = (
    (None, {
        'classes': ('wide',),
        'fields': (
            'email', 'password1', 'password2', 'subscription_type'
        )
    }),
)
search_fields = ('email',)
ordering = ('email',)
filter_horizontal = ()

admin.site.unregister(Token)
admin.site.register(CustomUser, CustomUserAdmin)

Thank you.

[int-ua]

Here is how to fix the database after this error:

from django.db import connection
# select a relevant line from the output of ./manage.py sqlsequencereset [appname]
sql = '''SELECT setval(pg_get_serial_sequence('"appname_modelname"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "appname_modelname";'''
cursor = connection.cursor()
cursor.execute(sql)

Using IDs for new records looks like a reasonable usage for [backward] compatibility with some other database.

[diptipsawant]

@int-ua thanks for quick reply but where to put above code in admin.py or model ?

[int-ua]

It's a one-time solution, you can execute in Django shell (./manage.py shell)

[diptipsawant]

sorry,but i don't understand. could you elaborate,please ?
and is there any effect on existing records of table ?

[int-ua]

Please search Django and SQL database of your choice documentations for pieces you don't understand.
I suggest you start with this link: https://docs.djangoproject.com/en/1.8/ref/django-admin/#shell

No, it should not affect any existing records.

[diptipsawant]

Thanks int-ua. I don't need to do above suggested solution. my problem is solved. I was not using 'id' in export filter.

[surajprasad]

Hi,
I have tried both the approaches mentioned above. But I am getting this error, I believe this is something to do with how my models are designed, but any help will be appreciated.

The error signature here.

Line number: 1 - 'NoneType' object is not iterable
OrderedDict([('ID', ''), ('STRATEGIE_REF_ID', '100'), ('IND_1', 'CLOSE'), ('IND_2', 'SMA200'), ('DESCRIPTION', '1000')])
Traceback (most recent call last):
File "/Users/Suraj/Django/django/bin/Ahes-src/import_export/resources.py", line 356, in import_data
instance, new = self.get_or_init_instance(instance_loader, row)
File "/Users/Suraj/Django/django/bin/Ahes-src/import_export/resources.py", line 186, in get_or_init_instance
instance = self.get_instance(instance_loader, row)
TypeError: 'NoneType' object is not iterable

I have multiple models and data import is to be done on a daily basis. So I would also like to understand why and how this error is there, so I can make sure it doesnt happen in future.

Will be grateful

[surajprasad]

I am getting the same exact error in other table / model imports as well. I think now that it is more of a configuration / settings issue.

Please help

[platzhersh]

I am having the same problem and I would like to add that I need to import / export the ids to keep the relationships between the imported / exported data rows. Also this problem only seems to occur when using postgres afaik.

I found this as a possibility to fix it manually:

select id from table order by id DESC LIMIT 1;
 id 
----
 11
alter sequence auth_group_id_seq restart with 12;

This will fix the problem, but I can't do this everytime and for every table I import data into..
The main reason why I import / export the id fields are:

1. keep relationships between data rows
2. make updating existing records easier

[drayendrff]

I have recently been experiencing the same problem as follow :

• I bulk import the old export plus one new row with the new id
• now I try to manually add one new value but it tels me the id is duplicate until I save as many times as I added new row in the bulk import
• I tried with no id in the column but get the error ValueError: Field 'id' expected a number but got ''.

[raratiru]

@drayendrff This is a very old issue, please open a new one.