Error during South datamigration of raw string value

[mik3y]

This seems to be related to issue #33.

I have written a South datamigration which simply copies one table's JSONField values to another table. Although it works as expected for some types, it fails when the data being migrated/saved is a plain string, with the following stack trace:

   [...]
  File "migrations/0002_copy_data.py", line 12, in forwards
    new_row = orm['testapp.bar'].objects.create(data=row.data)
  File "site-packages/django/db/models/manager.py", line 157, in create
    return self.get_queryset().create(**kwargs)
  File "site-packages/django/db/models/query.py", line 320, in create
    obj = self.model(**kwargs)
  File "site-packages/django/db/models/base.py", line 407, in __init__
    setattr(self, field.attname, val)
  File "site-packages/jsonfield/subclassing.py", line 41, in __set__
    obj.__dict__[self.field.name] = self.field.pre_init(value, obj)
  File "site-packages/jsonfield/fields.py", line 80, in pre_init
    raise ValidationError(_("Enter valid JSON"))
django.core.exceptions.ValidationError: [u'Enter valid JSON']

I've created a minimal Django project which reproduces the issue: https://github.com/mik3y/jsonfield-datamigration-bug . It includes a database with a row triggering the problem; should be possible to check it out and run migrate.

[wmertens]

I'm encountering a similar issue.

I'm using Postgres 9.3, so the field is stored as a json field, and I store a simple string in the json field. The deserialization checks if the returned field is a string and if so, it tries to json.loads().

This is wrong; there should be a check higher up that turns of deserialization for postgres json fields.

[wmertens]

Workaround: Convert the field type to text
(alter table TABLENAME alter COLUMNNAME type text;)

[mik3y]

I also found a workaround, after staring at Django/JSONField for a while. In my south migration, I always serialize the the value to JSON first.

This works because the bug essentially means JSONField aggressively deserializes strings (and strings only) on save. Make everything a serialized string first, and JSONField will deserialize back to the original value, which may itself be a string.

Before (triggers bug):

      for row in orm['testapp.foo'].objects.all():
        new_row = orm['testapp.bar'].objects.create(data=row.data)

After:

      for row in orm['testapp.foo'].objects.all():
        new_row = orm['testapp.bar'].objects.create(data=json.dumps(row.data))

[bradjasper]

Thanks @mik3y and @wmertens, I will try to get this officially cleaned up, but glad you both found workarounds.

JSONField is likely moving back to TEXT columns anyway, as we've had a ton of problems trying to using the JSON column in PG. Here's a temp branch if you're interested: https://github.com/bradjasper/django-jsonfield/tree/postgresql

[wmertens]

In fact, changing the field to text doesn't fix it 100%, for example
get_or_create fails, probably because it assumes a JSON type.

To be robust, the field type should be detected at init time I think.
On Jun 6, 2014 9:23 PM, "Brad Jasper" notifications@github.com wrote:

Thanks @mik3y https://github.com/mik3y and @wmertens
https://github.com/wmertens, I will try to get this officially cleaned
up, but glad you both found workarounds.

JSONField is likely moving back to TEXT columns anyway, as we've had a ton
of problems trying to using the JSON column in PG. Here's a temp branch if
you're interested:
https://github.com/bradjasper/django-jsonfield/tree/postgresql

—
Reply to this email directly or view it on GitHub
#80 (comment)
.

[diwu1989]

I wonder if this bug is still reproducible after #85

[wmertens]

@bradjasper are you still going to remove JSON field support in PostGres? It would be great to have it removed so we can upgrade to PostGres 9.3...

[bradjasper]

This change has been made. 1.0.0 no longer supports a native JSON datatype on PostgreSQL.