Migration fails with utf8mb4 charset

[ckoehn]

Environment:

• python (3.4.2)
• django (1.8.16)
• mysqlclient (1.3.7)
• MariaDB (10.0.30)

Steps to reproduce:

• CREATE DATABASE utf8mb4_test DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
• python manage.py migrate

Output:

...
Running migrations:
  Rendering model states... DONE
  Applying auditlog.0001_initial... OK
  Applying auditlog.0002_auto_support_long_primary_keys... OK
  Applying auditlog.0003_logentry_remote_addr... OK
  Applying auditlog.0004_logentry_detailed_object_repr... OK
  Applying auditlog.0005_logentry_additional_data_verbose_name... OK
  Applying auditlog.0006_object_pk_index...Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 217, in execute
    res = self._query(query)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 378, in _query
    rowcount = self._do_query(q)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 341, in _do_query
    db.query(q)
  File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1071, 'Specified key was too long; max key length is 767 bytes')

The error is caused by the length (255) of the object_pk field. The InnoDB storage engine has a maximum index length of 767 bytes. This leads to 255 indexable chars with utf8 and 191 with utf8mb4.

[jjkester]

Given the maximum length of 767 bytes this error makes sense. However, it somewhat surprises me that even with utf8mb4 a VARCHAR can still be 255 characters but an index fails on this.

From the top of my head I can think of two solutions:

1. Optionally lower the length of the object_pk field
2. Optionally disable the index of the object_pk field

At the moment I think I like the first one a bit more because it does not result in very poor performance. Also, since primary keys are indexed we can assume that a primary key from a database using utf8mb4 is not longer than the mentioned 191 characters. (The issue may still be that the logs are stored in a separate database with another character set...)

This at least is an interesting problem, given the issues with other utf8 character sets in MySQL/MariaDB I think that the mb4 variant is/will become quite popular so this has to be addressed somehow.