Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to import schema with mysql utf8mb4 charset encoding. #6197

Closed
iolo opened this issue Dec 10, 2015 · 5 comments
Closed

failed to import schema with mysql utf8mb4 charset encoding. #6197

iolo opened this issue Dec 10, 2015 · 5 comments
Labels
bug [triage] something behaving unexpectedly
Milestone

Comments

@iolo
Copy link

iolo commented Dec 10, 2015

it looks like email column of users table and token column of accesstokens and refreshtokens table.

by the MySQL official document, indexed varchar should be 191 or smaller number of characters.

see also: http://stackoverflow.com/questions/20123824/how-can-i-alter-an-indexed-varchar255-from-utf8-to-utf8mb4-and-still-stay-unde

this is a simple patch:

diff --git a/core/server/data/schema.js b/core/server/data/schema.js
index 1ce4bbf..66cefc2 100644
--- a/core/server/data/schema.js
+++ b/core/server/data/schema.js
@@ -27,7 +27,7 @@ var db = {
             name: {type: 'string', maxlength: 150, nullable: false},
             slug: {type: 'string', maxlength: 150, nullable: false, unique: true},
             password: {type: 'string', maxlength: 60, nullable: false},
-            email: {type: 'string', maxlength: 254, nullable: false, unique: true, validations: {isEmail: true}},
+            email: {type: 'string', maxlength: 191, nullable: false, unique: true, validations: {isEmail: true}},
             image: {type: 'text', maxlength: 2000, nullable: true},
             cover: {type: 'text', maxlength: 2000, nullable: true},
             bio: {type: 'string', maxlength: 200, nullable: true},
@@ -182,14 +182,14 @@ var db = {
         },
         accesstokens: {
             id: {type: 'increments', nullable: false, primary: true},
-            token: {type: 'string', nullable: false, unique: true},
+            token: {type: 'string', maxlength: 191, nullable: false, unique: true},
             user_id: {type: 'integer', nullable: false, unsigned: true, references: 'users.id'},
             client_id: {type: 'integer', nullable: false, unsigned: true, references: 'clients.id'},
             expires: {type: 'bigInteger', nullable: false}
         },
         refreshtokens: {
             id: {type: 'increments', nullable: false, primary: true},
-            token: {type: 'string', nullable: false, unique: true},
+            token: {type: 'string', maxlength: 191, nullable: false, unique: true},
             user_id: {type: 'integer', nullable: false, unsigned: true, references: 'users.id'},
             client_id: {type: 'integer', nullable: false, unsigned: true, references: 'clients.id'},
             expires: {type: 'bigInteger', nullable: false}

FYI, full debug logs here:

Migrations: Database initialisation required for version 004
Migrations: Creating tables...
Migrations: Creating table: posts
Migrations: Creating table: users

ERROR: ER_TOO_LONG_KEY: Specified key was too long; max key length is 767 bytes

 Error: ER_TOO_LONG_KEY: Specified key was too long; max key length is 767 bytes
    at Query.Sequence._packetToError (/home/deploy/ghost/Ghost/node_modules/mysql/lib/protocol/sequences/Sequence.js:30:14)
    at Query.ErrorPacket (/home/deploy/ghost/Ghost/node_modules/mysql/lib/protocol/sequences/Query.js:82:18)
    at Protocol._parsePacket (/home/deploy/ghost/Ghost/node_modules/mysql/lib/protocol/Protocol.js:202:24)
    at Parser.write (/home/deploy/ghost/Ghost/node_modules/mysql/lib/protocol/Parser.js:62:12)
    at Protocol.write (/home/deploy/ghost/Ghost/node_modules/mysql/lib/protocol/Protocol.js:37:16)
    at Socket.<anonymous> (/home/deploy/ghost/Ghost/node_modules/mysql/lib/Connection.js:72:28)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)
    at TCP.onread (net.js:523:20)
    --------------------
    at Protocol._enqueue (/home/deploy/ghost/Ghost/node_modules/mysql/lib/protocol/Protocol.js:110:48)
    at Connection.query (/home/deploy/ghost/Ghost/node_modules/mysql/lib/Connection.js:165:25)
    at /home/deploy/ghost/Ghost/node_modules/knex/lib/dialects/mysql/runner.js:42:16
    at tryCatcher (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._resolveFromResolver (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/promise.js:480:31)
    at new Promise (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/promise.js:70:37)
    at Runner_MySQL.<anonymous> (/home/deploy/ghost/Ghost/node_modules/knex/lib/dialects/mysql/runner.js:41:10)
    at Runner_MySQL.tryCatcher (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/util.js:26:23)
    at Runner_MySQL._query (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/method.js:15:34)
    at Runner_MySQL.<anonymous> (/home/deploy/ghost/Ghost/node_modules/knex/lib/runner.js:125:15)
    at Runner_MySQL.tryCatcher (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/util.js:26:23)
    at Runner_MySQL.query (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/method.js:15:34)
    at Runner_MySQL.<anonymous> (/home/deploy/ghost/Ghost/node_modules/knex/lib/runner.js:134:19)
    at Runner_MySQL.tryCatcher (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/util.js:26:23)
    at ReductionPromiseArray._promiseFulfilled (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/reduce.js:109:18)
    at Promise._settlePromiseAt (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/promise.js:586:26)
    at Promise._settlePromises (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/promise.js:697:14)
    at Async._drainQueue (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/home/deploy/ghost/Ghost/node_modules/bluebird/js/main/async.js:15:14)
    at processImmediate [as _immediateCallback] (timers.js:383:17)
@iolo
Copy link
Author

iolo commented Dec 10, 2015

I've just noticed, my patch causes another problem

Value in [accesstokens.token] exceeds maximum length of 191 characters.

I need more digging :'(

@iolo
Copy link
Author

iolo commented Dec 10, 2015

another dirty patch here:

--- a/core/server/middleware/oauth.js
+++ b/core/server/middleware/oauth.js
@@ -13,7 +13,7 @@ function exchangeRefreshToken(client, refreshToken, scope, done) {
             return done(new errors.NoPermissionError('Invalid refresh token.'), false);
         } else {
             var token = model.toJSON(),
-                accessToken = utils.uid(256),
+                accessToken = utils.uid(191),
                 accessExpires = Date.now() + utils.ONE_HOUR_MS,
                 refreshExpires = Date.now() + utils.ONE_WEEK_MS;

@@ -46,8 +46,8 @@ function exchangePassword(client, username, password, scope, done) {
         // Validate the user
         return models.User.check({email: username, password: password}).then(function then(user) {
             // Everything validated, return the access- and refreshtoken
-            var accessToken = utils.uid(256),
-                refreshToken = utils.uid(256),
+            var accessToken = utils.uid(191),
+                refreshToken = utils.uid(191),
                 accessExpires = Date.now() + utils.ONE_HOUR_MS,
                 refreshExpires = Date.now() + utils.ONE_WEEK_MS;

I guess, we need more elegant patch to close this issue.

@justinvoelker
Copy link

Seems the best way to go about this would be to leave the field lengths as-is but set the index length to something smaller (since this only appears to be an issue with the unique index on those few fields). Surely the first 191 (or even 150) characters of the tokens and email address would be unique even if the value is longer.

The only problem is that I don't see where knex has the ability to specify an index length of anything other than the field max length.

@ErisDS
Copy link
Member

ErisDS commented Jan 25, 2016

Knex does not currently provide us with a way to change field lengths (see #4134). Given the existing outstanding issues with utf-8 (see #5945) and the fact that this is pretty edge-case I think it's unlikely that this is going to get fixed in the near future.

I'd love to see someone work on providing a way to change field lengths in knex so that we can start work on things like this.

@ErisDS ErisDS added the bug [triage] something behaving unexpectedly label Jan 25, 2016
@kirrg001 kirrg001 added this to the 1.0.0-alpha.2 milestone Sep 19, 2016
kirrg001 added a commit to kirrg001/Ghost that referenced this issue Sep 20, 2016
✨  use utf8mb4 encoding

closes TryGhost#5519, closes TryGhost#6197

- default encoding is utf8mb4 instead of utf8
- support emojis
- read here why: http://dev.mysql.com/doc/refman/5.7/en/innodb-restrictions.html
- read here why: https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-conversion.html
kirrg001 added a commit to kirrg001/Ghost that referenced this issue Sep 20, 2016
@ErisDS ErisDS closed this as completed in 32a5982 Sep 20, 2016
@kirrg001 kirrg001 modified the milestones: 1.0.0-alpha.1, 1.0.0-alpha.2 Sep 21, 2016
@iolo
Copy link
Author

iolo commented Sep 22, 2016

💯 great!

mixonic pushed a commit to mixonic/Ghost that referenced this issue Oct 28, 2016
geekhuyang pushed a commit to geekhuyang/Ghost that referenced this issue Nov 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug [triage] something behaving unexpectedly
Projects
None yet
Development

No branches or pull requests

4 participants