Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use utf8mb4 character set by default for MySQL database #33608

Merged
merged 4 commits into from Sep 11, 2018

Conversation

@yahonda
Copy link
Contributor

@yahonda yahonda commented Aug 13, 2018

Summary

This pull request implements #33596. It includes these changes:

  • Replace utf8 character set with utf8mb4 to support supplementary characters including emoji
  • Removed utf8_unicode_ci collation from Active Record unit test databases to let MySQL server use the default collation for the character set
  • Bump the minimum version of MySQL to 5.7.9 and MariaDB to 10.2.2 to support utf8mb4 character set and 3072 bytes key length with InnoDB
  • Addressed Specified key was too long; max key length is 1000 bytes for MyISAM table in the test by using InnoDB storage engine
  • CI against MySQL 5.7
@rails-bot
Copy link

@rails-bot rails-bot commented Aug 13, 2018

r? @kamipo

(@rails-bot has picked a reviewer for you, use r? to override)

@yahonda
Copy link
Contributor Author

@yahonda yahonda commented Aug 14, 2018

@jeremy I have opened a work in progress PR for #33596. There is one failure with PostgreSQL 9.2 https://travis-ci.org/rails/rails/jobs/415702539 . I don't think it is relevant to my pull request.

@yahonda yahonda force-pushed the yahonda:use_utf8mb4 branch Aug 14, 2018
@yahonda
Copy link
Contributor Author

@yahonda yahonda commented Aug 14, 2018

Restarted CI by changing the last commit hash and found the failure with PostgreSQL 9.2 https://travis-ci.org/rails/rails/jobs/415715778 needs addressed by changing .travis.yml not to upgrade MySQL server if PostgreSQL 9.2 is configured or something like that.

Bottom line: All of CI against MySQL 5.7 is green.

@yahonda
Copy link
Contributor Author

@yahonda yahonda commented Aug 14, 2018

Another idea is dropping PostgreSQL 9.2 support for Rails 6 since PostgreSQL 9.2 itself already EOLed https://www.postgresql.org/support/versioning/ .

@yahonda yahonda force-pushed the yahonda:use_utf8mb4 branch 24 times, most recently Aug 14, 2018
@jeremy jeremy added this to the 6.0.0 milestone Sep 11, 2018
@jeremy
jeremy approved these changes Sep 11, 2018
Copy link
Member

@jeremy jeremy left a comment

Excellent. Thank you for this contribution @yahonda!

@jeremy jeremy merged commit d54d0c9 into rails:master Sep 11, 2018
2 checks passed
2 checks passed
codeclimate All good!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@yahonda
Copy link
Contributor Author

@yahonda yahonda commented Sep 11, 2018

Thanks for merging.
Let me have some time to work on a pull request to fall back to utf8 if utf8mb4 is not available by version or by configuration.

yahonda added a commit to yahonda/rails that referenced this pull request Sep 12, 2018
…longer index key prefix

Once rails#33608 merged If users create a new database using MySQL 5.1.x, it will fail to create databases
since MySQL 5.1 does not know `utf8mb4` character set.

This pull request removes `encoding: utf8mb4` from `mysql.yml.tt`
to let create_database method handles default character set by MySQL server version.

`supports_longer_index_key_prefix?` method will need to validate if MySQL 5.5 and 5.6 server configured
correctly to support longer index key prefix, but not yet.
yahonda added a commit to yahonda/rails that referenced this pull request Sep 12, 2018
…longer index key prefix

Once rails#33608 merged If users create a new database using MySQL 5.1.x, it will fail to create databases
since MySQL 5.1 does not know `utf8mb4` character set.

This pull request removes `encoding: utf8mb4` from `mysql.yml.tt`
to let create_database method handles default character set by MySQL server version.

`supports_longer_index_key_prefix?` method will need to validate if MySQL 5.5 and 5.6 server configured
correctly to support longer index key prefix, but not yet.
yahonda added a commit to yahonda/rails that referenced this pull request Sep 12, 2018
…longer index key prefix

Once rails#33608 merged If users create a new database using MySQL 5.1.x, it will fail to create databases
since MySQL 5.1 does not know `utf8mb4` character set.

This pull request removes `encoding: utf8mb4` from `mysql.yml.tt`
to let create_database method handles default character set by MySQL server version.

`supports_longer_index_key_prefix?` method will need to validate if MySQL 5.5 and 5.6 server configured
correctly to support longer index key prefix, but not yet.
yahonda added a commit to yahonda/rails that referenced this pull request Sep 13, 2018
Once rails#33608 merged If users create a new database using MySQL 5.1.x, it will fail to create databases
since MySQL 5.1 does not know `utf8mb4` character set.

This pull request removes `encoding: utf8mb4` from `mysql.yml.tt`
to let create_database method handles default character set by MySQL server version.

`supports_longer_index_key_prefix?` method will need to validate if MySQL 5.5 and 5.6 server configured
correctly to support longer index key prefix, but not yet.
yahonda added a commit to yahonda/rails that referenced this pull request Sep 13, 2018
…pported

Once rails#33608 merged If users create a new database using MySQL 5.1.x, it will fail to create databases
since MySQL 5.1 does not know `utf8mb4` character set.

This pull request removes `encoding: utf8mb4` from `mysql.yml.tt`
to let create_database method handles default character set by MySQL server version.

`supports_longer_index_key_prefix?` method will need to validate if MySQL 5.5 and 5.6 server configured
correctly to support longer index key prefix, but not yet.
yahonda added a commit to yahonda/rails that referenced this pull request Sep 13, 2018
…pported

Once rails#33608 merged If users create a new database using MySQL 5.1.x, it will fail to create databases
since MySQL 5.1 does not know `utf8mb4` character set.

This pull request removes `encoding: utf8mb4` from `mysql.yml.tt`
to let create_database method handles default character set by MySQL server version.

`supports_longer_index_key_prefix?` method will need to validate if MySQL 5.5 and 5.6 server configured
correctly to support longer index key prefix, but not yet.
@yahonda yahonda deleted the yahonda:use_utf8mb4 branch Sep 23, 2018
yahonda added a commit to yahonda/rails that referenced this pull request Dec 16, 2018
yahonda added a commit to yahonda/rails that referenced this pull request Dec 16, 2018
yahonda added a commit to yahonda/rails that referenced this pull request Dec 24, 2018
yahonda added a commit to yahonda/rails that referenced this pull request Dec 24, 2018
yahonda added a commit to yahonda/rails that referenced this pull request Dec 24, 2018
yahonda added a commit to yahonda/rails that referenced this pull request Dec 24, 2018
kamipo added a commit to kamipo/rails that referenced this pull request May 20, 2020
…e agnostic

5 years ago, I made dumping full table options at rails#17569, especially to
dump `ENGINE=InnoDB ROW_FORMAT=DYNAMIC` to use utf8mb4 with large key
prefix.

In that time, omitting the default engine `ENGINE=InnoDB` was not useful
since `ROW_FORMAT=DYNAMIC` always remains as long as using utf8mb4 with
large key prefix.

But now, MySQL 5.7.9 has finally changed the default row format to
DYNAMIC, utf8mb4 with large key prefix can be used without dumping the
default engine and the row format explicitly.

So now is a good time to make the default engine is omitted.

Before:

```ruby
create_table "accounts", options: "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```

After:

```ruby
create_table "accounts", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```

To entirely omit `:options` option to make schema agnostic, I've added
`:charset` and `:collation` table options to exclude `CHARSET` and
`COLLATE` from `:options`.

Fixes rails#26209.
Closes rails#29472.

See also rails#33608, rails#33853, and rails#34742.
kamipo added a commit to kamipo/rails that referenced this pull request May 23, 2020
…e agnostic

5 years ago, I made dumping full table options at rails#17569, especially to
dump `ENGINE=InnoDB ROW_FORMAT=DYNAMIC` to use utf8mb4 with large key
prefix.

In that time, omitting the default engine `ENGINE=InnoDB` was not useful
since `ROW_FORMAT=DYNAMIC` always remains as long as using utf8mb4 with
large key prefix.

But now, MySQL 5.7.9 has finally changed the default row format to
DYNAMIC, utf8mb4 with large key prefix can be used without dumping the
default engine and the row format explicitly.

So now is a good time to make the default engine is omitted.

Before:

```ruby
create_table "accounts", options: "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```

After:

```ruby
create_table "accounts", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```

To entirely omit `:options` option to make schema agnostic, I've added
`:charset` and `:collation` table options to exclude `CHARSET` and
`COLLATE` from `:options`.

Fixes rails#26209.
Closes rails#29472.

See also rails#33608, rails#33853, and rails#34742.
kamipo added a commit to kamipo/mysql2 that referenced this pull request Jan 26, 2021
mysql2 gem dropped testing for MySQL 5.1 which was released in Dec 2008
and already EOL in Dec 2013 at efa47a9.

If we no longer support MySQL 5.1, utf8 (utf8mb3) is not appropriate
default encoding for MySQL 5.5 or higher, utf8mb4 should be set for
that.

FYI, Rails 6.0 no longer support MySQL 5.1 and set utf8mb4 by default.

rails/rails#33608
rails/rails#33853
junaruga added a commit to junaruga/mysql2 that referenced this pull request Apr 15, 2021
mysql2 gem dropped testing for MySQL 5.1 which was released in Dec 2008
and already EOL in Dec 2013 at efa47a9.

If we no longer support MySQL 5.1, utf8 (utf8mb3) is not appropriate
default encoding for MySQL 5.5 or higher, utf8mb4 should be set for
that.

FYI, Rails 6.0 no longer support MySQL 5.1 and set utf8mb4 by default.

rails/rails#33608
rails/rails#33853
kamipo added a commit to kamipo/mysql2 that referenced this pull request Apr 22, 2021
mysql2 gem dropped testing for MySQL 5.1 which was released in Dec 2008
and already EOL in Dec 2013 at efa47a9.

If we no longer support MySQL 5.1, utf8 (utf8mb3) is not appropriate
default encoding for MySQL 5.5 or higher, utf8mb4 should be set for
that.

FYI, Rails 6.0 no longer support MySQL 5.1 and set utf8mb4 by default.

rails/rails#33608
rails/rails#33853
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants