Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
utf8mb4 #367
utf8mb4 #367
Comments
|
It also appears that github cannot handle 4byte UTF-8 either |
|
+1 on this issue, I was not able to specify |
|
This error hit me when I was upgrading from Ruby 1.8.7 to Ruby 1.9.3. |
|
Are you using mysql2 0.3.11 or 0.3.12bX? utf8mb4 is not present in 0.3.11. |
|
Also, as a workaround I had to monkey patch I simply renamed the default utf8 to utf8mb4 in order to get the mysql client to work: <charset name="utf8mb4">
<family>Unicode</family>
<description>UTF-8 Unicode</description>
<alias>utf-8</alias>
<collation name="utf8_general_ci" id="33">
<flag>primary</flag>
<flag>compiled</flag>
</collation>
<collation name="utf8_bin" id="83">
<flag>binary</flag>
<flag>compiled</flag>
</collation>
</charset>I also wrote this script to help me convert my databases. It only outputs DDL statements and does nto automatically do anything. I wrote it so I could paste the results into my query window and execute them after review... use with caution and at your own risk. Copy and paste into utf8mb4helper.rb and alter the settings in top of the script and then run # ruby script to generate ddl statements to convert utf8 to utf8mb4
# you will need ruby, rubygems and the mysql2 gem to run this script
# => gem install mysql2 --no-rdoc --no-ri
require 'rubygems'
require 'mysql2'
### settings
dbhost='localhost'
dbuser='root'
dbuserpwd='password'
database='testdbname'
charset='utf8mb4'
collation='utf8mb4_unicode_ci'
### these will store the statements
no_index_and_data_less_than_191=""
alter_tables=""
alter_column_sizes=""
needs_index_resized=""
no_index_but_data_greater_than_191=""
has_index_but_less_than_191 = ""
### connect to database
con = Mysql2::Client.new(:host=>dbhost,:username=>dbuser,:password=>dbuserpwd)
# For each table in database
tables = con.query("SELECT table_name from information_schema.tables where table_schema = '" + database + "';")
tables.each do |h|
# check and update columns
columns = con.query("SHOW FULL COLUMNS FROM `" + database + "`.`" + h["table_name"] + "`;")
columns.each do |c|
# if the column has an index, set the max field size to 191
checkindex=con.query("SELECT * FROM information_schema.statistics WHERE table_schema = '#{database}' AND table_name = '#{h["table_name"]}' AND column_name = '#{c["Field"]}';")
if c["Type"] =~ /^varchar/
# varchar.. check the size
maxsize = con.query("SELECT MAX(LENGTH(`#{c["Field"]}`)) FROM `#{database}`.`#{h["table_name"]}`;")
size = maxsize.first.each_value.collect.first || 0
if checkindex.size > 0
# varchar with an index
if size > 191
needs_index_resized << "/* ALTER TABLE `#{h["table_name"]}` CHANGE `#{c["Field"]}` `#{c["Field"]}` varchar(191); */\n"
else
has_index_but_less_than_191 << "ALTER TABLE `#{h["table_name"]}` CHANGE `#{c["Field"]}` `#{c["Field"]}` varchar(191);\n"
end
else # no index
utf8 = con.query("SELECT character_set_name FROM information_schema.`COLUMNS` C WHERE table_schema = '#{database}' AND table_name = '#{h["table_name"]}' AND column_name = '#{c["Field"]}' AND character_set_name='utf8';")
if utf8.size > 0 # utf8 varchar
if size > 191
no_index_but_data_greater_than_191 << "ALTER TABLE `#{h["table_name"]}` CHANGE `#{c["Field"]}` `#{c["Field"]}` #{c["Type"]} CHARACTER SET '#{charset}' COLLATE '#{collation}';\n"
else
no_index_and_data_less_than_191 << "ALTER TABLE `#{h["table_name"]}` CHANGE `#{c["Field"]}` `#{c["Field"]}` #{c["Type"]} CHARACTER SET '#{charset}' COLLATE '#{collation}';\n"
end
end
end
end
end
# check and update table
alter_tables << "ALTER TABLE `#{h["table_name"]}` CONVERT TO CHARACTER SET '#{charset}' COLLATE '#{collation}';\n"
end
puts "use `#{database}`;"
puts "/* ############ THESE COLUMNS HAVE INDEXES BUT NO DATA > THAN 191-- MAY NEED TO REBUILD INDEX ############ */"
puts has_index_but_less_than_191
puts "\n"
puts "/* ############ THESE COLUMNS HAVE INDEXES AND DATA > 191.. ALTERING THEM WOULD TRUNCATE DATA!!! RESIZE INDEX */"
puts needs_index_resized
puts "\n"
puts "/* ############ THESE COLUMNS DO NOT HAVE AN INDEX AND BUT HAVE DATA > 191 -- BE AWARE OF FUTURE INDICES ############ */"
puts no_index_but_data_greater_than_191
puts "\n"
puts "/* ############ THESE COLUMNS DO NOT HAVE AN INDEX AND CURRENTLY DO NOT HAVE DATA > 191 ############ */"
puts no_index_and_data_less_than_191
puts "\n"
puts "/* ############ CONVERT TABLES TO UTF8MB4 ############ */"
puts alter_tables
puts "\n"
puts "/* ############ CONVERT DATABASE ########### */"
puts "ALTER DATABASE `#{database}` CHARACTER SET = utf8mb4 COLLATE = `#{collation}`;"
puts "\n"
puts "/* ############ SOME HELPFULL COMMANDS TO RUN BEFORE AND/OR AFTER ########### */"
puts "# use `#{database}`; show session variables like 'character%';"
puts "# set names utf8mb4; use `#{database}`; show session variables like 'character%';"
puts "# select `SOMECOLUMN` from `#{database}` where NOT HEX(`SOMECOLUMN`) REGEXP '^([0-7][0-9A-F])*$'"
con.close |
|
Does this resolve the issue, may I close it? |
|
Upgrading mysql clients to natively handle utf8mb4 would resolve the issue.. all I have done is monkey patched files to get mine to work.. and that only works if you only have utf8mb4 clients only (no mix and match), at least on Ubuntu systems. I am not familiar enough with the Index.xml file, nor do I fully understand how it works within my os to get it to work with both utf8 and utf8mb4 charsets. Honestly, this has caused so much frustration and time converting everything to work with iOS emoji, that I am looking to migrate everything to postgres :( utf8 should be utf8. |
|
I got same issue but install MySQL-devel version 5.5.20 BEFORE installing mysql2 0.3.12b fixed this issue. |
|
Definitively closing the issue with these instructions:
|
|
I has the error on my CentOS 6.5 when i deploy my Sinatra app. I am trying to solve it by : But seems the path is not correct. How should i choose the path. Below is my system mysql path. PS: |
|
@williamhqs The default search for mysql-config includes Your use of |
I am using MariaDB 5.5 but had the same issue with MySQL 5.5
Using Rails 3.2.13 database.yml
I have tried with and without collation:, encoding: hoping it would use the databases default encoding as well as using just using utf8.
Using native Rails 3 gem
'gem mysql2':'Unsupported charset: '"utf8mb4"'messageUsing
gem 'mysql2', :git => "https://github.com/brianmario/mysql2.git"?'sI was able to fix this on an earlier project using mysql5.5 and writing a bunch of scripts to go through each table and column, changing charsets and collations to utf8mb4, and also setting indexes on utf8mb4 to a max of 191 characters.