Set charset/collation properly for each text column if using MySQL.

[knu]

With this change, Huginn is able to store up to 4-byte UTF-8 characters
in its database. This should fix #286.

[knu]

This is only for Ruby >=2.0 because I used Module#prepend in monkey patch, and it effectively does nothing on Ruby 1.9.
The migration is skipped for non-mysql database adapters, so I think it's safe for everyone.

[knu]

Hmm, the error above looks unfamiliar. There might be a strange situation in the migration chain.

[coveralls]

Coverage decreased (-0.42%) when pulling 3f4b640 on knu:set_charset_for_mysql into 67fcfef on cantino:master.

[cantino]

Very interesting approach @knu!

Will this work on Ruby <2.0?

Do you think we should pull this into a gem and require it? That might make this useful for other projects too.

[cantino]

Going forward, what do our migrations need to contain to have good UTF8 support?

[cantino]

@knu, please see my questions above when you have time.

[knu]

Will this work on Ruby <2.0?

You can make it work in Ruby 1.9.x by replacing prepend with the alias_method_chain technique, but since the 1.9 series of Ruby will be dying in about six month I'd rather urge users to move to a newer version of Ruby.

Do you think we should pull this into a gem and require it? That might make this useful for other projects too.

Maybe, but this is just a local hack after all. Naming it something like "per-column charset support for ActiveRecord/MySQL" would be a hype when it only works on DDL (i.e. migration) and not on DML, which means you can only practically use it for mixing utf8mb4 with utf8 (and ascii).

It would be a long way to make ActiveRecord and the underlying MySQL driver(s) support operating on a table with columns of mixed charsets properly. So, at the moment, while this patch works for many projects including Huginn, I'm still kind of reluctant to release it as a gem as it is now.

I'll have to watch the Rails development closely to figure out how to integrate this into AR.

[cantino]

Good to know. It'd be nice to maintain support for Ruby 1.9 for the time being, since we want Huginn to be as accessible as possible.

[virtadpt]

For what it's worth, one of the most common platforms these days is Ubuntu 14.04 LTS (EC2 and Digital Ocean come immediately to mind), which packages Ruby v1.9 and it's notoriously problematic to migrate it to Ruby v2.0. Being able to run Huginn with Ruby v1.9 until the next release of Ubuntu server comes out would be very, very helpful.

[cantino]

Yea, I'm not comfortable dropping 1.9 yet.

[coveralls]

Coverage decreased (-0.4%) when pulling 2fdaab5 on knu:set_charset_for_mysql into d7c5417 on cantino:master.

[coveralls]

Coverage decreased (-0.36%) when pulling 2302a59 on knu:set_charset_for_mysql into d7c5417 on cantino:master.

[cantino]

Is this done @knu?

[knu]

@cantino Yes, the utf8mb4 charset support has been extended to cover ruby 1.9 by the last commit.

[cantino]

Should we be using utf8-unicode-ci instead of utf8-general-ci?

http://stackoverflow.com/questions/1036454/what-are-the-diffrences-between-utf8-general-ci-and-utf8-unicode-ci

[cantino]

Could you explain again why we treat username differently from the other columns, or why all columns can't be utf8mb4_bin?

[knu]

This is used for login ID, so I thought it should be less sensitive to cases and in terms of uniqueness.

[cantino]

Testing a deploy now to my setup.

[knu]

@cantino Maybe. They are both broken depending on the language used. I chose utf8_general_ci because it's the default collation of MySQL. Actually, we avoid the use of utf8_unicode_ci here in Japan because Japanese words "haha" (mother), "papa" (dad) and "baba" (grandma) are all treated equal in comparison under that collation. But I imagine there'd be a similar kind of problem in other (Western) countries and that would be why Rails uses utf8_unicode_ci by default.

For now, few users would dare to use non-ASCII characters in their username, so I can live with utf8_unicode_ci.

[cantino]

Oh oh.

 ** [out :: aicomplete.com] == 20140813110107 SetCharsetForMysql: migrating ===============================
 ** [out :: aicomplete.com] 
 ** [out :: aicomplete.com] -- change_column("agents", "options", :text, {:limit=>16777215, :null=>true, :default=>nil, :charset=>"utf8mb4", :collation=>"utf8mb4_bin"})
 ** [out :: aicomplete.com] 
 ** [out :: aicomplete.com] rake aborted!
 ** [out :: aicomplete.com] StandardError: An error has occurred, all later migrations canceled:
 ** [out :: aicomplete.com] 
 ** [out :: aicomplete.com] Mysql2::Error: Unknown character set: 'utf8mb4': ALTER TABLE `agents` CHANGE `options` `options` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL

[cantino]

What version of mysql does this require?

[knu]

Oh. Looks like it's from MySQL 5.5.

[cantino]

Hmm. I what should we do for older versions?

[knu]

We could use binary (blob) types instead and implement serializer/deserializer, I guess. As for JSON fields, we have JsonSerializedField to tweak. I smell of unexpected encoding errors everywhere until we could finally get it to work, though...

[cantino]

Yes, that sounds difficult. How hard would it be to only run the migration on supported versions of MySQL, and make sure schema.rb in the repo works okay on older mysql versions -- that is, your library should just ignore collation commands if mysql cannot handle them?

Does feel like maybe pulling it into a gem would be good.

[knu]

@cantino I've added a quick hack to replace utf8mb4 with utf8 if the server has no support for utf8mb4, with which the stock schema.rb will work on any version of MySQL.

[coveralls]

Coverage decreased (-0.55%) when pulling f4f6930 on knu:set_charset_for_mysql into d7c5417 on cantino:master.

[cantino]

Unfortunately, I'm still getting the same error:

** [out :: aicomplete.com] == 20140813110107 SetCharsetForMysql: migrating ===============================
 ** [out :: example.com] 
 ** [out :: example.com] -- change_column("agents", "options", :text, {:limit=>16777215, :null=>true, :default=>nil, :charset=>"utf8mb4", :collation=>"utf8mb4_bin"})
 ** [out :: example.com] 
 ** [out :: example.com] rake aborted!
 ** [out :: example.com] StandardError: An error has occurred, all later migrations canceled:
 ** [out :: example.com] 
 ** [out :: example.com] Mysql2::Error: Unknown character set: 'utf8mb4': ALTER TABLE `agents` CHANGE `options` `options` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL
 ** [out :: example.com] /home/huginn/app/shared/bundle/ruby/1.9.1/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_mysql_

When I run ActiveRecord::Base.connection.utf8mb4_supported?, I get false.

One other question: once this is merged, do we have to do anything differently now when we add new tables or columns? How do we set the charset and collation correctly going forward?

[coveralls]

Coverage decreased (-0.34%) when pulling 857b8ea on knu:set_charset_for_mysql into 0227496 on cantino:master.

[cantino]

I think you're good to merge @knu. Going forward, will usages of utf8mb4 cause errors on older MySQL servers if we use it in future migration column definitions, or will your code translate these to utf8 when the server can't support it?

[knu]

@cantino The charset fallback will be effective in any future migration coumn definitions.

[cantino]

Awesome, nice work :)

[othreed]

I am having problems with switching to utf8mb4.
I deleted my old database and did the changed mentioned -> changed the mysql configuration in my.cnf to use the utf8mb4 as a default and also changed the DATABASE_ENCODING to utf8mb4
in the .env file.
When I run rake db:migrate I get the following error for the index.

== 20120728210244 DeviseCreateUsers: migrating ================================ create_table(:users) -> 0.0348s -- add_index(:users, :email, {:unique=>true}) rake aborted! StandardError: An error has occurred, all later migrations canceled: Mysql2::Error: Specified key was too long; max key length is 1000 bytes: CREATE UNIQUE INDEX index_users_on_email

what did I miss?
thank you for your help

[knu]

It is generally not a good idea to build everything in utf8mb4 by making it the default character set on the server side. Check out this comment. You could still extend the maximum key size by tweaking InnoDB parameters, though.

[othreed]

Thank you for this information. I disabled InnoDB and switched to MyISAM to save on RAM... was this a wrong decision? Also, do I need to do any more changed to huginn or is the code ready for this utf8mb4 tweak?

These changes are inside my my.cnf:
character_set_server=utf8mb4
collation_server=utf8mb4_unicode_ci
skip-innodb
default-storage-engine = myisam

and in the .env file I added this:
''DATABASE_ENCODING=utf8mb4

thanks for any help :)

[knu]

You don't need to change the server settings for this. Just let it use utf8. It's the client side that specifies character sets in DDL as necessary.

[othreed]

I removed the server settings and restarted mysql. This is what SHOW VARIABLES; says about my database encoding now:

character_set_client                              | latin1                                                                                                                 
character_set_connection                          | latin1                                                                                                                 
character_set_database                            | latin1                                                                                                                 character_set_filesystem                          | binary                                                                                                                
character_set_results                             | latin1                                                                                                                 
character_set_server                              | latin1                                                                                                                 
character_set_system                              | utf8                                                                                                                   
character_sets_dir                                | /usr/share/mysql/charsets/                                                                                             
collation_connection                              | latin1_swedish_ci                                                                                                      
collation_database                                | latin1_swedish_ci                                                                                                      
collation_server                                  | latin1_swedish_ci

I still get the same error running sudo RAILS_ENV=production bundle exec rake db:migrate --trace

== 20120728210244 DeviseCreateUsers: migrating ================================
-- create_table(:users)
   -> 0.0372s
-- add_index(:users, :email, {:unique=>true})
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Mysql2::Error: Specified key was too long; max key length is 1000 bytes: CREATE UNIQUE INDEX `index_users_on_email`  ON `users` (`email`) /home/pi/app/shared/bundle/ruby/1.9.1/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:303:in `query'

running SHOW CREATE DATABASE huginn_production; gives me the following output:

`CREATE DATABASE `huginn_production` /*!40100 DEFAULT CHARACTER SET utf8mb4 */

running SHOW CREATE TABLE users; gives me the following output:

| users | CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `email` varchar(255) NOT NULL DEFAULT '',
  `encrypted_password` varchar(255) NOT NULL DEFAULT '',
  `reset_password_token` varchar(255) DEFAULT NULL,
  `reset_password_sent_at` datetime DEFAULT NULL,
  `remember_created_at` datetime DEFAULT NULL,
  `sign_in_count` int(11) DEFAULT '0',
  `current_sign_in_at` datetime DEFAULT NULL,
  `last_sign_in_at` datetime DEFAULT NULL,
  `current_sign_in_ip` varchar(255) DEFAULT NULL,
  `last_sign_in_ip` varchar(255) DEFAULT NULL,
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 |

I did not understand the comments... where should I make the changes ?

[knu]

Maybe you need to set DATABASE_ENCODING to utf8 when creating a database and then change it to utf8mb4 when running the app. I'll check out and fix that.

[knu]

The above commit should fix the problem.

Please set DATABASE_ENCODING to utf8mb4 and start over from creating a new database. (either bundle exec rake db:create db:migrate or bundle exec rake db:setup)

[othreed]

in my .env file I have set:

DATABASE_ENCODING=utf8mb4

also I included your commit.

Now bundle exec rake db:setup --trace shows me more output but still I get the error:

** Invoke db:setup (first_time)
** Invoke db:schema:load_if_ruby (first_time)
** Invoke db:create (first_time)
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:create
** Invoke environment (first_time)
** Execute environment
** Execute db:schema:load_if_ruby
** Invoke db:schema:load (first_time)
** Invoke environment 
** Invoke db:load_config 
** Execute db:schema:load
-- enable_extension("plpgsql")
   -> 0.0098s
-- create_table("agent_logs", {:force=>true})
   -> 0.3347s
-- create_table("agents", {:force=>true})
   -> 0.0647s
-- add_index("agents", ["guid"], {:name=>"index_agents_on_guid", :using=>:btree})
   -> 0.5116s
-- add_index("agents", ["schedule"], {:name=>"index_agents_on_schedule", :using=>:btree})
   -> 0.0338s
-- add_index("agents", ["type"], {:name=>"index_agents_on_type", :using=>:btree})
   -> 0.0330s
-- add_index("agents", ["user_id", "created_at"], {:name=>"index_agents_on_user_id_and_created_at", :using=>:btree})
   -> 0.0333s
-- create_table("control_links", {:force=>true})
   -> 0.0353s
-- add_index("control_links", ["control_target_id"], {:name=>"index_control_links_on_control_target_id", :using=>:btree})
   -> 0.0294s
-- add_index("control_links", ["controller_id", "control_target_id"], {:name=>"index_control_links_on_controller_id_and_control_target_id", :unique=>true, :using=>:btree})
   -> 0.0307s
-- create_table("delayed_jobs", {:force=>true})
   -> 0.0410s
-- add_index("delayed_jobs", ["priority", "run_at"], {:name=>"delayed_jobs_priority", :using=>:btree})
   -> 0.0300s
-- create_table("events", {:force=>true})
   -> 0.0425s
-- add_index("events", ["agent_id", "created_at"], {:name=>"index_events_on_agent_id_and_created_at", :using=>:btree})
   -> 0.0283s
-- add_index("events", ["expires_at"], {:name=>"index_events_on_expires_at", :using=>:btree})
   -> 0.0310s
-- add_index("events", ["user_id", "created_at"], {:name=>"index_events_on_user_id_and_created_at", :using=>:btree})
   -> 0.0303s
-- create_table("links", {:force=>true})
   -> 0.0341s
-- add_index("links", ["receiver_id", "source_id"], {:name=>"index_links_on_receiver_id_and_source_id", :using=>:btree})
   -> 0.0272s
-- add_index("links", ["source_id", "receiver_id"], {:name=>"index_links_on_source_id_and_receiver_id", :using=>:btree})
   -> 0.0309s
-- create_table("scenario_memberships", {:force=>true})
   -> 0.0296s
-- add_index("scenario_memberships", ["agent_id"], {:name=>"index_scenario_memberships_on_agent_id", :using=>:btree})
   -> 0.0260s
-- add_index("scenario_memberships", ["scenario_id"], {:name=>"index_scenario_memberships_on_scenario_id", :using=>:btree})
   -> 0.0278s
-- create_table("scenarios", {:force=>true})
   -> 0.0424s
-- add_index("scenarios", ["user_id", "guid"], {:name=>"index_scenarios_on_user_id_and_guid", :unique=>true, :using=>:btree})
   -> 0.0331s
-- create_table("services", {:force=>true})
   -> 0.0457s
-- add_index("services", ["provider"], {:name=>"index_services_on_provider", :using=>:btree})
   -> 0.0276s
-- add_index("services", ["uid"], {:name=>"index_services_on_uid", :using=>:btree})
   -> 0.0316s
-- add_index("services", ["user_id", "global"], {:name=>"index_services_on_user_id_and_global", :using=>:btree})
   -> 0.0354s
-- create_table("user_credentials", {:force=>true})
   -> 0.0364s
-- add_index("user_credentials", ["user_id", "credential_name"], {:name=>"index_user_credentials_on_user_id_and_credential_name", :unique=>true, :using=>:btree})
rake aborted!
ActiveRecord::StatementInvalid: Mysql2::Error: Specified key was too long; max key length is 1000 bytes: CREATE UNIQUE INDEX `index_user_credentials_on_user_id_and_credential_name` USING btree ON `user_credentials` (`user_id`, `credential_name`) 

[knu]

What does show create database huginn (or whatever you named it) say? The database is expected to be utf8.

[othreed]

thanks for the help.

huginn_production | CREATE DATABASE `huginn_production` /*!40100 DEFAULT CHARACTER SET utf8mb4 */ 

[knu]

That's oh. Running rake db:create on the lastest master should create a database in utf8, not utf8mb4.

[othreed]

ok I will look into this , thanks :)

[othreed]

I must have messed up the settings somewhere because rake db:create makes the default to utf8mb4.
so I ran this:

ALTER DATABASE 'huginn_production' DEFAULT CHARACTER SET utf8;

after this db:migrate and db:seed worked with no errors. I will test and report :)

[othreed]

thanks for the help it works now..
I wanted to do this utf8mb4 because I thought it would help me with an error I have with a website agent processing a weather data rss.
http://www.geomar.de/service/wetter/feed/

But even after applying the change I still get the old error with the website agent.

I found out that the square symbol is what causes the data error in the rss feed: Summe Globalstrahlung: 1,89332kWh/m²

In the agent log I get this error:

"\xC2" from ASCII-8BIT to UTF-8

Is this another issue or does it count into this thread?

[knu]

"\xC2" is not a valid byte sequence in UTF-8, so I don't think it is related to this issue.

Maybe the WebSiteAgent failed to detect the encoding of the feed. Try force_encoding: "utf8". (And file another issue if you still have a problem)

[othreed]

thanks a bunch for the tip - now it works : )

[knu]

@othreed Great! Can you share the agent's options? The URL seems to return a Content-Type with charset=utf-8, so force_encoding shouldn't be essentially necessary. Also, I'd like to know the ruby version you are using for the create database issue above to reproduce for me.

[othreed]

ruby 1.9.3p194 (2012-04-20 revision 35410) [arm-linux-eabihf]
MySQL 5.5.38-0+wheezy1

Website Agent for Local_Weather_Feed

{
  "expected_update_period_in_days": "2",
  "force_encoding": "UTF-8",
  "url": "http://www.geomar.de/service/wetter/feed/",
  "type": "text",
  "mode": "on_change",
  "extract": {
    "weather_sensors": {
      "regexp": "([^:\\s<>][^:><]+): ([^<]+)",
      "index": 0
    }
  }
}

[knu]

Thanks! I'll look into this tomorrow.