New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: utf8mb4 database errors #8268
Conversation
In the installation application, InstallationModelDatabase::populateDatabase will convert automatically utf8mb4 to utf8 if utf8mb4 support is not present. See line 933. Likewise for JInstaller::parseSQLFiles, see line 954 and 960. The detection for utf8mb4 cannot be super precise. We take version's claims with a pinch of salt at JDatabaseDriverMysqli::serverClaimsUtf8mb4Support and JDatabaseDriverMysql::serverClaimsUtf8mb4Support. However, if you're using the PDO driver you do not have the required information, therefore JDatabaseDriverPdomysql::__construct can only hope for the best. However, in all of the above cases, if we end up trying to use utf8mb4 and it's rejected at server connection time (connect() method) we revert to plain old utf8. Now, if a server doesn't barf at connection time but complains at table creation time there's nothing I can do about it except suggest an upgrade to a newer version of MySQL. For what is worth this is exactly what WordPress does regarding utf8mb4 support so we're not exactly the odd sheep in the herd. |
Holy inconsistent Joomla! code, Batman! It looks like the Fix button goes through SQL file parsing of its own instead of using any of the other TWO places in Joomla! where we also parse and apply SQL update files. FFS... You can solve by changing JSchemaChangeitem line 243 from $this->db->setQuery($this->updateQuery); to $query = $this->db->convertUtf8mb4QueryToUtf8($this->updateQuery);
$this->db->setQuery($query); @zero-24 can you please add that to your PR? |
Thanks @nikosdion done with: ed0e3e7 @zjw can you please try the changes with this patch applyed? If you need help to apply the patch please have a look here: https://joom.la/patchtesting |
After merging jah-tz/joomla-cms@ed0e3e750c93cb23b1e44f12952f876378c4ce90 -- Extensions->Manage->Database shows:
This error illustrates that you omitted fixing the After tapping the |
Thanks i can fix #_menu tomorrow if i'm back to the computer. |
I have encountered another issue. To test: Install Joomla 3.4.3 (which doesn't have the utf8mb4 patch from #7173) onto a system that doesn't support utf8mb4. (As was done in #8267. In my case it is a CentOS 6.7 system.) Prepare a full update package for 3.5.0-dev (e.g., Go to Results:
So the update didn't complete. Checking the status on the server, it doesn't look like any of the 3.5.0 schema updates were applied. Looking at Extensions->Manage->Database, it shows:
Tapping the I suspect the problem arose because I was using the 3.4.3 installer to install the 3.5.0-dev update. The 3.4.3 installer doesn't know anything about the special handling of utf8mb4 SQL files, so it fails when it tries to apply the 3.5.0-dev SQL update from Perhaps there's a way of modifying the 3.5.0-dev manifest script ( |
@@ -270,7 +269,7 @@ public function convertTablesToUtf8mb4() | |||
// Get the SQL file to convert the core tables. Yes, this is hardcoded because we have all sorts of index | |||
// conversions and funky things we can't automate in core tables without an actual SQL file. | |||
$serverType = $db->getServerType(); | |||
$fileName = JPATH_ADMINISTRATOR . "/components/com_admin/sql/updates/$serverType/3.5.0-2015-01-01.sql"; | |||
$fileName = JPATH_ADMINISTRATOR . "/components/com_admin/sql/updates/$serverType/3.5.0-2015-07-01.sql"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use strict verification of legacy Joomla MD5 hash (Fix #8326)
Deprecate JCryptPassword
Smartsearch: collation issue when searching in back-end index
Fix language keys which trigger wrong plural detection in translation tools
Improvement to updatenotification plugin language strings
@zero I can confirm @zjw 's latest issue. I tried to update a copy of my 3.4.5 production site to 3.5. Beta 1 using an update container patched by the (up to now) latest changes from this PR here, and it happens what zjw describes: The update first fails with the 11 database problems, then fix database works, the new plugins (stats, update notification) have to be installed with the discover method, and possibly other database changes not detected by the database check are not performed. This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8268. |
[3.5.0 Beta] Fix permissons editing with the PDO Driver
Use hash_equals() for constant-time string comparison
Utilize safe string methods in JCrypt
give the user the option if anonymous data will sent
Fixed long title in Smart Search: Indexed Content
…thor Fixed long title for Hathor in Smart Search: Indexed Content
Yes we have still problems here but i'm unable to fix it. I will leave this PR here open for referece if somone is willing to take that over or is able to find the problem please let me know. I would call this issue a release blocker as it will break much sites on updating. Thanks This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8268. |
Ok i finaly broken this PR. 😢 So only the code will be there: https://github.com/jah-tz/joomla-cms/tree/c6cb680fbb0e55fab3b4d8e726c9db5dd634c1d7 and I need to close here. Thanks to anyone how is able to fix this issue. @zjw I'm going to reopen your issue now. Sorry for all the mess 😞 |
Hmm, why not leave the conversion of the tables to be manually done by the site admins as described for example here https://mathiasbynens.be/notes/mysql-utf8mb4 and so get rid of the character set conversion statements in sql scipt 3.5.0-2015-07-01.sql, which fail on mysql versions lower than 5.5 (because utf8mb4 not supported)? The rest of the PR would be ok then I think. This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8268. |
@richard67 this is no option for me (personaly). We have a B/C promise and 3.5.0 or more the PR #7173 break the CMS on old mysql versions (that are supported) as you will get this error back on all joomla updates (as it allway replaces the update SQL file) and the most ouf our users are unable to change the mysql version as it is controled by the host. |
OK I've merged the fix to the database update schema. We just need to work out the best fix for the article alias' now and we are good to go |
See #8376 for part of the issue |
This Fixes the issues introduced here: #7173 and reported here: #8267
Maybe @nikosdion can have a look here.
maybe he has a idea to fix for hosts that don't support utf8mb4:
As the file is preset in the update SQL Directory it will be picked up by the default updater anyway. Maybe just move the file (https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_admin/sql/updates/mysql/3.5.0-2015-07-01.sql) to another directory?
@zjw
Here I'm fixing your point 1 and 2 the third point (
Unknown character set: 'utf8mb4'
) is not fixed yet.Original Issue
Tested with Joomla 3.5.0-dev, current master (as of 7abe00f).
Database Version MySQL 5.1.73 (mysqlnd 5.0.10)
Database Connection Collation utf8_general_ci
PHP Version 5.4.16
To test (requires using a system that does not support utf8mb4 -- see PR #7173):
Go to Extensions->Manage->Database:
Note the errors shown:
Tap the "Fix" button. Note the error:
The errors obviously relate to PR #7173.
I'm not sure what the proper fix is, but I do see a couple things in that PR that I have questions about:
$fileName = JPATH_ADMINISTRATOR . "/components/com_admin/sql/updates/$serverType/3.5.0-2015-01-01.sql";
. I believe the file it is referring to is actuallymysql/3.5.0-2015-07-01.sql
(note the different date).varchar(191)
, but the actual type found in the database isvarchar(255)
. So the SQL file is obviously not being applied properly (hence the errors). But that also raises the point that the default schema (installation/sql/mysql/joomla.sql
) needs updating so that the columns at issue are created asvarchar(191)
, notvarchar(255)
.