Prevent MySQL collation errors

[Oreilles]

MySQL uses a total of 8 different system variables to determine how to handle strings.

• character_set_client
• character_set_results
• character_set_connection( &&collation_connection )
• character_set_database( &&collation_database )
• character_set_server ( &&collation_server )

When receiving a query, all strings are interpreted as character_set_client, translated to character_set_connection, then all results are sent back to the client after translation from character_set_database (or whatever the table's character set is) to character_set_results.

character_set_connection is important for two different reason: it will use that specific character set to compare two literal string sent by the client, e.g. select 'Hello' = 'Hëllo' - and, more importantly, all table and columns created from that connection will be created with that character set (and collation).

The problem many people are facing, is that they are managing their database with tables and columns that matches their database default, and are then using Directus, which uses node-mysql, which uses UTF8 for the connection regardless of the database default.

What we want is the IO to be performed in UTF8, and the database operations to be performed with the database default character set and collation. And there's a command that does exactly that:

SET CHARACTER SET 'UTF8MB4'.
-- translates to:
SET character_set_client = 'UTF8MB4';
SET character_set_results = 'UTF8MB4';
SET collation_connection = @@collation_database;

This change is going to be painful for a lot of people. Because it used to be recommended that people set their DB_CHARSET environment variable for MySQL, and that people who have been deploying directus using npm init directus-project or npx directus bootstrap didn't have an environment file yet, close to 100% of MySQL users currently have all of their Directus system tables ( as well as all the table they created with the app ) in utf8_general_ci.

This change means that until they fix all those tables and columns collation to match their database default, every relation creation will fail. For that reason, I think we should fix the bug allowing the relational field to be created even if the relation creation fails. (#10564) before merging this PR, or the amount of related bug report will skyrocket.

Fixes #9897
Fixes #11242

[rijkvanzanten]

@Oreilles Could you provide a brief set of instructions on how to ensure that the Directus tables use the correct charset? Would be even nicer if we could provide some SQL to run 🙂

[rijkvanzanten]

On my end, mysql seems to correctly default to the current database charset/collation. If that's consistently the case, we should be able to clean this up a bit, and mostly lean on warnings to prevent user mistakes 🤔