Fix import query parsing

[JanJakes]

Description:

This is a quick fix of import query parsing, replacing #13 for now. It adds multiple fixes and improvements that make the query recognition in the input dump almost complete.

The only known edge cases that aren't supported yet are the usage of NO_BACKSLASH_ESCAPES SQL mode, and the MySQL DELIMITER ... statement. We likely won't implement those in the current logic—it's better to make the new query tokenizer fully support streaming to handle all edge cases correctly.

Related issues:

• Fixes STU-844.
• Fixes Failed to parse the MySQL query error WordPress/sqlite-database-integration#263.
• Fixes WP_MySQL_Token::get_value() throws error when importing wp_aioseo_notifications SQL query WordPress/sqlite-database-integration#250.

Testing Instructions

• Create a blank site using Studio
• It doesn't matter if the site is running or not.
• Edit the file ~/Library/Application Support/Studio/server-files/sqlite-command/src/Import.php with the changes from this PR.
• In Studio go to the Import/Export tab
• Try multiple queries like those provided on STU-844 or WP_MySQL_Token::get_value() throws error when importing wp_aioseo_notifications SQL query WordPress/sqlite-database-integration#250, or wp_redirection_404_with-create-table.sql
• Confirm that the SQL files are imported correctly.
• Try importing the same SQL dump and confirm that an alert appears stating that "Create Table" couldn't be executed. This is expected because you already imported that dump and created the table.
• Feel free to try other sites and queries to confirm everything works as expected.

[sejas]

@JanJakes, thanks for improving the WP-CLI parser.
I tested it with the two problematic SQL dumps, and both were executed correctly 🥳

It makes the import process a bit slower. Similarly to #13, it increases the import time by ~50%.
I tested it by importing a single 60MB SQL dump file which took 27 seconds compared to 18 seconds with the previous parser. I think the increase is due to checking the encoding. Maybe we could try executing the query as it is, which will work in most scenarios and if the query fails, we could then check the encoding of the statement.

The good part of streaming the file is that we don't reach the memory limit, and I was able to import a big site.

cc @wojtekn

[sejas]

I tested it by importing a single 60MB SQL dump file which took 27 seconds compared to 18 seconds with the previous parser.

@JanJakes I've created this PR using your branch as base. It keeps the same speed during the import because it converts the encoding only if the execution fails. Could you review it?

• Speedup encoding check by executing it as a fallback on execute_statements #15

[JanJakes]

@sejas @wojtekn This seems to be done and approved. I'm not sure if I should merge, so please, go ahead anytime.

[sejas]

Thank you @JanJakes ! @wojtekn will take another look and after that I'll merge it and create a new wp-cli sqlite release and close STU-844 and other issues.

[wojtekn]

I left one comment.

Besides that, it works as expected - all my test imports, along with problematic queries, work fine.

[wojtekn]

Would it be safer and more reliable to detect encoding based on initial part of the file e.g. a few kilobytes and then if it differs, convert it for all statements?

[JanJakes]

Or, zooming out, is there a better way to detect and fix encoding earlier in the process, e.g., using Node.js? The problem with mb_detect_encoding is that it's not very reliable: https://www.php.net/manual/en/function.mb-detect-encoding.php

[sejas]

Reading only the first bytes doesn't identify the correct encoding. We need to read the whole file at once or statement by statement.

Reading the encoding in Node.js and passing it as an argument to WP-CLI sqlite could be a solution. For efficiency, I'll merge and release this PR, and we can consider future improvements if necessary.