Skip to content

Fix import query parsing #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Oct 10, 2025
Merged

Fix import query parsing #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
053f717
Fix character escaping to support escaped backslashes
JanJakes Oct 9, 2025
da3e71a
Detect and fix encoding of the database dump when importing
JanJakes Oct 9, 2025
6c22b98
Fix whitespace being trimmed in multi-line strings
JanJakes Oct 9, 2025
c1a41cd
Fix comment handling - allow them to start anywhere, don't match them…
JanJakes Oct 9, 2025
a84c954
Simplify quote matching state, add tests
JanJakes Oct 9, 2025
8e78cf7
Correctly parse backtick-quoted strings
JanJakes Oct 9, 2025
79f24f9
Interpret escape sequences only in strings that support it
JanJakes Oct 9, 2025
5d7507f
Fix handling of empty lines, add test
JanJakes Oct 10, 2025
cfdd79c
Move encoding check to execute_statements
sejas Oct 10, 2025
8556960
Merge pull request #15 from Automattic/update/speed-up-encoding-check…
sejas Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions features/bootstrap/SQLiteFeatureContext.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ public function aSqlDumpFileNamedWithContent( $filename, PyStringNode $content )
*/
public function theSqliteDatabaseShouldContainATableNamed( $table_name ) {
$this->connectToDatabase();
$result = $this->db->query( "SELECT name FROM sqlite_master WHERE type='table' AND name='$table_name'" );
$result = $this->db->query( "SELECT name FROM sqlite_master WHERE type='table' AND name='" . $this->db->escapeString( $table_name ) . "'" );
$row = $result->fetchArray();
if ( ! $row ) {
throw new Exception( "Table '$table_name' not found in the database." );
Expand All @@ -34,10 +34,11 @@ public function theSqliteDatabaseShouldContainATableNamed( $table_name ) {

/**
* @Then /^the "([^"]*)" should contain a row with name "([^"]*)"$/
* @Then /^the "([^"]*)" should contain a row with name:$/
*/
public function theTableShouldContainARowWithName( $table_name, $name ) {
$this->connectToDatabase();
$result = $this->db->query( "SELECT * FROM $table_name WHERE name='$name'" );
$result = $this->db->query( "SELECT * FROM $table_name WHERE name='" . $this->db->escapeString( $name ) . "'" );
$row = $result->fetchArray();
if ( ! $row ) {
throw new Exception( "Row with name '$name' not found in table '$table_name'." );
Expand Down
128 changes: 128 additions & 0 deletions features/sqlite-import.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,131 @@ Feature: WP-CLI SQLite Import Command
Success: Imported from 'STDIN'.
"""
And the SQLite database should contain the imported data

@require-sqlite
Scenario: Import a file with escape sequences
Given a SQL dump file named "test_import.sql" with content:
"""
SET sql_mode='NO_BACKSLASH_ESCAPES';
CREATE TABLE test_table (id INTEGER PRIMARY KEY AUTO_INCREMENT, name TEXT);
INSERT INTO test_table (name) VALUES ('Test that escaping a backslash \\ works');
INSERT INTO test_table (name) VALUES ('Test that escaping multiple backslashes \\\\\\ works');
INSERT INTO test_table (name) VALUES ('Test that escaping a character \a works');
INSERT INTO test_table (name) VALUES ('Test that escaping a backslash followed by a character \\a works');
INSERT INTO test_table (name) VALUES ('Test that escaping a backslash and a character \\\a works');
"""
When I run `wp sqlite --enable-ast-driver import test_import.sql`
Then STDOUT should contain:
"""
Success: Imported from 'test_import.sql'.
"""
And the SQLite database should contain a table named "test_table"
And the "test_table" should contain a row with name "Test that escaping a backslash \\ works"
And the "test_table" should contain a row with name "Test that escaping multiple backslashes \\\\\\ works"
And the "test_table" should contain a row with name "Test that escaping a character \a works"
And the "test_table" should contain a row with name "Test that escaping a backslash followed by a character \\a works"
And the "test_table" should contain a row with name "Test that escaping a backslash and a character \\\a works"

@require-sqlite
Scenario: Import a file with newlines in strings
Given a SQL dump file named "test_import.sql" with content:
"""
CREATE TABLE test_table (id INTEGER PRIMARY KEY AUTO_INCREMENT, name TEXT);
INSERT INTO test_table (name) VALUES ('Test that a string containing
a newline character and some whitespace works');
"""
When I run `wp sqlite --enable-ast-driver import test_import.sql`
Then STDOUT should contain:
"""
Success: Imported from 'test_import.sql'.
"""
And the SQLite database should contain a table named "test_table"
And the "test_table" should contain a row with name:
"""
Test that a string containing
a newline character and some whitespace works
"""

@require-sqlite
Scenario: Import a file with comments
Given a SQL dump file named "test_import.sql" with content:
"""
CREATE TABLE test_table (id INTEGER PRIMARY KEY AUTO_INCREMENT, name TEXT);
-- This is an inline comment.
# This is an inline comment.
INSERT INTO test_table (name) VALUES ('one'); -- This is an inline comment.
/* This is a block comment */
INSERT INTO test_table (name) VALUES ('two'); /* This
is a block comment
on multiple lines */ INSERT INTO test_table (name) VALUES ('three');
INSERT INTO test_table (name) VALUES ('fo -- this looks like a comment ur');
INSERT INTO test_table (name) VALUES ('fi/* this looks like a comment */ve');
"""
When I run `wp sqlite --enable-ast-driver import test_import.sql`
Then STDOUT should contain:
"""
Success: Imported from 'test_import.sql'.
"""
And the SQLite database should contain a table named "test_table"
And the "test_table" should contain a row with name "one"
And the "test_table" should contain a row with name "two"
And the "test_table" should contain a row with name "three"
And the "test_table" should contain a row with name "fo -- this looks like a comment ur"
And the "test_table" should contain a row with name "fi/* this looks like a comment */ve"

@require-sqlite
Scenario: Import a file quoted strings
Given a SQL dump file named "test_import.sql" with content:
"""
CREATE TABLE test_table (id INTEGER PRIMARY KEY AUTO_INCREMENT, name TEXT);
INSERT INTO test_table (name) VALUES ('a single-quoted string with \' '' some " tricky ` chars');
INSERT INTO test_table (name) VALUES ("a double-quoted string with ' some \" "" tricky ` chars");
"""
When I run `wp sqlite --enable-ast-driver import test_import.sql`
Then STDOUT should contain:
"""
Success: Imported from 'test_import.sql'.
"""
And the SQLite database should contain a table named "test_table"
And the "test_table" should contain a row with name:
"""
a single-quoted string with ' ' some " tricky ` chars
"""
And the "test_table" should contain a row with name:
"""
a double-quoted string with ' some " " tricky ` chars
"""

@require-sqlite
Scenario: Import a file backtick-quoted identifiers
Given a SQL dump file named "test_import.sql" with content:
"""
CREATE TABLE `a'strange``identifier\\name` (id INTEGER PRIMARY KEY AUTO_INCREMENT, name TEXT);
"""
When I run `wp sqlite --enable-ast-driver import test_import.sql`
Then STDOUT should contain:
"""
Success: Imported from 'test_import.sql'.
"""

And the SQLite database should contain a table named "a'strange`identifier\name"

@require-sqlite
Scenario: Import a file with whitespace and empty lines
Given a SQL dump file named "test_import.sql" with content:
"""

CREATE TABLE test_table (id INTEGER PRIMARY KEY AUTO_INCREMENT, name TEXT);


INSERT INTO test_table (name) VALUES ('Test Name');

"""
When I run `wp sqlite --enable-ast-driver import test_import.sql`
Then STDOUT should contain:
"""
Success: Imported from 'test_import.sql'.
"""

And the SQLite database should contain a table named "test_table"
And the "test_table" should contain a row with name "Test Name"
106 changes: 68 additions & 38 deletions src/Import.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,21 @@ public function run( $sql_file_path, $args ) {
*/
protected function execute_statements( $import_file ) {
foreach ( $this->parse_statements( $import_file ) as $statement ) {
$result = $this->driver->query( $statement );
if ( false === $result ) {
WP_CLI::warning( 'Could not execute statement: ' . $statement );
echo $this->driver->get_error_message();
try {
$this->driver->query( $statement );
} catch ( Exception $e ) {
try {
// Try converting encoding and retry
$detected_encoding = mb_detect_encoding( $statement, mb_list_encodings(), true );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be safer and more reliable to detect encoding based on initial part of the file e.g. a few kilobytes and then if it differs, convert it for all statements?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, zooming out, is there a better way to detect and fix encoding earlier in the process, e.g., using Node.js? The problem with mb_detect_encoding is that it's not very reliable: https://www.php.net/manual/en/function.mb-detect-encoding.php

Screenshot 2025-10-10 at 14 53 12

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading only the first bytes doesn't identify the correct encoding. We need to read the whole file at once or statement by statement.

Reading the encoding in Node.js and passing it as an argument to WP-CLI sqlite could be a solution. For efficiency, I'll merge and release this PR, and we can consider future improvements if necessary.

if ( $detected_encoding && 'UTF-8' !== $detected_encoding ) {
$converted_statement = mb_convert_encoding( $statement, 'UTF-8', $detected_encoding );
echo 'Converted ecoding for statement: ' . $converted_statement . PHP_EOL;
$this->driver->query( $converted_statement );
}
} catch ( Exception $e ) {
WP_CLI::warning( 'Could not execute statement: ' . $statement );
echo $e->getMessage();
}
}
}
}
Expand All @@ -73,52 +84,70 @@ public function parse_statements( $sql_file_path ) {
WP_CLI::error( "Unable to open file: $sql_file_path" );
}

$single_quotes = 0;
$double_quotes = 0;
$in_comment = false;
$buffer = '';
$starting_quote = null;
$in_comment = false;
$buffer = '';

// phpcs:ignore
while ( ( $line = fgets( $handle ) ) !== false ) {
$line = trim( $line );

// Skip empty lines and comments
if ( empty( $line ) || strpos( $line, '--' ) === 0 || strpos( $line, '#' ) === 0 ) {
continue;
}

// Handle multi-line comments
if ( ! $in_comment && strpos( $line, '/*' ) === 0 ) {
$in_comment = true;
}
if ( $in_comment ) {
if ( strpos( $line, '*/' ) !== false ) {
$in_comment = false;
}
continue;
}

$strlen = strlen( $line );
for ( $i = 0; $i < $strlen; $i++ ) {
$ch = $line[ $i ];

// Handle escaped characters
if ( $i > 0 && '\\' === $line[ $i - 1 ] ) {
$buffer .= $ch;
continue;
// Handle escape sequences in single and double quoted strings.
// TODO: Support NO_BACKSLASH_ESCAPES SQL mode.
if ( "'" === $ch || '"' === $ch ) {
// Count preceding backslashes.
$slashes = 0;
while ( $slashes < $i && '\\' === $line[ $i - $slashes - 1 ] ) {
++$slashes;
}

// Handle escaped characters.
// A characters is escaped only when the number of preceding backslashes
// is odd - "\" is an escape sequence, "\\" is an escaped backslash.
if ( 1 === $slashes % 2 ) {
$buffer .= $ch;
continue;
}
}

// Handle quotes
if ( "'" === $ch && 0 === $double_quotes ) {
$single_quotes = 1 - $single_quotes;
// Handle comments.
if ( null === $starting_quote ) {
$prev_ch = isset( $line[ $i - 1 ] ) ? $line[ $i - 1 ] : null;
$next_ch = isset( $line[ $i + 1 ] ) ? $line[ $i + 1 ] : null;

// Skip inline comments.
if ( ( '-' === $ch && '-' === $next_ch ) || '#' === $ch ) {
break; // Stop for the current line.
}

// Skip multi-line comments.
if ( ! $in_comment && '/' === $ch && '*' === $next_ch ) {
$in_comment = true;
continue;
}
if ( $in_comment ) {
if ( '*' === $prev_ch && '/' === $ch ) {
$in_comment = false;
}
continue;
}
}
if ( '"' === $ch && 0 === $single_quotes ) {
$double_quotes = 1 - $double_quotes;

// Handle quotes
if ( null === $starting_quote && ( "'" === $ch || '"' === $ch || '`' === $ch ) ) {
$starting_quote = $ch;
} elseif ( null !== $starting_quote && $ch === $starting_quote ) {
$starting_quote = null;
}

// Process statement end
if ( ';' === $ch && 0 === $single_quotes && 0 === $double_quotes ) {
yield trim( $buffer );
if ( ';' === $ch && null === $starting_quote ) {
$buffer = trim( $buffer );
if ( ! empty( $buffer ) ) {
yield $buffer;
}
$buffer = '';
} else {
$buffer .= $ch;
Expand All @@ -127,8 +156,9 @@ public function parse_statements( $sql_file_path ) {
}

// Handle any remaining buffer content
$buffer = trim( $buffer );
if ( ! empty( $buffer ) ) {
yield trim( $buffer );
yield $buffer;
}

fclose( $handle );
Expand Down