Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Fixed SQL placeholder parsing #113

Merged
merged 1 commit into from

6 participants

@avit

I reworked DoctrineSQLParser to fix these problems:

Named placeholder patterns

The parser used a pattern that would match multiple colons, so junk like :fir:stNa::m:e was a "valid" placeholder name.

The parser neglected the underscore so :first_name was invalid and only captured :first.

This is cleaned up so placeholder names are consistent with PHP variable names: "starts with a letter or underscore, followed by any number of letters, numbers, or underscores".

Quote characters in strings

The parser had a simplistic way of matching quote pairs, ignoring that a quote character could be escaped by a backslash inside a string.

It now recognizes 'it\'s a trap?' as a whole literal string instead of breaking out after it\ and parsing the ? as a SQL placeholder.

Speed

The original algorithm looped a regular expression over each character position in the whole statement. The new algorithm handles matching in two steps:

  • One regular expression to select unquoted SQL.
  • One regular expression to match placeholder tokens in the result.

I measured the result to be over 100% faster given the simple strings from the test case (even with 2 new runs added to the test data). The improvement would be greater on longer statements:

Doctrine\DBAL\SQLParserUtils::getPlaceholderPositions
| version:     | invocation count: | total inclusive time: |
| original     | 37                | 5719                  |
| this patch   | 39                | 2380                  |
@stof

@avit will it work if a \ is escaped just before a quote, e.g. "SELECT * FROM foo WHERE bar = 'something\\'" ?

@avit

@stof Yes it did work, but it didn't check if the backslashes were paired, so it could be fooled. This addition I just posted is more rigorous about that. (Also: refactored a bit.)

lib/Doctrine/DBAL/SQLParserUtils.php
@@ -176,4 +173,22 @@ static public function expandListParameters($query, $params, $types)
return array($query, $params, $types);
}
+
+ /**
+ * Slice the SQL statement around pairs of quotes and
+ * return string fragments of SQL outside of quoted literals.
+ * Each fragment is captured as a 2-element array:
+ *
+ * 0 => matched fragment string,
+ * 1 => offset of fragment in $statement
+ *
+ * @param string $statement
+ * @return array
+ */
+ static public function getUnquotedStatementFragments($statement) {
@stof
stof added a note

the curly brace should be on its own line

@stof
stof added a note

and does it make sense to use this method in another context ? If no, it should be private

@avit
avit added a note

Is there a such thing as a static private function? If this is not useful for anything I can just fold it back into getPlaceholderPositions.

@stof
stof added a note

static has nothing to do with the visibility. You can use it for public, protected or private methods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
tests/Doctrine/Tests/DBAL/SQLParserUtilsTest.php
@@ -28,12 +28,20 @@ static public function dataGetPlaceholderPositions()
array("SELECT '?' FROM foo", true, array()),
array('SELECT "?" FROM foo WHERE bar = ?', true, array(32)),
array("SELECT '?' FROM foo WHERE bar = ?", true, array(32)),
+ array(
+<<<SQLDATA
+SELECT * FROM foo WHERE bar = 'it\\'s a trap? \\\\' OR bar = ?
@stof
stof added a note

is it valid ? It seems to me that their is an extra escape here. Should be it\'s, isn't it ?

@avit
avit added a note

Yes it's valid. Backslashes are all doubled in PHP strings, in any kind of quotes. The resulting string shows 1 backslash in "it's" and 2 backslashes after "trap?"

@stof
stof added a note

@avit switch it to Nowdoc instead of Heredoc so that you don't need to escape the backslashes. It will make things more readable

@avit
avit added a note

Cheers. Today I learned...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@avit avit commented on the diff
lib/Doctrine/DBAL/SQLParserUtils.php
@@ -32,6 +32,13 @@
*/
class SQLParserUtils
{
+ const POSITIONAL_TOKEN = '\?';
+ const NAMED_TOKEN = ':[a-zA-Z_][a-zA-Z0-9_]*';
+
+ // Quote characters within string literals can be preceded by a backslash.
+ const ESCAPED_SINGLE_QUOTED_TEXT = "'(?:[^'\\\\]|\\\\'|\\\\\\\\)*'";
+ const ESCAPED_DOUBLE_QUOTED_TEXT = '"(?:[^"\\\\]|\\\\"|\\\\\\\\)*"';
+
@avit
avit added a note

These backslashes are also doubled due to PHP quoting. I could make them "nowdoc" as well, if it reads better. (The indentation is ugly though.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@beberlei
Owner

Can you rebase this to master? Then i will gladly merge it.

@avit avit referenced this pull request
Closed

Fixed SQL placeholder parsing #216

@avit

@beberlei I did rebase this for you at the time last year... If you still want to use this PR, let me know if you need it rebased once more. (I'm not using Symfony anymore, so I haven't been keeping up to date with recent updates.)

@mvrhov

IMO this would be a nice addition to the 2.4 ....

@beberlei
Owner

@avit i want this PR, i tried rebasing but its not so easy anymore :-( If you could try as well it would be great.

@avit

@beberlei Please try the freshly rebased code: tests are passing on my end. The only significant change was updating from:

array(':placeholder' => array(22, 44))
# to
array(22 => 'placeholder', 44 => 'placeholder')

I hope it's right.

@avit

Another conflict was with the recent 1105261 (Ticket DBAL-389), but it looks like my original regex covered that too. Tests are green, but @roverwolf can verify to be sure?

@avit

@beberlei will you be able to review and merge this soon? I can't keep updating this... I'm forgetting more and more PHP every day! :wink:

@beberlei
Owner

@avit yes, will review today then merge. Sorry to keep you waiting os long

@roverwolf

Verifying that this does seem to fix the issue from DBAL-389 as well. Thanks (this seems much cleaner as well).

@guilhermeblanco

Patch is good. We can still improve it a bit, removing the else condition, but it can be done later.

@guilhermeblanco guilhermeblanco merged commit f8604e1 into doctrine:master

1 check passed

Details default The Travis build passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
53 lib/Doctrine/DBAL/SQLParserUtils.php
@@ -32,6 +32,13 @@
*/
class SQLParserUtils
{
+ const POSITIONAL_TOKEN = '\?';
+ const NAMED_TOKEN = ':[a-zA-Z_][a-zA-Z0-9_]*';
+
+ // Quote characters within string literals can be preceded by a backslash.
+ const ESCAPED_SINGLE_QUOTED_TEXT = "'(?:[^'\\\\]|\\\\'|\\\\\\\\)*'";
+ const ESCAPED_DOUBLE_QUOTED_TEXT = '"(?:[^"\\\\]|\\\\"|\\\\\\\\)*"';
+
@avit
avit added a note

These backslashes are also doubled due to PHP quoting. I could make them "nowdoc" as well, if it reads better. (The indentation is ugly though.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
/**
* Get an array of the placeholders in an sql statements as keys and their positions in the query string.
*
@@ -49,27 +56,18 @@ static public function getPlaceholderPositions($statement, $isPositional = true)
return array();
}
- $count = 0;
- $inLiteral = false; // a valid query never starts with quotes
- $stmtLen = strlen($statement);
+ $token = ($isPositional) ? self::POSITIONAL_TOKEN : self::NAMED_TOKEN;
$paramMap = array();
- for ($i = 0; $i < $stmtLen; $i++) {
- if ($statement[$i] == $match && !$inLiteral && ($isPositional || $statement[$i+1] != '=')) {
- // real positional parameter detected
+
+ foreach (self::getUnquotedStatementFragments($statement) as $fragment) {
+ preg_match_all("/$token/", $fragment[0], $matches, PREG_OFFSET_CAPTURE);
+ foreach ($matches[0] as $placeholder) {
if ($isPositional) {
- $paramMap[$count] = $i;
+ $paramMap[] = $placeholder[1] + $fragment[1];
} else {
- $name = "";
- // TODO: Something faster/better to match this than regex?
- for ($j = $i + 1; ($j < $stmtLen && preg_match('(([a-zA-Z0-9_]{1}))', $statement[$j])); $j++) {
- $name .= $statement[$j];
- }
- $paramMap[$i] = $name; // named parameters can be duplicated!
- $i = $j;
+ $pos = $placeholder[1] + $fragment[1];
+ $paramMap[$pos] = substr($placeholder[0], 1, strlen($placeholder[0]));
}
- ++$count;
- } else if ($statement[$i] == "'" || $statement[$i] == '"') {
- $inLiteral = ! $inLiteral; // switch state!
}
}
@@ -180,4 +178,23 @@ static public function expandListParameters($query, $params, $types)
return array($query, $paramsOrd, $typesOrd);
}
-}
+
+ /**
+ * Slice the SQL statement around pairs of quotes and
+ * return string fragments of SQL outside of quoted literals.
+ * Each fragment is captured as a 2-element array:
+ *
+ * 0 => matched fragment string,
+ * 1 => offset of fragment in $statement
+ *
+ * @param string $statement
+ * @return array
+ */
+ static private function getUnquotedStatementFragments($statement)
+ {
+ $literal = self::ESCAPED_SINGLE_QUOTED_TEXT . '|' . self::ESCAPED_DOUBLE_QUOTED_TEXT;
+ preg_match_all("/([^'\"]+)(?:$literal)?/s", $statement, $fragments, PREG_OFFSET_CAPTURE);
+
+ return $fragments[1];
+ }
+}
View
8 tests/Doctrine/Tests/DBAL/SQLParserUtilsTest.php
@@ -28,6 +28,13 @@ static public function dataGetPlaceholderPositions()
array("SELECT '?' FROM foo", true, array()),
array('SELECT "?" FROM foo WHERE bar = ?', true, array(32)),
array("SELECT '?' FROM foo WHERE bar = ?", true, array(32)),
+ array(
+<<<'SQLDATA'
+SELECT * FROM foo WHERE bar = 'it\'s a trap? \\' OR bar = ?
+AND baz = "\"quote\" me on it? \\" OR baz = ?
+SQLDATA
+ , true, array(58, 104)
+ ),
// named
array('SELECT :foo FROM :bar', false, array(7 => 'foo', 17 => 'bar')),
@@ -37,6 +44,7 @@ static public function dataGetPlaceholderPositions()
array('SELECT :foo_id', false, array(7 => 'foo_id')), // Ticket DBAL-231
array('SELECT @rank := 1', false, array()), // Ticket DBAL-398
array('SELECT @rank := 1 AS rank, :foo AS foo FROM :bar', false, array(27 => 'foo', 44 => 'bar')), // Ticket DBAL-398
+ array('SELECT * FROM Foo WHERE bar > :start_date AND baz > :start_date', false, array(30 => 'start_date', 52 => 'start_date')) // Ticket GH-113
);
}
Something went wrong with that request. Please try again.