Incorrect regex for matching the String.Single token

[john-bodley]

The regex for determining a tokens.String.Single seems incorrect, i.e., it is too greedy, when the SQL contains an escape character:

>>> from sqlparse import parse

>>> stmt = parse(r"foo = '\' AND bar = 'baz'")[0]
>>> stmt._pprint_tree()
|- 0 Identifier 'foo = ...'
|  |- 0 Comparison 'foo = ...'
|  |  |- 0 Identifier 'foo'
|  |  |  `- 0 Name 'foo'
|  |  |- 1 Whitespace ' '
|  |  |- 2 Comparison '='
|  |  |- 3 Whitespace ' '
|  |  `- 4 Single ''\' AN...'
|  `- 1 Identifier 'baz'
|     `- 0 Name 'baz'
`- 1 Error "'"

I'm not sure why the ' is present in ...|\\'|... and was wondering whether this should be removed.

[villebro]

To my knowledge, MySQL and BigQuery are one of the few databases that support escaping single quotes using a backslash: https://dev.mysql.com/doc/refman/8.0/en/string-literals.html#character-escape-sequences and https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical#literals. But all other databases I can think of will retain the backslash when when placed before the single quote:

Postgres, sqlite, Oracle, MSSQL, Presto, Athena etc

SELECT '\'' as test

ERROR

SELECT '\''' as test

\'

MySQL, BigQuery:

SELECT '\'' as test

'

SELECT '\''' as test

ERROR