New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unrecognized token error #1270

Closed
isrep opened this Issue Dec 15, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@isrep
Copy link

isrep commented Dec 15, 2017

Details for the issue

Selecting values in double quotes containing double hyphen substring returns "unrecognized token" error.

Please see example below.
--Error unrecognized token:
select "anything--something"

--no error
select 'anything--something'

Useful extra information

I'm opening this issue because:

  • DB4S is crashing
  • DB4S has a bug
  • DB4S needs a feature
  • DB4S has another problem

I'm using DB4S on:

  • Windows: ( version: ___ )
  • Linux: ( _distro:Ubuntu )
  • Mac OS: ( version: ___ )
  • Other: ___

I'm using DB4S version:

  • 3.10.1
  • 3.10.0
  • 3.9.1
  • Other: 8896ecc

I have also:

@justinclift justinclift added the bug label Dec 15, 2017

@justinclift

This comment has been minimized.

Copy link
Member

justinclift commented Dec 15, 2017

Ouch, that definitely sounds like a bug. Have you tried any of the commits prior to that, to see if that's the one which introduces it?

@isrep

This comment has been minimized.

Copy link
Author

isrep commented Dec 16, 2017

The same error with version 3.10.99 from 2017-12-09, I see on my system. Thank you for looking into it.

@justinclift

This comment has been minimized.

Copy link
Member

justinclift commented Dec 16, 2017

Oops. Just realised something that should have been obvious to me right away before.

This is probably not a bug after all. With SQL, the single and double quote characters aren't the same thing.

  • Single quote characters ' are used to start and end strings. eg:
SELECT foo FROM bar WHERE baz = 'stuff';
  • Double quote characters " are used to start and end identifiers (such as field names). eg:
SELECT "my field with spaces in its name" FROM bar;

In the above example, the table bar contains a field called my field with spaces in its name.

Further reference info, in case it's useful:

    https://www.sqlite.org/lang_keywords.html

With your double quote example, the "unrecognized token" message is probably meaning that it's looking for a field called (literally) "anything--something", and not finding it. 😉

Does that help?

@justinclift justinclift removed the bug label Dec 16, 2017

@isrep

This comment has been minimized.

Copy link
Author

isrep commented Dec 16, 2017

Thank you for pointing to SQLite documentation.
I have bumped into this while operating with strings already containing single quotes.
It seems that official sqlite3 command line tool does not produce "unrecognized token" error.
Thanks anyway.

sqlite3
SQLite version 3.21.0 2017-10-24 18:55:49
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> select 'anything--something';
anything--something
sqlite> select "anything--something";
anything--something
sqlite>

sqlite> create table bar (c1 TEXT);
sqlite> insert into bar (c1) values ('TEXT1');
sqlite> insert into bar (c1) values ("TEXT2");
sqlite> insert into bar (c1) values ("TEXT2---something");
sqlite> SELECT "my field with spaces in its name" from bar;
my field with spaces in its name
my field with spaces in its name
my field with spaces in its name
sqlite> SELECT "my field with spaces in its name", c1 from bar;
my field with spaces in its name|TEXT1
my field with spaces in its name|TEXT2
my field with spaces in its name|TEXT2---something
sqlite>
@justinclift

This comment has been minimized.

Copy link
Member

justinclift commented Dec 16, 2017

Hmmm, yeah you're right. I've just tried it here with the sqlite3 client (and older version, 3.7.17 on my CentOS 7 desktop), and a recent build of DB4S.

It looks like DB4S is getting misunderstanding the double quotes. For example, with this (from above):

insert into bar (c1) values ("TEXT2");

In the sqlite3 CLI it works, adding a new row into the table bar. DB4S gives this error instead:

unrecognized token: ""TEXT2": 

Looking at that error message, there's an extra " at the start of the TEXT2 string. Guessing here, but this looks like we're somehow getting the quotes wrong after all.

So... yep this does seem like a bug. Our Execute SQL tab should accept anything that the SQLite CLI does. Thanks for being persistent. 😀

@justinclift justinclift added the bug label Dec 16, 2017

@justinclift

This comment has been minimized.

Copy link
Member

justinclift commented Dec 19, 2017

I'm kind of wondering if this is related to the regular expression crafted for #1004.

MKleusberg added a commit that referenced this issue Jan 5, 2018

Fix removing comments from SQL when there are quoted strings
In the code for removing comments from SQL statements we have to make
sure to only match the '--' characters when they are not inside a quoted
string or identifier. This works fine and as expected for single quotes.
However, for double quotes it doesn't. This is fixed by this commit.

See issue #1270.
@MKleusberg

This comment has been minimized.

Copy link
Member

MKleusberg commented Jan 5, 2018

Thanks for pointing this out, @isrep! I think I managed to tweak the regular expression to not screw up your SQL statement. Can you give tomorrow's nightly build a try and see if it's working for you? 😃

@MKleusberg

This comment has been minimized.

Copy link
Member

MKleusberg commented Jan 5, 2018

Sorry, I'm going to revert my changes. So no need to test the nightly.

It's introducing new issues which I can't solve right now 😦 If anybody with leet regular expression skills wants to try themselves, please go ahead 😉

@justinclift

This comment has been minimized.

Copy link
Member

justinclift commented Jan 5, 2018

@h4yn0nnym0u5e @revolter Any interest? 😄

@MKleusberg

This comment has been minimized.

Copy link
Member

MKleusberg commented Jan 5, 2018

You can check out my fumblings in eae0730 and 29fa332.

We need to make sure it's working for all these statements:

SELECT "something--something" -- comment
SELECT 'something--something' -- comment
SELECT "something--some'thing" -- comment
SELECT 'something--some"thing' -- comment

And as a bonus maybe for these too:

SELECT `something--something` FROM table -- comment
SELECT [something--something] FROM table -- comment

And for these:

SELECT '/*test*/' -- comment
SELECT "/*test*/" -- comment
SELECT '/*te"st*/' -- comment
SELECT "/*te'st*/" -- comment
SELECT `/*test*/` FROM table -- comment
SELECT [/*test*/] FROM table -- comment

@mgrojo mgrojo referenced this issue Mar 5, 2018

Closed

"Execute SQL" strips newlines #1334

6 of 14 tasks complete
@mgrojo

This comment has been minimized.

Copy link
Contributor

mgrojo commented Sep 1, 2018

I'm going to close this issue because all of this is solved after #1455 since we no longer try to strip comments before passing SQL code to SQLite for execution.

@mgrojo mgrojo closed this Sep 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment