SQL lexer fails to detect string end if it the last character is U+FEFF ZERO WIDTH NO-BREAK SPACE

[monetdb-team]

Date: 2014-12-17 00:35:11 +0100
From: sorear
To: SQL devs <>
Version: 11.19.7 (Oct2014-SP1)

Last updated: 2015-01-29 14:07:41 +0100

Comment 20533

Date: 2014-12-17 00:35:11 +0100
From: sorear

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
Build Identifier:

If the last character of a string literal is a U+FEFF, then the lexer will spuriously generate a "waiting for more input" response.

Reproducible: Always

Steps to Reproduce:

1. start mclient
2. enter "select ' ';", run it, see it return a value
3. paste a zero-width no-break space (aka a "byte order mark") immediately before the closing quote, and try to run it again

Actual Results:

MonetDB considers the version with just spaces to be a complete SQL statement, but responds to the version with the ZWNBS with a secondary prompt (my actual code uses MAPI, and receives a \1\2\n).

Expected Results:

Both versions are complete SQL statements.

MonetDB 5 server v11.19.7 "Oct2014-SP1" (64-bit, 64-bit oids)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved
Visit http://www.monetdb.org/ for further information
Found 1.8GiB available memory, 2 available cpu cores
Libraries:
libpcre: 7.8 2008-09-05 (compiled with 7.8)
openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013)
libxml2: 2.7.6 (compiled with 2.7.6)
Compiled by: gt@sorear1.dev.gudtech.com (x86_64-unknown-linux-gnu)
Compilation: gcc -O3 -fomit-frame-pointer -pipe -march=native -D_FORTIFY_SOURCE=2
Linking : /usr/bin/ld -m elf_x86_64

Comment 20545

Date: 2014-12-23 09:51:22 +0100
From: MonetDB Mercurial Repository <>

Changeset 72699051b01e made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=72699051b01e

Changeset description:

The server should not treat U+FEFF specially, that's an interface issue.
This fixes bug #3641.
Also added test.

Comment 20607

Date: 2015-01-29 14:07:41 +0100
From: @sjoerdmullender

Oct2014-SP2 has been released.