Crash by potentially use of bad escape characters

[suyZhong]

Describe the bug
Consider the test cases below. It would crash the monetdb server. The last lines of the server log (merovingian.log):

2024-02-20 13:39:53 ERR monetdb[624]: #client9246: GDKstrFromStr: ERROR: not a Unicode code point escape
2024-02-20 13:39:53 ERR monetdb[624]: #client9246: createExceptionInternal: ERROR: MALException:mal.interpreter:GDK reported error: GDKstrFromStr: not a Unicode code point escape
2024-02-20 13:39:53 ERR monetdb[624]: mserver5: /app/MonetDB/sql/backends/monet5/sql_cast.c:137: SQLbatstr_cast: Assertion `digits >= 0' failed.
2024-02-20 13:39:54 MSG merovingian[159]: database 'monetdb' (624) has crashed with signal SIGABRT (dumped core)

To Reproduce

Sorry for the long query, we haven't implemented a reducer for crashes yet, and it's hard for me to manually reduce it more. Sometimes I'll encounter the error message: Could not allocate space from mclient when trying to reduce the query, not sure if it's expected.

To reproduce, execute the mclient using docker exec -i monetdb-test mclient -E UTF-8 -d monetdb and copy the following queries:

DROP TABLE IF EXISTS t0 CASCADE;
DROP TABLE IF EXISTS t1 CASCADE;

CREATE TABLE t0(c0 BOOLEAN, c1 BOOLEAN, PRIMARY KEY(c0));
CREATE TABLE t1(c0 INTEGER, c1 INT, PRIMARY KEY(c0));
INSERT INTO t0(c1) VALUES (-2041868105), (true);
INSERT INTO t0(c1, c0) VALUES (-1820721628, -337528041);
INSERT INTO t0(c1) VALUES (true);
INSERT INTO t1(c0) VALUES (true);


SELECT * FROM t1, t0;
SELECT * FROM t1, t0 WHERE (CASE t0.c0 WHEN (t0.c0 IN (((t1.c0)<=(true)))) THEN (((-953136241) IS NOT NULL) NOT BETWEEN (CASE true WHEN '_hf4{r' THEN t0.c0 WHEN '\Uwc0g8g' THEN t1.c1 WHEN false THEN '' ELSE t1.c1 END ) AND 1239880767) ELSE t0.c1 END );

Expected behavior
Should return an error message, or return the correct result.

Screenshots
N/A

Software versions
Build from source:

> docker exec monetdb-test mserver5 --version --dbname=monetdb
MonetDB 5 server 11.50.0 (hg id: d333af8) (64-bit, 128-bit integers)
This is an unreleased version
Copyright (c) 1993 - July 2008 CWI
Copyright (c) August 2008 - 2023 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 503.5GiB available memory, 128 available cpu cores
Libraries:
Compiled by: @6421299871b6 (x86_64-pc-linux-gnu)
Compilation: /usr/bin/cc 
Linking    : /usr/bin/ld

Issue labeling
Bug

Additional context
Kindly inform me if you need more information (e.g. error trace, etc.)

[sjoerdmullender]

Fixed in Dec2023 branch (soon in default as well).