sage.libs.ecl: Fix unicode handling #30106

mkoeppe · 2020-07-10T21:11:07Z

As a follow-up to #29278, #29280: If we use Unicode variable names in SR, declaring a domain gives an error:

sage: SR.var('π', domain='real')
RuntimeError: ECL says: THROW: The catch MACSYMA-QUIT is undefined.
SystemError: <built-in method var of sage.symbolic.ring.SymbolicRing object at 0x334506908> returned a result with an error set

This comes from our ECL interface:

sage: from sage.libs.ecl import *
sage: u_symbol = EclObject('🔥')
sage: u_symbol
<repr(<sage.libs.ecl.EclObject at 0x337e7b3c8>) failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 2: invalid start byte>
sage: u_symbol.python()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 2: invalid start byte

Also note:

sage: b_symbol = EclObject(bytes([166]))
sage: b_symbol
<repr(<sage.libs.ecl.EclObject at 0x337e7b058>) failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 0: invalid start byte>

Depends on #22191

CC: @mwageringel @nbruin @dimpase @spaghettisalat

Component: symbolics

Author: Matthias Koeppe

Branch/Commit: 59dd62b

Reviewer: Markus Wageringel

Issue created by migration from https://trac.sagemath.org/ticket/30106

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-07-11T00:25:02Z

comment:3

Not sure if there is a function in the ECL C API that constructs a Unicode Lisp string.
So I would just be going through CODE-CHAR as in

sage: ecl_eval('''(type-of (map 'string #'code-char '(128293))))''')
<ECL: (SIMPLE-ARRAY CHARACTER (1))>

mkoeppe · 2020-07-11T01:31:31Z

Branch: u/mkoeppe/sage_libs_ecl__fix_unicode_handling

sagetrac-git · 2020-07-11T02:44:29Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b6ebb52`	`src/sage/cpython/string: Update documentation`
`6ce8f9e`	`python_to_ecl: Handle unicode strings`

sagetrac-git · 2020-07-11T02:44:29Z

Commit: 6ce8f9e

mkoeppe · 2020-07-11T02:45:21Z

comment:7

Now

sage: SR.var('π', domain='real')
π

but more work is needed.

sagetrac-git · 2020-07-11T03:16:06Z

Changed commit from 6ce8f9e to c983523

sagetrac-git · 2020-07-11T03:16:06Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c983523`	`EclObject.str: Handle unicode`

mkoeppe · 2020-07-11T03:17:01Z

comment:9

sage: from sage.libs.ecl import *
sage: u_symbol = EclObject('🔥')
sage: u_symbol
<ECL: 🔥>

sagetrac-git · 2020-07-11T04:28:51Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f237935`	`ecl_to_python: Handle unicode strings`

sagetrac-git · 2020-07-11T04:28:51Z

Changed commit from c983523 to f237935

mkoeppe · 2020-07-11T04:30:07Z

comment:11

sage: u_string = EclObject('"🐙"')
sage: u_string
<ECL: "🐙">
sage: _.python()
'"🐙"'

sagetrac-git · 2020-07-11T04:40:53Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6f740eb`	`ecl_eval: Handle unicode strings`

sagetrac-git · 2020-07-11T04:40:53Z

Changed commit from f237935 to 6f740eb

mkoeppe · 2020-07-11T04:41:38Z

comment:13

sage: clock = ecl_eval('''(defun clock (h) (string (elt "🕛🕐🕑🕒🕓🕔🕕🕖🕗🕘🕙🕚" (mod h 12))))''')
sage: clock(3).python()
'"🕒"'

mkoeppe · 2020-07-11T04:42:16Z

Author: Matthias Koeppe

mwageringel · 2020-07-11T13:25:14Z

comment:15

Thank you for solving this.

Should our Maxima interface support unicode with this branch now, or do you know what else is needed to make this work? The following used to give an error, but results in a crash now:

sage: var('ξ')._maxima_()

Condition of type: SIMPLE-ERROR
Cannot coerce string Cannot coerce string $_SAGE_VAR_ξ to a base-string to a base-string
No restarts available.
...
Excessive debugger depth! Probable infinite recursion!
Quitting.

Also, please add a doctest for the new functionality.

dimpase · 2020-07-11T14:33:11Z

comment:16

Are you on ecl 20.4 ?

mkoeppe · 2020-07-11T15:37:24Z

comment:17

Replying to @mwageringel:

The following used to give an error, but results in a crash now:

sage: var('ξ')._maxima_()

Condition of type: SIMPLE-ERROR
Cannot coerce string Cannot coerce string $_SAGE_VAR_ξ to a base-string to a base-string
No restarts available.
...
Excessive debugger depth! Probable infinite recursion!
Quitting.

Yes, I can confirm this here. I'll look into this.

Also, please add a doctest for the new functionality.

Sure, will do.

mkoeppe · 2020-07-11T15:38:09Z

comment:18

Replying to @dimpase:

Are you on ecl 20.4 ?

I'm still on 16.1.2.

sagetrac-git · 2020-07-11T15:50:44Z

Changed commit from 6f740eb to ee97a6d

sagetrac-git · 2020-07-11T15:50:44Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ee97a6d`	`Fix infinite recursion when error messages are not base-strings`

dimpase · 2020-07-11T16:35:17Z

comment:20

I suggest we work on ecl 20.4, for uniformity

mkoeppe · 2020-07-11T16:56:47Z

comment:21

The above error is due to a bug in ECL 16.1.2:

> (princ-to-string 'ξ)
"Ξ"
> (setf local-table (copy-readtable nil))
> (setf (readtable-case local-table) :invert)
> :INVERT
> (let ((*readtable* local-table) (*print-case* :upcase)) (princ-to-string 'ξ))
Cannot coerce string Ξ to a base-string

I will try with the ECL upgrade now.

sagetrac-git · 2020-07-11T18:17:11Z

Changed commit from ee97a6d to 2d87ee3

sagetrac-git · 2020-07-11T18:17:11Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`0076bc3`	`backport Maxima fix for bug #3629`
`1d074e8`	`doctest fixes`
`266d8c1`	`backport ECL PR #210`
`12447bc`	`reject old makeinfo`
`a3e0eca`	`add upstream fix from MR 215`
`89b006b`	`add the patch from upstream MR 214`
`0b77737`	`add upstream MR 216 (to fix cygwin fork)`
`8ca1c0e`	`Merge tag '9.2.beta2' into t/22191/public/packages/ecl20`
`f82c716`	`Commit 75877dd8 from upstream`
`2d87ee3`	`Merge commit 'f82c716fdf9c6e91a07166d36b6329a15ecfb41d' of git://trac.sagemath.org/sage into t/30106/sage_libs_ecl__fix_unicode_handling`

sagetrac-git · 2020-07-11T18:42:22Z

Branch pushed to git repo; I updated commit sha1. New commits:

`828a727`	`Add doctests`

sagetrac-git · 2020-07-11T18:42:22Z

Changed commit from 2d87ee3 to 828a727

mkoeppe · 2020-07-11T18:45:47Z

comment:24

Replying to @mkoeppe:

The above error is due to a bug in ECL 16.1.2:

> (princ-to-string 'ξ)
"Ξ"
> (setf local-table (copy-readtable nil))
> (setf (readtable-case local-table) :invert)
> :INVERT
> (let ((*readtable* local-table) (*print-case* :upcase)) (princ-to-string 'ξ))
Cannot coerce string Ξ to a base-string

I will try with the ECL upgrade now.

This bug is still present in ECL 20.4.24. The above code is from Maxima's PRINT-INVERT-CASE function in commac.lisp.

mkoeppe · 2020-07-11T18:48:38Z

Dependencies: #22191

mkoeppe · 2020-07-11T19:12:46Z

comment:27

Note that after commit ee97a6d, the test case from comment 17 now gives a proper error message, no longer a crash:

sage: var('ξ')._maxima_()
....: 
<repr(<sage.interfaces.maxima_lib.MaximaLibElement at 0x32c1d4828>) failed: RuntimeError: ECL says: Cannot coerce string $_SAGE_VAR_ξ to a base-string>

Fixing this ECL bug or working around it is outside of the scope of this ticket.

mwageringel · 2020-07-12T10:31:27Z

comment:28

Ok, thanks for the fix. There is one more place where ecl_string_to_python should be used – in print_objects:

sage: from sage.libs.ecl import *
sage: u_symbol = EclObject('🔥')
sage: print_objects()  # crashes

Moreover, the docstring of ecl_eval should be changed into a raw string, as the examples contain backslashes.

Once that is fixed, you can set a positive review on my behalf.

mwageringel · 2020-07-12T10:31:27Z

Reviewer: Markus Wageringel

sagetrac-git · 2020-07-12T16:13:22Z

Branch pushed to git repo; I updated commit sha1. New commits:

`99b894a`	`print_objects: Handle unicode`
`59dd62b`	`ecl_eval: Make docstring raw`

sagetrac-git · 2020-07-12T16:13:22Z

Changed commit from 828a727 to 59dd62b

mkoeppe · 2020-07-12T16:14:13Z

comment:30

Thank you!

mkoeppe · 2020-07-12T16:21:03Z

comment:31

Follow-up ticket to keep track of the ECL issue: #30122

vbraun · 2020-07-25T22:51:04Z

Changed branch from u/mkoeppe/sage_libs_ecl__fix_unicode_handling to 59dd62b

mkoeppe added c: symbolics labels Jul 10, 2020

This comment has been minimized.

Sign in to view

mkoeppe added this to the sage-9.2 milestone Jul 11, 2020

mkoeppe removed the wishlist item label Jul 11, 2020

mkoeppe added the s: needs review label Jul 11, 2020

mkoeppe added s: needs work and removed s: needs review labels Jul 11, 2020

mkoeppe added s: needs review and removed s: needs work labels Jul 11, 2020

mkoeppe added s: positive review and removed s: needs review labels Jul 12, 2020

vbraun removed the s: positive review label Jul 25, 2020

vbraun closed this as completed in 8bac45e Jul 25, 2020

embray mentioned this issue Aug 19, 2020

Provide unicode aliases for some globals in Python 3 #28966

Closed

This was referenced Jul 28, 2020

sage.libs.ecl: Make it possible to convert Python strings to Lisp strings #30105

Closed

Meta-ticket: Unicode support #30111

Open

Fix maxima/ecl unicode handling #30122

Closed

sage.libs.ecl: Fix unicode handling #30106

sage.libs.ecl: Fix unicode handling #30106

Comments

mkoeppe commented Jul 10, 2020

This comment has been minimized.

mkoeppe commented Jul 11, 2020

This comment has been minimized.

mkoeppe commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

mwageringel commented Jul 11, 2020

dimpase commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

dimpase commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

sagetrac-git mannequin commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

mkoeppe commented Jul 11, 2020

mwageringel commented Jul 12, 2020

mwageringel commented Jul 12, 2020

sagetrac-git mannequin commented Jul 12, 2020

sagetrac-git mannequin commented Jul 12, 2020

mkoeppe commented Jul 12, 2020

mkoeppe commented Jul 12, 2020

vbraun commented Jul 25, 2020