Need help for Python error: wrong char code outside of range [0..$10ffff]

[Alexey-T]

I use this fork
https://github.com/alexey-t/python-for-lazarus
In the folder demo_lazarus I have a demo, which shows the problem. I cannot test this on Delphi (no Delphi here, I use Linux Ubuntu 18.x x64, Python 3.7).

• i run the demo
• demo provides Py module "demo" with function "s1". So i test it.
• in demo console, i enter

import demo
=demo.s1()

and have this error.

This is from Unicode string with high values:

function Py_s1(Self, Args : PPyObject): PPyObject; cdecl;
const
  S0: string = 'begin.𠏽𠏽𠏽𠏽𠏽.end';
begin
  with GetPythonEngine do
    Result:= PyString_FromString(PChar(S0));
end;

Can you repro this on Delphi?

[pyscripter]

PyString_FromString is expecting an Ansichar,

function TPythonEngine.PyString_FromString( str: PAnsiChar): PPyObject;
var
  _text : UnicodeString;
begin
  if IsPython3000 then
  begin
    _text := UnicodeString(str);
    Result := PyUnicode_FromWideString(_text);
  end
  else
    Result := DLL_PyString_FromString(str);
end; 

What is the default string type? What is the encoding of the source file?

𠏽 is U+203FD (two words. four bytes). You can debug and see what str and _text is.
It would be better to call PyUnicode_FromWideString directly with the correct UTF16 representation.

See also:
https://stackoverflow.com/questions/33459384/unicode-character-not-in-range-when-calling-locale-strxfrm

[Alexey-T]

What is the default string type? What is the encoding of the source file?

• string type is utf8 encoded string
• utf8, see the $codepage in that demo.

[Alexey-T]

In the engine there is code which converts unicodestring to Ucs4string. I guess it fails somehow..

[pyscripter]

function TPythonEngine.PyUnicode_FromWideString( const AString : UnicodeString) : PPyObject;
{$IFDEF LINUX}
var
  _ucs4Str : UCS4String;
{$ENDIF}
begin
{$IFDEF LINUX}
  // Note that Linux uses UCS4 strings, whereas it declares using UCS2 strings!!!
  _ucs4Str := WideStringToUCS4String(AString);
  Result := PyUnicode_FromWideChar( {PWideChar}(@_ucs4Str[0]), Length(AString) );
{$ELSE}
  Result := PyUnicode_FromWideChar( PWideChar(AString), Length(AString) );
{$ENDIF}
end;

Again you can debug and see the value of AString and ucs4Str. This will tell you where it fails.

[pyscripter]

By the way PyScripter handles the printing of the character correctly.

[Alexey-T]

Will see it.

Did you test PyScripter to pass this char from Delphi code to python code?

[pyscripter]

In PyScripter everything is UTF16 (UnicodeString), so I do not get such conversions.

[Alexey-T]

But you will anyway go through Wide->UCS4 conversion, so need to test it.

[pyscripter]

In Windows Python expects UTF16. not UCS4. Look at TPythonEngine.PyUnicode_FromWideString.

[Alexey-T]

yes, right.

[pyscripter]

Pls let me know if the fault is with P4D (I doubt it).

[Alexey-T]

OMG

  // Note that Linux uses UCS4 strings, whereas it declares using UCS2 strings!!!
  _ucs4Str := WideStringToUCS4String(AString);
  Result := PyUnicode_FromWideChar( {PWideChar}(@_ucs4Str[0]), Length(AString) );

seems we need length of _ucs4str, as last param. Coz for surrogate pair, this length is smaller than original len. will test..

@Artem3213212 Funny bug

[pyscripter]

Yes, we should use the length of _ucs4Str.

[Alexey-T]

It was the reason, pls fix too
Alexey-T/Python-for-Lazarus@70662b0