400 MByte string truncated at 33 MByte

[guettli]

My function module creates a big string containing JSON.

The string is 400 MByte long.

The screenshot shows the variables just before the function module returns the data via rfc to the client.

The client receives the variable EV_RESULT_JSON truncated.

The JSON is truncated and only 33 MByte long.

Since no code of me gets executed between the screenshot and receiving the data I am guess that the issue needs to be in pyRFC or the "sapnwrfc" library.

Related question at StackOveflow: https://stackoverflow.com/questions/55850646/data-gets-truncated-in-sap-rfc

[bsrdjan]

The maximum string size possible in RFC communication is 2GByte and 400 MByte should therefore work. Trying to replicate the issue in my test system, I faced another issue already at ca. 200 MBytes.

To exclude possible byte zero inserted into your 400MB JSON string, leading to 33 MB truncation, would it be possible to implement a following ABAP function module in your system and test with python script below?

ABAP

FUNCTION ZLONG_STRING.
*"----------------------------------------------------------------------
*"*"Local Interface:
*"  IMPORTING
*"     VALUE(IV_STRING) TYPE  STRING DEFAULT 'a'
*"     VALUE(IV_COUNT) TYPE  INT8 DEFAULT 1
*"  EXPORTING
*"     VALUE(EV_STRING) TYPE  STRING
*"----------------------------------------------------------------------

DATA CHUNK_SIZE TYPE I.
DATA REST_SIZE TYPE I.
DATA CHUNK_STRING TYPE STRING VALUE ''.

CHUNK_SIZE = SQRT( IV_COUNT ).
IF ( CHUNK_SIZE * CHUNK_SIZE ) > IV_COUNT.
  CHUNK_SIZE = CHUNK_SIZE - 1.
ENDIF.
REST_SIZE = IV_COUNT - ( CHUNK_SIZE * CHUNK_SIZE ).

EV_STRING = ''.

DO CHUNK_SIZE TIMES.
  "CONCATENATE CHUNK_STRING IV_STRING INTO CHUNK_STRING.
  CHUNK_STRING = |{ CHUNK_STRING }{ IV_STRING }|.
ENDDO.

DO CHUNK_SIZE TIMES.
  " CONCATENATE EV_STRING CHUNK_STRING INTO EV_STRING.
  EV_STRING = |{ EV_STRING }{ CHUNK_STRING }|.
ENDDO.

DO REST_SIZE TIMES.
  " CONCATENATE EV_STRING IV_STRING INTO EV_STRING.
  EV_STRING = |{ EV_STRING }{ IV_STRING }|.
ENDDO.

ENDFUNCTION.

Python

# -*- coding: utf-8 -*

from pyrfc import Connection

conn = Connection(**ABAP)

MAXLEN = 214748364  # 0xCCC CCCC works

LENGTH = MAXLEN+1  # 0xCCC CCCD fails

try:
    result = conn.call('ZLONG_STRING', IV_COUNT=LENGTH)

    print 'result', \
        LENGTH - len(result['EV_STRING']), \
        LENGTH, result['EV_STRING'][:22]
except Exception as ex:
    print ex

In my test system, the test ends with a shortdump, when IV_COUNT bigger than 214,748,364 bytes (0xCCC CCCC).

Could you please test in your system and check if any shortdump recorded (transaction ST22) ?

[bsrdjan]

Just a side question related to ABAP, Python and JSON encoding in general.

Passing big JSON payload from ABAP, over the network and across SAP NW RFC SDK stack to Python, could be IMHO slower than passing plain ABAP data to Python and running JSON encoding in Python (e.g. simplejson). Have you tried both these possibilities and eventually experienced advantages/disadvantages of one or another way?

[guettli]

@bsrdjan thank very much you for your reply. I know that json is bloated compared to RFC (binary) data transfer. But in my case the function module should be generic. The data structure is not known before.

I asked here, and I was told that returning N tables of generic structure is not possible. If you know a better solution (without json), then I am curious: https://stackoverflow.com/questions/55419056/return-rows-of-tables-known-at-runtime-only-via-rfc

I will try your rfc abap code and tell you.

[guettli]

my system does not not int8. I took integer. SAP 7.4 SP 0013.

BTW I had #89 again. First I accidentally did not set "remote capable function module.".

Then I set "remote capable", and activated it. But when I called my python script I got (like before) CALL_FUNCTION_NOT_REMOTE. I activated it several times. Some result. It looks like old abap code gets executed. Then I ran "test function module". After this, my python script called the ZLONG_STRING.

[guettli]

The result of the python script:

user@host:~$ python tmp/json-cut-issue.py
result 214748365 214748365 

This means the result is empty.

[guettli]

I modified your example a bit, but it works nice until the limit at 2.1 GByte gets reached:

ABAP:

FUNCTION /MLK/ZLONG_STRING .
*"----------------------------------------------------------------------
*"*"Lokale Schnittstelle:
*"  IMPORTING
*"     VALUE(IV_STRING) TYPE  STRING DEFAULT 'a'
*"     VALUE(IV_COUNT) TYPE  INTEGER DEFAULT 1
*"  EXPORTING
*"     VALUE(EV_STRING) TYPE  STRING
*"     VALUE(EV_STRLEN) TYPE  INTEGER
*"----------------------------------------------------------------------


*MESSAGE 'foo' TYPE 'E'.

EV_STRING = ''.

DO IV_COUNT TIMES.
  EV_STRING = |{ EV_STRING }{ IV_STRING }|.
ENDDO.

EV_STRLEN = strlen( ev_string ).

ENDFUNCTION.

Python:

# https://github.com/SAP/PyRFC/issues/97
# -*- coding: utf-8 -*

from pyrfc import Connection
conn = Connection(**ABAP)

for i in range(1000):
    iv_count = i*10*10**6
    result = conn.call('/MLK/ZLONG_STRING', IV_COUNT=iv_count, IV_STRING='a')
    ev_string = result['EV_STRING']
    ev_strlen = result['EV_STRLEN']
    print('iv_count=%s len(ev_string)=%s ev_strlen=%s' % (iv_count, len(ev_string), ev_strlen))

(BTW: if I omit the IV_STRING='a', then the output string is always empty. I guess there is something wrong with the handling the default value).

Using: pyrfc==1.9.94

output:

user@host:~$ python tmp/json-cut-issue.py
iv_count=0 len(ev_string)=0 ev_strlen=0
iv_count=10000000 len(ev_string)=10000000 ev_strlen=10000000
iv_count=20000000 len(ev_string)=20000000 ev_strlen=20000000
iv_count=30000000 len(ev_string)=30000000 ev_strlen=30000000
iv_count=40000000 len(ev_string)=40000000 ev_strlen=40000000
iv_count=50000000 len(ev_string)=50000000 ev_strlen=50000000
iv_count=60000000 len(ev_string)=60000000 ev_strlen=60000000
iv_count=70000000 len(ev_string)=70000000 ev_strlen=70000000
iv_count=80000000 len(ev_string)=80000000 ev_strlen=80000000
iv_count=90000000 len(ev_string)=90000000 ev_strlen=90000000
iv_count=100000000 len(ev_string)=100000000 ev_strlen=100000000
iv_count=110000000 len(ev_string)=110000000 ev_strlen=110000000
iv_count=120000000 len(ev_string)=120000000 ev_strlen=120000000
iv_count=130000000 len(ev_string)=130000000 ev_strlen=130000000
iv_count=140000000 len(ev_string)=140000000 ev_strlen=140000000
iv_count=150000000 len(ev_string)=150000000 ev_strlen=150000000
iv_count=160000000 len(ev_string)=160000000 ev_strlen=160000000
iv_count=170000000 len(ev_string)=170000000 ev_strlen=170000000
iv_count=180000000 len(ev_string)=180000000 ev_strlen=180000000
iv_count=190000000 len(ev_string)=190000000 ev_strlen=190000000
iv_count=200000000 len(ev_string)=200000000 ev_strlen=200000000
iv_count=210000000 len(ev_string)=210000000 ev_strlen=210000000
Traceback (most recent call last):
  File "tmp/json-cut-issue.py", line 30, in <module>
    result = conn.call('/MLK/ZLONG_STRING', IV_COUNT=iv_count, IV_STRING='a')
  File "src/pyrfc/_pyrfc.pyx", line 413, in pyrfc._pyrfc.Connection.call
  File "src/pyrfc/_pyrfc.pyx", line 254, in pyrfc._pyrfc.Connection._error
pyrfc._exception.CommunicationError: RFC_COMMUNICATION_FAILURE (rc=1): key=RFC_COMMUNICATION_FAILURE, message=connection closed without message (CM_NO_DATA_RECEIVED) [MSG: class=, type=, number=, v1-4:=;;;]

Sad, I hoped to be able to reproduce the problem with this script, but up to now I failed.

[bsrdjan]

Sorry, I added INT8 internally, for the next release.

Regarding 400 to 33 MB truncation of JSON, could it be that zero byte is somehow inserted into JSON content on ABAP side (bug in serializer?), leading to truncation?

With REPLACE statement below uncommented, Python receives an empty string. Perhaps you could try adding a small test on ABAP side, after JSON payload generated, to check if unwanted zero byte inserted ?

FUNCTION zlong_string.
*"----------------------------------------------------------------------
*"*"Local Interface:
*"  IMPORTING
*"     VALUE(IV_STRING) TYPE  STRING DEFAULT 'a'
*"     VALUE(IV_COUNT) TYPE  INT8 DEFAULT 1
*"  EXPORTING
*"     VALUE(EV_STRING) TYPE  STRING
*"     VALUE(EV_ZERO_POS) TYPE  INT8
*"----------------------------------------------------------------------

DATA l_null_character TYPE c LENGTH 1.
l_null_character = cl_abap_conv_in_ce=>uccpi( 0 ).

ev_string = ''.
ev_zero_pos = -1.

DO iv_count TIMES.
  ev_string = |{ ev_string }{ iv_string }|.
ENDDO.

" REPLACE FIRST OCCURRENCE OF IV_STRING IN EV_string WITH l_null_character.

" search for zero byte
IF ev_string CA l_null_character.
  ev_zero_pos = SY-FDPOS.
ENDIF.

ENDFUNCTION.

[guettli]

Which other characters can result in truncated strings? I guess this is because in ANSI C \0 terminates strings.

I could switch from string to xstring. Maybe this would help.

I guess (but don't know) that \0 is allowed in xstring - or am I wrong?

In my case the AUFK table seems to contain broken encodings. See my related question at StackOverflow: https://stackoverflow.com/questions/56055822/how-to-solve-rfc-conversion-field-error-in-sap

[bsrdjan]

The ANSI C \0 terminates for sure, other values should not terminate. The xstring is a byte string with all values 0-255 allowed, should therefore work.

RFC_CONVERSION_FIELD shortdumps look like ABAP kernel bug and SAP Note #2788696 might help. Could you please check if relevant for your ABAP kernel release, apply the patch and re-test?

[guettli]

I added this code to my function module:

if ev_result_json CA l_null_character.
  MESSAGE 'ev_result_json contains null_character' TYPE 'E'.
endif.

The error does not happen. But the json is still cut.

Next thing I will try: I will use xstring instead of string.

[guettli]

This is the end of my function module:

ev_result_json = |\{ | &&  concat_lines_of( table = results_json sep = `, `) && | \} |.


*data(utf_8) = cl_abap_codepage=>convert_to(
*               source      = ev_result_json
*               codepage    = `UTF-8` ).
ENDFUNCTION.

The result is ok, if I leave the comment like this. But if I uncomment the comment then there is suddenly a \0 at the end!

The string looks like this:

"TOWNSHIP":"","MC_COUNTY":"","MC_TOWNSHIP":"","XPCPT":""}]  } \x00'

Isn't this very strange?

The part I comment/uncomment should not alter the value of ev_result_json, or am I wrong?

ev_result_json does only contain pure 7-bit ascii. No special characters, no umlauts. I checked it with LESSCHARSET=ascii less /tmp/foo

[guettli]

@bsrdjan you wrote:

Trying to replicate the issue in my test system, I faced another issue already at ca. 200 MBytes.

Have you already found more details about your issue?

[guettli]

For testing I added this useless convert to utf8 and convert back:

data(my_len) = strlen( ev_result_json ).

* TODO remove useless converting
* https://github.com/SAP/PyRFC/issues/97
data(utf_8) = cl_abap_codepage=>convert_to(
               source      = ev_result_json
               codepage    = `UTF-8` ).

ev_result_json = cl_abap_codepage=>convert_from(
               source      = utf_8
               codepage    = `UTF-8` ).

But no useful result. The var my_len is 432933973 and Remote-Debugger shows CString{432933973} for ev_result_json. Same result before and after.

Maybe there is a bug in pyrfc or the sdk?

[PawelJagus]

This is the end of my function module:

ev_result_json = |\{ | &&  concat_lines_of( table = results_json sep = `, `) && | \} |.


*data(utf_8) = cl_abap_codepage=>convert_to(
*               source      = ev_result_json
*               codepage    = `UTF-8` ).
ENDFUNCTION.

The result is ok, if I leave the comment like this. But if I uncomment the comment then there is suddenly a \0 at the end!

The string looks like this:

"TOWNSHIP":"","MC_COUNTY":"","MC_TOWNSHIP":"","XPCPT":""}]  } \x00'

Isn't this very strange?

The part I comment/uncomment should not alter the value of ev_result_json, or am I wrong?

ev_result_json does only contain pure 7-bit ascii. No special characters, no umlauts. I checked it with LESSCHARSET=ascii less /tmp/foo

Can you please test the following one, wheter you get 00 at the end as well?

REPORT ZZZ.

DATA(g_string) = `ąćęłńóśżźĄĆĘŁŃÓŚŻŹ`.

DATA(g_utf8) = CL_ABAP_CODEPAGE=>CONVERT_TO(
  SOURCE = g_string
  CODEPAGE = `UTF-16`
).

BREAK-POINT.

If there is, then there has to be a bug in your kernel version.

[PawelJagus]

I might be wrong but the issue that you are having is a combination of you putting the space at the end of your string in the string template DATA(g_string) = |ąćęłńóśżźĄĆĘŁŃÓŚŻŹ | (the last character - space on my system is notated as 2000 (little endian?)) and possibly a bug in the kernel (method CL_ABAP_CODEPAGE=>CONVERT_TO) which splits somehow the notation of this to x20 and x00 in UTF-8, therefore the null at the end.

As the first workaround try not to put a space at the end of your string template. Change ev_result_json = |\{ | && concat_lines_of( table = results_json sep = , ) && | \} | to ev_result_json = |\{ | && concat_lines_of( table = results_json sep = , ) && | \}|