Exporting large Arrow array crashes with cryptic error

[mauropagano]

This might very well being one of those "you shouldn't be doing this", but it raises an error that is not very user friendly

oracledb.exceptions.ProgrammingError: DPY-2060: Arrow C Data Interface operation failed with error code 75

1. What versions are you using?
   3.3DB is 19c but shouldn't matter

2. Is it an error or a hang or a crash?
   Crash, DPY-2060

3. What error(s) or behavior you are seeing?
   DPY-2060

4. Does your application call init_oracle_client()?
   Thin, but shouldn't matter

5. Include a runnable Python script that shows the problem.

import oracledb

conn = oracledb.connect(...")

with conn.cursor() as cursor:

    try:
        print("Dropping table")
        cursor.execute("drop table oracle_crash")
    except Exception:
        print("Couldn't drop, moving on")
        pass

    print("Creating source table, might take a bit as it's 100M rows")
    cursor.execute(
        """
        create table oracledb_crash parallel 4 as
          select 'THIS IS A TEST TO BLOW THINGS UP' c1
            from (select * from dual connect by rownum <= 10000),
                 (select * from dual connect by rownum <= 10000)
    """
    )


df = conn.fetch_df_all("select * from oracledb_crash", arraysize=10000)
print(dir(df))

[mauropagano]

Omitted one bit, I understand the 75 is likely EOVERFLOW (that's why my opening line "maybe you shouldn't be doing this").

Maybe the user facing error could be a little more informative?

[cjbj]

Our stress tester pointed out that he saw this behavior last month and logged an Oracle bug. (For the record, it's bug 38280378).

[mauropagano]

Thanks for sharing!
Unfortunately it's INTERNAL so I can't see much about it.

Out of curiosity though, why is it an RDBMS bug? Asking because from the code that that seems more around nanoarrow (not claiming it's a bug or not, I really don't know)

[anthony-tuininga]

That bug is not an RDBMS bug but a python-oracledb bug. What we are planning on doing is changing the code to use the large variants of strings and binary values in order to avoid this issue. You will also be able to explicitly use the short variants of strings and binary values specifically if you wish to save the memory associated with using 32-bit offsets instead of 64-bit offsets.

[mauropagano]

Thank you Anthony, makes sense.

Is my understanding wrong or the overhead of defaulting to LargeString is 4 bytes per element?
Asking because if yes then maybe the benefit of being able to pick shorter version is negligible?

FWIW this was a good excuse to switch some of my code to RecordBatches

[anthony-tuininga]

Yes, using LargeString requires an additional 4 bytes per element. The difference isn't going to be substantial unless you have a large number of rows with very short strings -- which in most cases won't be important. You have the option, however, if that memory matters!

[anthony-tuininga]

I have pushed a patch that should correct this issue and have initated a build from which you can download pre-built development wheels once it completes. You can also build from source if you prefer. If you can test your scenario and confirm the patch works as expected, that would be appreciated!

[anthony-tuininga]

This was included in python-oracledb 3.4.0 which was just released.