setinputsizes(CLOB) is extremely slow, do we need to setinputsizes(CLOB)?

[zzzeek]

Hey Anthony -

today on setinputsizes(), apparently CLOB is incredibly slow. Omitting it, an INSERT seems to proceed without issue, and I've tried somewhat larger text sizes up to a few dozen megs in size and it still round trips correctly. The only thing setinputiszes(CLOB) seems to get me is that I can persist a blank string '' without it turning into NULL.

Can I get some clarity on this one? test program below produces:

0.05559845399693586
11.789099370995245

e.g., calling setinputsizes(CLOB) makes a modest insert of 1000 rows 200 times slower, with cx_Oracle 6.3.

import cx_Oracle
import decimal

conn = cx_Oracle.connect(
    user="scott",
    password="tiger",
    dsn=cx_Oracle.makedsn(
        "oracle1120", 1521,
    )
)

def setup():
    cursor = conn.cursor()
    try:
        cursor.execute("drop table t")
    except:
        pass

    cursor.execute("""
    CREATE TABLE t (
        x CLOB
    )
    """)
    return cursor



def go1(cursor):
    cursor.executemany(
        "insert into t (x) values (:x)",
        [
            {"x": "text %s" % i} for i in range(1000)
        ]
    )

def go2(cursor):
    cursor.setinputsizes(x=cx_Oracle.CLOB)
    cursor.executemany(
        "insert into t (x) values (:x)",
        [
            {"x": "text %s" % i} for i in range(1000)
        ]
    )

import timeit

cursor = setup()
print(timeit.timeit("go1(cursor)", "from __main__ import go1, cursor", number=5))

cursor = setup()
print(timeit.timeit("go2(cursor)", "from __main__ import go2, cursor", number=5))

[anthony-tuininga]

This is expected behaviour. There is no reason to use setinputsizes(CLOB) for inserts. If you do, a temporary CLOB is created and multiple round-trips to the database are required. Depending on your database and client configuration there can be a significant reduction in speed, as you noticed! I have no reason to believe this has changed, but if you see different behaviour with different versions of cx_Oracle, please let me know.

Just for reference, I did this with my own installation and got 0.07 seconds for the first case and 2.3 seconds for the second case.

[zzzeek]

There is no reason to use setinputsizes(CLOB) for inserts.

this is so endlessly confusing. when would I want to use setinputsizes(CLOB)? why is it even there? how are users supposed to know all of these hidden gotchas?

[zzzeek]

additionally, how do I solve the empty string becomes NULL issue without setinputsizes(CLOB) ?

[cjbj]

@anthony-tuininga do we need to tweak samples/ReturnLobsAsStrings.py ?

@zzzeek I just got off the phone from some doc writers who are just getting started on writing some user doc. It may take them a while, but it's moving forward.

Empty string == NULL is an 'Oracle thing', no? :)

[zzzeek]

@cjbj if you use setinputsizes(CLOB) for the INSERT, the empty string in the INSERT successfully round-trips as an empty string coming back. Not sure if perhaps setinputsizes(CHAR) or something would do the same thing.

[zzzeek]

nope...only setinputsizes(CLOB)

[naor2013]

Not setting setinputsizes seem to try to insert any text as a varchar since I get an error if I try to insert strings that has more than 4000 chars. Is there any workaround for that maybe? Can I specify in the query itself that I want to upload the string as a clob?

[anthony-tuininga]

@naor2013, what error are you getting? Modifying @zzzeek's example to insert data greater than 4000 characters works as expected. Also include your Oracle client and database versions.

@cjbj, what did we need to tweak in samples/ReturnLobsAsStrings.py?

@zzzeek, for reasons that have are shrouded in mystery (mostly due to the fact that they were made so long ago!) the value "" and NULL are considered equivalent for strings, but not for CLOBs. There isn't anything that can be done about it at this point.

[anthony-tuininga]

There is no reason to use setinputsizes(CLOB) for inserts.

this is so endlessly confusing. when would I want to use setinputsizes(CLOB)? why is it even there? how are users supposed to know all of these hidden gotchas?

The only reason you would want to use setinputsizes(CLOB) is because you have two PL/SQL procedures with an overload (one that accepts strings and one that accepts CLOBs) and you want to make sure you are calling the one that accepts CLOBs. I wouldn't consider that even remotely a common scenario -- so in general you don't need it. Oracle has a lot of baggage that has been carried along over the years, just because it has been around for so long and been so successful. I am trying to help you navigate those waters and perhaps not doing a good a job as you might like. :-) Feel free to keep asking questions, though! At some point perhaps I'll explain it right...

Note as well that you are writing a generic module for accessing and populating data. Most people just writing applications never run into this sort of thing. Things "just work" and if they don't in some situations they can make an adjustment and move on. You don't have that luxury, unfortunately!

[anthony-tuininga]

FYI, using cx_Oracle 5.3 you need to use cursor.setinputsizes(cx_Oracle.LONG_STRING) or you get the error "ORA-01461: can bind a LONG value only for insert into a LONG column". This is not needed any longer in cx_Oracle 6.x but won't do any harm there either. I also note with interest that 5.3 requires almost 10x longer to process the CLOB version than cx_Oracle 6.x and is almost 50% faster for the non-CLOB case, too. Frankly, I'd drop support for cx_Oracle 5.x to make your life simpler. Almost all of the downloads for cx_Oracle are now for 6.x anyway.

[naor2013]

I'm actually getting this issue you're talking about with the LONG value thing but I'm using cx_oracle 6.3.1! Do you know what might cause this issue?
What does it mean to use setinputsizes as LONG_STRING and how is it any different than setting it as a CLOB?

[anthony-tuininga]

Using type cx_Oracle.CLOB will create a temporary CLOB, which requires storage in the temporary tablespace and performance is poor (as noted). Setting LONG_STRING instead simply passes the string directly through to Oracle. There is a limit (currently) of about 1 GB but that's generally not a concern. :-)

Can you provide a test case that demonstrates the issue you are getting? It may be a configuration difference as I don't experience this problem at all. But using setinputsizes(cx_Oracle.LONG_STRING) doesn't hurt anything either in this case.

[naor2013]

My issue is with sqlalchemy's code so I don't think I can find a specific case on my end that causes this issue but would replacing setinputsizes(CLOB) with setinputsizes(LONG_STRING) in sqlalchemy fix the performance issue? (And the LONG issue too I guess)

[anthony-tuininga]

Unless you plan to insert data larger than 1 GB, I think replacing CLOB with LONG_STRING would solve your problem and get good performance. If your data is larger than 1 GB you'll have other issues, I think! :-)