Added clob-to-str to sql.utils

[isaacsu]

This would be useful when dealing with JdbcClob fields.

Example:

(ns shoebox.models.photo
  (:require [korma.sql.utils :as utils]))

(defentity photos
  (transform (fn [{metadata :metadata :as v}]
               (if metadata
                 (assoc v :metadata (utils/clob-to-str metadata)) v))))

[vbauer]

+1

[bitemyapp]

Is this really specific to what is conventionally known as "CLOB" á la DB2 and Oracle or is this reusable with things like PostgreSQL TEXT types?

Also, if it's a CLOB type, then that means it might be seriously huge, is a "all-in-memory-at-once" really a good idea?

I'm also concerned by the absence of a test. Generalizing this (if there is a comparable need for other types) might make it easier to test, but I just don't know a lot about the use-case here.

In general, I'm basically looking for documentation/explanation of the problem being solved.

[Josh-Tilles]

I know that the korma.incubator project hasn't been used very much, but it seems reasonable to put clob-to-str there for now.

Just an option to keep in mind.

[bitemyapp]

@MerelyAPseudonym I don't want to move my food around on my plate, I want to understand the purpose of the commit.

[vbauer]

@bitemyapp I agree, JDBC drivers for PostgreSQL (8.x, 9.x) contains a lot of bugs. For example: I had to make some workarounds in my Java projects to prevent some problems with encoded "bytea" data. So.. It is possible to make universal method, but it will be a little bit complex than this one.

I use clob-to-str method in a tiny project with H2. It helps to read big text descriptions from DB. Unfortunately I can't use VARCHAR type (field value may be a little bit longer), so I thought that clob-to-str could be useful for others. Maybe I'm wrong..

[bitemyapp]

"I use clob-to-str method in a tiny project with H2. It helps to read big text descriptions from DB."

This is the problem I'm struggling with here, why would you ever read self-described "big" text descriptions from the DB into memory universally? That's a way to quickly eat up all your memory and hammer your poor server.

I realize you might have relatively simple use-cases but I don't feel comfortable endorsing the loading in of arbitrarily large data that is intended for streaming in JDBC in a library like Korma, however flawed Korma may already be.

Is there a safer approach/alternative to that would make more sense as a utility-fn in Korma?

[vbauer]

I'll try to explain my use-case:

• It's really simple project for local network (for company usage, something like guestbook).
• Server has 4Gb of RAM. It's enough for this project (I've made some tests).
• The number of users is less than 100.

Ofc, I can leave this functionality only in this project (if it is not useful for others).

[bitemyapp]

@vbauer my problem here is two-fold

1. It's facile (apply str + line-seq + BufferedReader)
2. Doesn't require any domain knowledge or knowledge of Korma to implement (meaning you're not saving anybody any real trouble)
3. It encourages unsafe idioms/habits

There's no leverage or substantial saved effort being offered here beyond enabling somebody to avoid managing their own utility functions.

I'm still open to explanations of why this is valuable but I'm not seeing it at present.

I would be interested in "more convenient" but streaming based interfaces to these data types and enabling those interfaces optionally in the entity spec if somebody has an idea as to what they would look like.

[isaacsu]

Here's how I arrived at suggesting the inclusion of clob-to-str in Korma.

A bit of background: I come from years of programming against the LAMP stack and Ruby on Rails. I am relatively new to Clojure and the Java libraries that underpin in.

While working on a project using the Clojure Luminus framework backed onto h2, I needed a way to store more than 255 characters in a database field. Easily enough, I found the h2 equivalent of a MySQL TEXT field, made a CLOB field in my schema and was on my way.

Writing rows to the database was straightforward, but I encountered difficulties reading out the data from the CLOB field.

I'd expected the result of my query to give me the string that I'd saved, instead I was given a JdbcClob object.

From there, it was a arduous journey looking up the JdbcClob api docs, taking a crash-course in Streams and Readers (completely foreign to me), and finally stumbling upon (clob-to-string) in #200.

I might even add, that if it weren't for encountering the "The object is already closed" problem, I might not have found @vbauer's (clob-to-string) solution.

I share @bitemyapp's concerns about endorsing the loading of arbitrarily large data that is intended for streaming. I wholeheartedly agree that at some point, one should be expected to achieve a basic understanding of JdbcClob Streams and Readers before doing too much damage.

But at the same time, for someone starting out in Clojure, setting up and maintaining one's own utility functions just to read a slightly longer string from the database might be a bit overwhelming.

So instead of:

1. Oh, got a JdbcClob object.
2. Look up JdbcClob API, figure that it's meant to be used as a Stream.
3. Try calling the object's methods to see what comes out.
4. Try a go at collecting them all into a string.
5. Encounter "The object is already closed".
6. Discover @vbauer's (clob-to-string).
7. Copy-and-paste the (clob-to-string) function to some random location in the project.
8. Add (defentity ... (transform ...)).
9. Get on with making stuff.

I hope it could be more like this:

1. Oh, got a JdbcClob object.
2. Google "Korma JdbcClob".
3. Discover example of (defentity ... (transform ...)) with implementation in korma.sql.utils.
4. Heed big flashing warning about loading arbitrarily large data intended for streaming.
5. Get on with making stuff.

Hope this helps.

[isaacsu]

Responding to an earlier question from @bitemyapp:

Is this really specific to what is conventionally known as "CLOB" á la DB2 and Oracle or is this reusable with things like PostgreSQL TEXT types?

I've just discovered that while Korma returns a JdbcClob object when querying a h2 CLOB field, it returns the full string when querying a MySQL TEXT field.

I imagine it'll be a similar thing with PostgreSQL TEXT?

[lokori]

No, it's not similar with PostgreSQL TEXT. It works fine (9.2 db server version, postgresql "9.1-901.jdbc4" driver,org.clojure/java.jdbc "0.3.0-alpha5" and korma "0.3.0-RC6") in our setup without any tweaking. Which is very nice.

And I think the default should be as described in this issue. Yes, there might be huge clobs and huge resultsets but I think it is an exception that one does not want to read the data as described here.

If the use case is putting gigabytes of data into a relational database text field and streaming it, I think the relational database is not the right tool to use for persistent storage. It might happen occasionally but it's not the default case in my opinion. If it happens, the programmer will have so much headache anyway that Korma's stance on the matter is a minor issue.

However, Korma's preference is a major issue when you run against varchar2 limits. With Oracle this doesn't take very much (4001 characters is too much for Oracle). There are many reasonable applications where you need more than 4000 but will not be storing gigabytes. It's not a "string" because Oracle is Oracle, but it is very seldom a "large binary object" in reality.

[isaacsu]

@lokori What I meant to say is that PostgreSQL TEXT behaves similar to MySQL TEXT in that the whole string is returned.

[lokori]

Ah, I misunderstood. Nevertheless we seem to agree that treating clobs as strings should be the default behaviour from the application programmer's point of view :)