Refactored DataFrame JDBC API plus DataSource handling

[zaleslaw]

This commit introduces new DataFrame JDBC extension functions with DataSource support, removing redundant duplicate utilities. It includes revisions to streamline table/schema reading and validation features, delegating reusable connection-handling logic to a dedicated utility class. Also refactor file structure for better organization of DB-related code.

Closes #1424 #680 #454 #1266

[Copilot]

Pull Request Overview

This PR refactors the DataFrame JDBC API by adding DataSource support and streamlining the codebase. It extracts common utilities to dedicated files, removes code duplication, and introduces new extension functions for working with DataSource objects.

• Extracted validation utilities and connection management to separate files for better organization
• Added comprehensive DataSource support through new extension functions
• Reorganized metadata classes into the dedicated db package structure

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/validationUtil.kt	New file containing extracted validation utilities and connection management functions
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readJdbc.kt	Major refactoring removing duplicate code and adding DataSource extension functions
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/TableMetadata.kt	New file containing extracted TableMetadata class
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/TableColumnMetadata.kt	New file containing extracted TableColumnMetadata class
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/DbConnectionConfig.kt	New file containing extracted DbConnectionConfig class
Multiple db/*.kt files	Updated import statements to reflect the new package structure

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[koperagen]

Jupyter integration automatically imports org.jetbrains.kotlinx.dataframe.io package
If you want to move, add import org.jetbrains.kotlinx.dataframe.io.db there

[zaleslaw]

is it import only the direct package without subpackages? in that case, yes, it should be

[koperagen]

yes, only direct package

[Jolanrensen]

Can quotes be escaped in SQL? if so, we shouldn't count them when escaped

[Jolanrensen]

also, what about backticks?

[Jolanrensen]

https://learnsql.com/cookbook/how-to-escape-single-quotes-in-sql/ According to here, single quotes are usually escaped by writing them twice,
so INSERT INTO customer (id, customer_name) VALUES (502, 'Lay''s');. In this case your check works.
However, on MySQL and PostgreSQL, you can write INSERT INTO customer (id, customer_name)VALUES (502, 'Lay\'s'); in which case this check will trigger a false negative. There are 3 single quotes but the statement is still valid.

Finally, this is how Oracle does it: INSERT INTO customer (id, customer_name) VALUES (502, q'[Lay's]'); in which case your check will also trigger a false negative.

Similarly, I believe alternating is allowed, so ' hi " there ' is valid, but again a false negative is triggered because there's no even number of double quotes.

I'd remove this check altogether and find a more general solution. There's too many edge cases

[Jolanrensen]

escape [],

[Jolanrensen]

still shows this in the IDE but it renders correctly, so I'm happy :)

[Jolanrensen]

I'm not a fan of the introduction of MutableList in DbType. In many cases, nothing needs to be modified, in which case returning the immutable list is just as efficient, and if values do need to be mapped, they only need to be read once after which they can be garbage collected, so again not much benefit from making it mutable.

Plus, things can easily get confusing when you mix mutable and immutable lists. Just look at:

postProcessColumnValues(values: MutableList<Any?>, ...): List<Any?>

How values will be used is unclear looking at this. Will values inside be overridden? will a new list be returned? will the types change? Very hard to tell.

You could decide to save some memory (which I think will be negligible) by making the list mutable, but then, the function should probably not return anything:

postProcessColumnValues(values: MutableList<Any?>, ...)

However, I'd keep it simple and stick with the immutability of DataFrame:

postProcessColumnValues(values: List<Any?>, ...): List<Any?>

[zaleslaw]

@Jolanrensen regarding Mutablity vs Immutability for internals - the initial idea was to follow immutability style for easy-to-mantain approach, but after couple of request and communication, I've decided to optimize some obvious place to avoid additional copying before building a final DataFrame, also I'm planning to provide comment for each such situation to avoid problems in future, the same should be done for the working with metadata

[Jolanrensen]

try to avoid ::class, it's way heavier than typeOf<String>(). Also, don't you mean subType? instead of superType?