Support type mapping in Dataframe queries (#494).

Author: anthony-tuininga

doc/src/api_manual/async_connection.rst

@@ -105,7 +105,7 @@ AsyncConnection Methods
 
     .. versionchanged:: 3.4.0
 
-        The ``fetch_decimals`` parameter was added.
+        The ``fetch_decimals`` and ``requested_schema`` parameters were added.
 
     .. versionadded:: 3.0.0
 
@@ -115,7 +115,7 @@ AsyncConnection Methods
 
     .. versionchanged:: 3.4.0
 
-        The ``fetch_decimals`` parameter was added.
+        The ``fetch_decimals`` and ``requested_schema`` parameters were added.
 
     .. versionadded:: 3.0.0
 

doc/src/api_manual/connection.rst

@@ -95,7 +95,7 @@ Connection Methods
 
     .. versionchanged:: 3.4.0
 
-        The ``fetch_decimals`` parameter was added.
+        The ``fetch_decimals`` and ``requested_schema`` parameters were added.
 
     .. versionadded:: 3.0.0
 
@@ -107,7 +107,7 @@ Connection Methods
 
     .. versionchanged:: 3.4.0
 
-        The ``fetch_decimals`` parameter was added.
+        The ``fetch_decimals`` and ``requested_schema`` parameters were added.
 
     .. versionadded:: 3.0.0
 

doc/src/release_notes.rst

@@ -21,7 +21,8 @@ Thin Mode Changes
 
 #)  Added support for Oracle Database's :ref:`Direct Path Load
     <directpathloads>` functionality which is very efficient for loading large
-    datasets into a database.
+    datasets into a database. Data may be a list of sequences or a DataFrame
+    object.
 #)  Fixed bug when setting values of type ``datetime.date`` on variables (such
     as created by :meth:`Cursor.var()` or implicitly by
     :meth:`Cursor.setinputsizes()`) of types
@@ -44,16 +45,26 @@ Thick Mode Changes
 Common Changes
 ++++++++++++++
 
-#)  Added support for all of the signed and unsigned fixed width integer types
-    when ingesting data frames supporting the Arrow PyCapsule interface.
-    Previously only ``int64`` was supported.
-#)  Added support for types ``date32`` and ``date64`` when ingesting data
-    frames supporting the Arrow PyCapsule interface as requested
-    (`issue 535 <https://github.com/oracle/python-oracledb/issues/535>`__).
+#)  Changes to :ref:`data frame <dataframeformat>` support:
+
+    - Support for data frames is no longer considered a pre-release.
+    - Added parameter ``requested_schema`` to :meth:`Connection.fetch_df_all()`
+      and :meth:`Connection.fetch_df_batches()` to support type mapping when
+      querying.
+    - Added support for all of the signed and unsigned fixed width integer
+      types when ingesting data frames supporting the Arrow PyCapsule
+      interface. Previously only ``int64`` was supported.
+    - Added support for types ``date32`` and ``date64`` when ingesting data
+      frames supporting the Arrow PyCapsule interface as requested
+      (`issue 535 <https://github.com/oracle/python-oracledb/issues/535>`__).
+    - Data frames with multiple chunks are now supported.
+    - Fixed bug when fetching NCHAR and NVARCHAR2 column data.
+    - Fixed bug when attempting to convert an integer that cannot be
+      represented as a native C ``int`` value to an Arrow data frame.
+
 #)  Added a ``batch_size`` parameter to :meth:`Cursor.executemany()` and
     :meth:`AsyncCursor.executemany()` to let these methods operate on data in
     batches.
-#)  Data frames with multiple chunks are now supported.
 #)  Added ``fetch_lobs`` and ``fetch_decimals`` parameters where applicable to
     the methods used for fetching rows or data frames from the database. Note
     that for the creation of pipeline operations, if these parameters are not
@@ -81,12 +92,8 @@ Common Changes
     DocumentDisplay?id=742060.1>`__.
 #)  Pin Cython to 3.1.x instead of 3.1.0 as requested
     (`issue 530 <https://github.com/oracle/python-oracledb/issues/530>`__).
-#)  Support for :ref:`data frames <dataframeformat>` is no longer considered a
-    pre-release.
 #)  Fixed bug when attempting to execute an empty statement
     (`issue 525 <https://github.com/oracle/python-oracledb/issues/525>`__).
-#)  Fixed bug when attempting to convert an integer that cannot be represented
-    as a native C ``int`` value to an Arrow data frame.
 #)  Fixed bug when attempting to append an element to a
     :ref:`DbObject <dbobjecttype>` which is not actually a collection.
 #)  API documentation is now generated from the source code.

doc/src/user_guide/batch_statement.rst

@@ -629,7 +629,7 @@ You can control the data transfer by changing your SELECT statement.
 Direct Path Loads
 =================
 
-Direct Path Loads allows data being inserted into Oracle Database to bypass
+Direct Path Loads allow data being inserted into Oracle Database to bypass
 code layers such as the database buffer cache. Also there are no INSERT
 statements used. This can be very efficient for ingestion of huge amounts of
 data but, as a consequence of the architecture, there are restrictions on when

doc/src/user_guide/dataframes.rst

@@ -110,14 +110,18 @@ Or to iterate:
 Data Frame Type Mapping
 -----------------------
 
+Default Data Frame Type Mapping
++++++++++++++++++++++++++++++++
+
 Internally, python-oracledb's :ref:`DataFrame <oracledataframeobj>` support
 makes use of `Apache nanoarrow <https://arrow.apache.org/nanoarrow/>`__
 libraries to build data frames.
 
-The following data type mapping occurs from Oracle Database types to the Arrow
-types used in python-oracledb DataFrame objects. Querying any other data types
-from Oracle Database will result in an exception. :ref:`Output type handlers
-<outputtypehandlers>` cannot be used to map data types.
+When querying, the following default data type mapping occurs from Oracle
+Database types to the Arrow types used in python-oracledb DataFrame
+objects. Querying any other data types from Oracle Database will result in an
+exception. :ref:`Output type handlers <outputtypehandlers>` cannot be used to
+map data types.
 
 .. list-table-with-summary:: Mapping from Oracle Database to Arrow data types
     :header-rows: 1
@@ -258,6 +262,99 @@ When converting Oracle Database DATEs and TIMESTAMPs:
       * - 7 - 9
         - nanoseconds
 
+Explicit Data Frame Type Mapping
+++++++++++++++++++++++++++++++++
+
+You can explicitly set the data types and names that a :ref:`DataFrame
+<oracledataframeobj>` will use for query results. This provides fine-grained
+control over the physical data representation of the resulting Arrow arrays. It
+allows you to specify a representation that is more efficient for its specific
+use case. This can reduce memory consumption and improve processing speed.
+
+The parameter ``requested_schema`` parameter to
+:meth:`Connection.fetch_df_all()`, :meth:`Connection.fetch_df_batches()`,
+:meth:`AsyncConnection.fetch_df_all()`, or
+:meth:`AsyncConnection.fetch_df_batches()` should be an object implementing the
+`Arrow PyCapsule schema interface
+<https://arrow.apache.org/docs/python/generated/pyarrow.Schema.html>`__.
+
+For example, the ``pyarrow.schema()`` factory function can be used to create a
+new schema. This takes a list of field definitions as input. Each field can be
+a tuple of ``(name, DataType)``:
+
+.. code-block:: python
+
+    import pyarrow
+
+    # Default fetch
+
+    odf = connection.fetch_df_all(
+        "select 123 c1, 'Scott' c2 from dual"
+    )
+    tab = pyarrow.table(odf)
+    print("Default Output:", tab)
+
+    # Fetching with an explicit schema
+
+    schema = pyarrow.schema([
+        ("col_1", pyarrow.int16()),
+        ("C2", pyarrow.string())
+    ])
+
+    odf = connection.fetch_df_all(
+        "select 456 c1, 'King' c2 from dual",
+        requested_schema=schema
+    )
+    tab = pyarrow.table(odf)
+    print("\nNew Output:", tab)
+
+The schema should have an entry for each queried column.
+
+Running the example shows that the number column with the explicit schema was
+fetched into the requested type INT16. Its name has also changed::
+
+    Default Output: pyarrow.Table
+    C1: double
+    C2: string
+    ----
+    C1: [[123]]
+    C2: [["Scott"]]
+
+    New Output: pyarrow.Table
+    col_1: int16
+    C2: string
+    ----
+    col_1: [[456]]
+    C2: [["King"]]
+
+**Supported Explicit Type Mapping**
+
+The following table shows the explicit type mappings that are supported. An
+error will occur if the database type or the data cannot be represented in the
+requested schema type.
+
+  .. list-table-with-summary::
+      :header-rows: 1
+      :class: wy-table-responsive
+      :widths: 1 1
+      :align: left
+      :summary: The first column is the Oracle Database data type. The second column shows supported Arrow data types.
+
+      * - Oracle Database Type
+        - Arrow Data Types
+      * - DB_TYPE_NUMBER
+        - INT8, INT16, INT32, INT64, UINT8, UINT16, UINT32, UINT64, DECIMAL128(p, s), DOUBLE, FLOAT
+      * - DB_TYPE_RAW, DB_TYPE_LONG_RAW
+        - BINARY, FIXED SIZE BINARY, LARGE BINARY
+      * - DB_TYPE_BOOLEAN
+        - BOOLEAN
+      * - DB_TYPE_DATE, DB_TYPE_TIMESTAMP, DB_TYPE_TIMESTAMP_LTZ, DB_TYPE_TIMESTAMP_TZ
+        - DATE32, DATE64, TIMESTAMP
+      * - DB_TYPE_BINARY_DOUBLE, DB_TYPE_BINARY_FLOAT
+        - DOUBLE, FLOAT
+      * - DB_TYPE_VARCHAR, DB_TYPE_CHAR, DB_TYPE_LONG, DB_TYPE_NVARCHAR, DB_TYPE_NCHAR, DB_TYPE_LONG_NVARCHAR
+        - STRING, LARGE_STRING
+
 .. _convertingodf:
 
 Converting python-oracledb's DataFrame to Other Data Frames

samples/dataframe_types.py

@@ -0,0 +1,63 @@
+# -----------------------------------------------------------------------------
+# Copyright (c) 2025, Oracle and/or its affiliates.
+#
+# This software is dual-licensed to you under the Universal Permissive License
+# (UPL) 1.0 as shown at https://oss.oracle.com/licenses/upl and Apache License
+# 2.0 as shown at http://www.apache.org/licenses/LICENSE-2.0. You may choose
+# either license.
+#
+# If you elect to accept the software under the Apache License, Version 2.0,
+# the following applies:
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -----------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------------
+# dataframe_types.py
+#
+# Shows how to change the schema types and names of a dataframe
+# -----------------------------------------------------------------------------
+
+import pyarrow
+
+import oracledb
+import sample_env
+
+# determine whether to use python-oracledb thin mode or thick mode
+if sample_env.run_in_thick_mode():
+    oracledb.init_oracle_client(lib_dir=sample_env.get_oracle_client())
+
+connection = oracledb.connect(
+    user=sample_env.get_main_user(),
+    password=sample_env.get_main_password(),
+    dsn=sample_env.get_connect_string(),
+    params=sample_env.get_connect_params(),
+)
+
+
+SQL = "select * from SampleQueryTab where id < 5"
+
+# Default fetch with no type mapping
+
+odf = connection.fetch_df_all(SQL)
+tab = pyarrow.table(odf)
+print("Default Output:", tab)
+
+# Fetching with an explicit schema
+
+schema = pyarrow.schema(
+    [("COL_1", pyarrow.int16()), ("COL_2", pyarrow.string())]
+)
+odf = connection.fetch_df_all(SQL, requested_schema=schema)
+tab = pyarrow.table(odf)
+print("\nNew Output:", tab)

src/oracledb/arrow_impl.pxd

@@ -105,6 +105,7 @@ cdef class ArrowSchemaImpl:
         ArrowSchema *arrow_schema
         ArrowType child_arrow_type
         int child_element_size
+        list child_schemas
 
     cdef bint _is_sparse_vector(self) except*
     cdef int _set_child_arrow_type(self, ArrowType child_arrow_type) except -1
@@ -122,19 +123,24 @@ cdef class ArrowArrayImpl:
         ArrowArray *arrow_array
         ArrowSchemaImpl schema_impl
 
+    cdef int _extract_int(self, const void* ptr, ArrowType arrow_type,
+                          int64_t index, int64_t* value) except -1
+    cdef int _extract_uint(self, const void* ptr, ArrowType arrow_type,
+                           int64_t index, uint64_t* value) except -1
     cdef int _get_is_null(self, int64_t index, bint* is_null) except -1
     cdef int _get_list_info(self, int64_t index, ArrowArray* arrow_array,
                             int64_t* offset, int64_t* num_elements) except -1
     cdef int append_bytes(self, void* ptr, int64_t num_bytes) except -1
     cdef int append_decimal(self, void* ptr, int64_t num_bytes) except -1
     cdef int append_double(self, double value) except -1
     cdef int append_float(self, float value) except -1
-    cdef int append_int64(self, int64_t value) except -1
+    cdef int append_int(self, int64_t value) except -1
     cdef int append_last_value(self, ArrowArrayImpl array) except -1
     cdef int append_null(self) except -1
     cdef int append_sparse_vector(self, int64_t num_dimensions,
                                   array.array indices,
                                   array.array values) except -1
+    cdef int append_uint(self, uint64_t value) except -1
     cdef int append_vector(self, array.array value) except -1
     cdef int finish_building(self) except -1
     cdef int get_bool(self, int64_t index, bint* is_null,

src/oracledb/arrow_impl.pyx

@@ -32,6 +32,7 @@
 
 cimport cpython
 
+from libc.errno cimport EINVAL
 from libc.stdint cimport uintptr_t
 from libc.string cimport memcpy, memset, strlen, strchr
 from cpython cimport array

src/oracledb/base_impl.pxd

@@ -337,8 +337,9 @@ cdef class Buffer:
     cdef int read_sb4(self, int32_t *value) except -1
     cdef int read_sb8(self, int64_t *value) except -1
     cdef bytes read_null_terminated_bytes(self)
-    cdef int read_oracle_data(self, OracleMetadata metadata,
-                              OracleData* data, bint from_dbobject) except -1
+    cdef object read_oracle_data(self, OracleMetadata metadata,
+                                 OracleData* data, bint from_dbobject,
+                                 bint decode_str)
     cdef object read_str(self, int csfrm, const char* encoding_errors=*)
     cdef object read_str_with_length(self)
     cdef int read_ub1(self, uint8_t *value) except -1
@@ -495,6 +496,10 @@ cdef class OracleMetadata:
     cdef int _create_arrow_schema(self) except -1
     cdef int _finalize_init(self) except -1
     cdef int _set_arrow_schema(self, ArrowSchemaImpl schema_impl) except -1
+    cdef int check_convert_from_arrow(self,
+                                      ArrowSchemaImpl schema_impl) except -1
+    cdef int check_convert_to_arrow(self,
+                                    ArrowSchemaImpl schema_impl) except -1
     cdef OracleMetadata copy(self)
     @staticmethod
     cdef OracleMetadata from_arrow_schema(ArrowSchemaImpl schema_impl)
@@ -718,6 +723,7 @@ cdef class BaseCursorImpl:
         public bint suspend_on_success
         public bint fetch_lobs
         public bint fetch_decimals
+        public ArrowSchemaImpl schema_impl
         uint32_t _buffer_rowcount
         uint32_t _buffer_index
         uint32_t _fetch_array_size

src/oracledb/base_impl.pyx

@@ -39,7 +39,7 @@ from libc cimport errno
 from libc.stdint cimport int8_t, int16_t, int32_t, int64_t
 from libc.stdint cimport uint8_t, uint16_t, uint32_t, uint64_t
 from libc.stdint cimport UINT8_MAX, UINT16_MAX, UINT32_MAX, UINT64_MAX
-from libc.stdlib cimport strtod, strtoll
+from libc.stdlib cimport strtod, strtof, strtoll, strtoull
 from libc.string cimport memcpy
 from cpython cimport array
 from cpython.conversion cimport PyOS_snprintf