JSON datatype support #2558

divang · 2024-12-06T13:24:44Z

To add support for the new JSON datatype, the following changes have been made to the codebase:

Extended TDS feature support
Added JDBC JSON data type support in DataTypes
Implemented JSON strategy in DTV
Included JSON character encoding and constants
Handled JSON datatype support in ResultSet, TVP, and IOBuffer
Supported JSON as IN/OUT parameters in SQL prepared and callable statements
Managed JSON datatype in SQLServerBulkCopy and related bulk copy scenarios

codecov · 2024-12-06T13:47:40Z

Codecov Report

Attention: Patch coverage is 82.22222% with 8 lines in your changes missing coverage. Please review.

Project coverage is 50.09%. Comparing base (5bb3353) to head (508a1b9).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
.../microsoft/sqlserver/jdbc/SQLServerConnection.java	75.00%	3 Missing and 2 partials ⚠️
...oft/sqlserver/jdbc/SQLServerParameterMetaData.java	0.00%	1 Missing and 1 partial ⚠️
...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2558      +/-   ##
============================================
- Coverage     51.01%   50.09%   -0.93%     
+ Complexity     3921     3829      -92     
============================================
  Files           147      147              
  Lines         33483    33544      +61     
  Branches       5609     5619      +10     
============================================
- Hits          17081    16803     -278     
- Misses        13991    14347     +356     
+ Partials       2411     2394      -17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…rios.

src/test/java/com/microsoft/sqlserver/jdbc/unit/statement/RegressionTest.java

lilgreenbird · 2024-12-19T23:11:21Z

/azp run public-mssql-jdbc.windows

azure-pipelines · 2024-12-19T23:11:32Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2024-12-19T23:18:15Z

No pipelines are associated with this pull request.

lilgreenbird · 2024-12-19T23:18:56Z

/azp run public-mssql-jdbc.linux

lilgreenbird · 2024-12-19T23:19:05Z

/azp run CI-MacOS

azure-pipelines · 2024-12-19T23:19:05Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2024-12-19T23:19:15Z

Azure Pipelines successfully started running 1 pipeline(s).

src/test/java/com/microsoft/sqlserver/jdbc/callablestatement/CallableStatementTest.java

…m/microsoft/mssql-jdbc into user/divang/json-datatype-support

…ype conversion

…ues.

src/main/java/microsoft/sql/Types.java

Copilot

Copilot reviewed 11 out of 26 changed files in this pull request and generated no comments.

Files not reviewed (15)

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerResource.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/Parameter.java: Evaluated as low risk
src/test/java/com/microsoft/sqlserver/jdbc/bulkCopy/BulkCopyCSVTest.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/SQLServerDataTable.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/Column.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/IOBuffer.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/SQLServerResultSetMetaData.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java: Evaluated as low risk
src/test/java/com/microsoft/sqlserver/jdbc/bulkCopy/SqlTypeMapping.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/SQLServerConnection.java: Evaluated as low risk
src/main/java/microsoft/sql/Types.java: Evaluated as low risk
src/main/java/com/microsoft/sqlserver/jdbc/dtv.java: Evaluated as low risk
src/test/java/com/microsoft/sqlserver/jdbc/callablestatement/CallableStatementTest.java: Evaluated as low risk
src/test/java/com/microsoft/sqlserver/jdbc/tvp/TVPTypesTest.java: Evaluated as low risk

…uteUpdate

…using the column label instead of the index.

src/test/java/com/microsoft/sqlserver/jdbc/JSONFunctionTest.java

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCSVFileRecord.java

…otes, preventing misclassification of non-JSON braces, and adding test cases based on review comments.

…mnNameWithJson.csv.

…al and local temporary tables, UDF with SELECT, WHERE, and FROM clauses, and SELECT INTO queries.

…ata check, direct function call validation, and assertion for JSON structure.

…BulkCopy=true in connection string

…here

lilgreenbird · 2025-03-27T21:52:37Z

src/test/java/com/microsoft/sqlserver/testframework/Constants.java

@@ -29,6 +29,7 @@ private Constants() {}
     * reqExternalSetup  - For tests requiring external setup
     * clientCertAuth  - - For tests requiring client certificate authentication setup
     * Fedauth - - - - - - For Fedauth tests
+     * JSONTest -  - - - - For tests requiring JSON setup


why do we need to add a separate tag for this? this should always run along with the other datatype tests

@lilgreenbird We would need to exclude these tests against test endpoints that don't support JSON. (I'm kind of surprised we don't have one for Data Classification, for example. But it looks like we've used SQL Server version for that, which isn't quite proper.)

we need a new tag for new SQL Server version that supports JSON. (SQL Server 2023 or ?)

yeah, why not? data classification was in 2019 only

lilgreenbird · 2025-03-27T21:54:00Z

src/test/java/com/microsoft/sqlserver/testframework/AbstractTest.java

@@ -158,8 +158,7 @@ public static void setup() throws Exception {
            // no config file used
        }

-        connectionString = getConfiguredPropertyOrEnv(Constants.MSSQL_JDBC_TEST_CONNECTION_PROPERTIES);
-
+        connectionString = getConfiguredPropertyOrEnv(Constants.MSSQL_JDBC_TEST_CONNECTION_PROPERTIES); 


why remove the blank line?

lilgreenbird

the tests need to take into consideration the SQL server version. JSON support is only avail on SQL 2023 (or ?) so this means we need to exclude this for xSQLv16 and below. Need new tag for this new SQL Server version

lilgreenbird · 2025-03-28T00:52:35Z

src/main/java/com/microsoft/sqlserver/jdbc/dtv.java

@@ -1459,6 +1461,7 @@ final void executeOp(DTVExecuteOp op) throws SQLServerException {
                case NVARCHAR:
                case LONGNVARCHAR:
                case NCLOB:
+                case JSON:


I think we're missing a case in the else (line #1599). This makes me think we're also missing a test case that will go to that path. Check SQL_VARIANT to see where this is tested.

I think we also need a case for JSON in readSqlVariant, and, a test for that

lilgreenbird and others added 2 commits November 28, 2024 13:16

JSON feature extension

17e21c1

Added JSON data type support in DataTypes

b480490

divang marked this pull request as draft December 10, 2024 20:23

Divang Sharma added 5 commits December 11, 2024 02:05

JSON startegy in DTV

1f2b95b

Fix Json character encoding and JSON constants.

e102633

Added JSON datatype test case

d994fec

Added JSON datatype test cases in ResultSet, TVP and Regression scena…

c164ef9

…rios.

Added JSON datatype support

f9db4f7

machavan reviewed Dec 19, 2024

View reviewed changes

src/test/java/com/microsoft/sqlserver/jdbc/unit/statement/RegressionTest.java Show resolved Hide resolved

Added JSON support in IOBuffer.java

dff9280

Ananya2 added 2 commits January 3, 2025 11:27

Json datatype support: SQLServerCallableStatement

0e7a053

Updated callableStatementTest.java

a8ebfd3

divang commented Jan 3, 2025

View reviewed changes

src/test/java/com/microsoft/sqlserver/jdbc/callablestatement/CallableStatementTest.java Outdated Show resolved Hide resolved

Divang Sharma and others added 8 commits January 9, 2025 18:31

Merge branch 'main' into user/divang/json-datatype-support

952e37e

Added test cases for the TVP type in Callable Statements.

44a185b

Fixed JSON op.execute method mapping

dc53208

Merge branch 'user/divang/json-datatype-support' of https://github.co…

508a1b9

…m/microsoft/mssql-jdbc into user/divang/json-datatype-support

Fixed Store procedure output param TDS metadata and fixed bulk copy t…

fd21d1a

…ype conversion

Fixed BulkCopy JSON datatype issue.

07bea45

Removed duplicate code and used existing method to solve Bulk opy iss…

afb3753

…ues.

Added test cases for Bulk copy

9c4a401

lilgreenbird reviewed Feb 12, 2025

View reviewed changes

src/main/java/microsoft/sql/Types.java Show resolved Hide resolved

Jeffery-Wasty requested review from machavan and Copilot February 18, 2025 21:55

Copilot AI reviewed Feb 18, 2025

View reviewed changes

Divang Sharma and others added 6 commits February 20, 2025 10:11

Fixed JSON datatype setObject issue

b67553e

Updated regression test case for JSON datatype: mix of addBatch, exec…

22b230f

…uteUpdate

Added more metadata validation check for JSON datatype

04cadb8

Updated as per the review comment to validate the getString() result …

0b6b392

…using the column label instead of the index.

Updated and added more test scenarios for Bulk Copy

c69dfb1

Added JSONFunctionTest to test JSON functions

81a19ed

muskan124947 reviewed Feb 20, 2025

View reviewed changes

src/test/java/com/microsoft/sqlserver/jdbc/JSONFunctionTest.java Outdated Show resolved Hide resolved

David-Engel reviewed Feb 20, 2025

View reviewed changes

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCSVFileRecord.java Outdated Show resolved Hide resolved

Ananya2 and others added 14 commits February 21, 2025 10:44

Optimized string parsing to avoid unnecessary allocations.

3d07480

formatted as per the older version

e2e4748

Improve CSV parsing by correctly detecting JSON fields with double qu…

8384c9d

…otes, preventing misclassification of non-JSON braces, and adding test cases based on review comments.

Remove reference to non-existent test file BulkCopyCSVTestInputNoColu…

ccb6256

…mnNameWithJson.csv.

Updated JSONFunction Test class

31bbf91

Adding tag JSONTest for some test cases

630c99e

Added and validated test cases for JSON support in JOIN queries, glob…

30aec56

…al and local temporary tables, UDF with SELECT, WHERE, and FROM clauses, and SELECT INTO queries.

Added test case to validate UDF returning JSON output; included metad…

56233bf

…ata check, direct function call validation, and assertion for JSON structure.

Added tests for OPENJSON()

4c4f064

Added tests for OPENJSON() function

6412517

Added tests to handle large JSON file upto 2GB

530bd94

Add test cases to perform operations using Prepared Statement and use…

6cc9392

…BulkCopy=true in connection string

Updated test to insert large json data and verify no packet loss is t…

bef793e

…here

Added JSONTest tag to all the test cases

60b602e

lilgreenbird reviewed Mar 27, 2025

View reviewed changes

lilgreenbird suggested changes Mar 27, 2025

View reviewed changes

github-project-automation bot moved this from Under Peer Review to In progress in MSSQL JDBC Mar 27, 2025

lilgreenbird reviewed Mar 28, 2025

View reviewed changes

JSON datatype support #2558

Are you sure you want to change the base?

JSON datatype support #2558

Uh oh!

Conversation

divang commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

lilgreenbird commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

lilgreenbird commented Dec 19, 2024

Uh oh!

lilgreenbird commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lilgreenbird Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

David-Engel Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

lilgreenbird Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

lilgreenbird Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

lilgreenbird Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

lilgreenbird left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lilgreenbird Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

lilgreenbird Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

divang commented Dec 6, 2024 •

edited

Loading

codecov bot commented Dec 6, 2024 •

edited

Loading

lilgreenbird left a comment •

edited

Loading