Slowness in MS SQL Server JDBC driver

[paragpandit123]

JDBC VERSION: mssql-jdbc-12.6.1.jre11.jar
On Prem SQL Server Version : Query is getting data from databases on SQL 2017 and SQL 2019
How big is the query? I mean how many records it fetches and tables it query from SQL: In the sample we’ve used the return set is 71 million rows

[paragpandit123]

We used to complete our data load of around 200 tables under 2 hrs with other driver and when we moved to SQL JDBC drivers , the load is not completing even after 7 hrs. the sample we mentioned where there are 71 M records , it use to complete under 30 mins and not it takes around 6 -7 hrs Thanks Parag Pandit Application Architect ***@***.******@***.***> 732.789.7326 [Logo Description automatically generated]<http://www.lumeris.com/> [Icon Description automatically generated] <https://www.linkedin.com/company/lumeris> [Logo Description automatically generated] <https://twitter.com/lumeris>

…

________________________________ From: Jeff Wasty ***@***.***> Sent: Friday, May 24, 2024 12:14 PM To: microsoft/mssql-jdbc ***@***.***> Cc: Parag Pandit ***@***.***>; Mention ***@***.***> Subject: [External] Re: [microsoft/mssql-jdbc] Slowness in MS SQL Server JDBC driver (Issue #2432) EXTERNAL Client / Partner / Business Vendor This email originated from a Lumeris affiliated entity. Please continue to exercise caution when opening links or attachments. Note: If there is a security concern, please contact the Service Desk immediately<https://lumeris.service-now.com/sp?id=sc_cat_item&sys_id=3f1dd0320a0a0b99000a53f7604a2ef9&sysparm_category=e15706fc0a0a0aa7007fc21e1ab70c2f>. Hi @paragpandit123<https://urldefense.com/v3/__https://github.com/paragpandit123__;!!IKIcsDr4!WsNtobhynlN7qZNvhllNrYLin7YsBqw13ORyIeRtItl4pfyIkMOiMpbuikV1QPgOLlpk2NsMR-Nkpd1dZTwYcEs$>, The issue is that there is "slowness" in the driver, how slow? How long does it take to complete the above mentioned query? How much longer is this than what you expect? When you say "slow", what is this in comparison to? Different queries? Or these same queries but tried with another version of the MSSQL-JDBC driver? Or are you comparing to another JDBC driver? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microsoft/mssql-jdbc/issues/2432*issuecomment-2129920890__;Iw!!IKIcsDr4!WsNtobhynlN7qZNvhllNrYLin7YsBqw13ORyIeRtItl4pfyIkMOiMpbuikV1QPgOLlpk2NsMR-Nkpd1dYQdM5YI$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A7SAXHWPA75VFMTWGEMAICTZD5RQFAVCNFSM6AAAAABIHXCWCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRZHEZDAOBZGA__;!!IKIcsDr4!WsNtobhynlN7qZNvhllNrYLin7YsBqw13ORyIeRtItl4pfyIkMOiMpbuikV1QPgOLlpk2NsMR-Nkpd1dwS0Hc64$>. You are receiving this because you were mentioned.Message ID: ***@***.***> This e-mail and any files accompanying this e-mail contain confidential information belonging to the sender. This information may contain protected health information and other material that is legally protected, privileged or proprietary. The information is intended only for the use of the individual or entity named above. The intended recipient of this e-mail is prohibited from disclosing this information to any other party unless required to do so by law or regulation and is required to delete or destroy the information after its stated need has been fulfilled. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on or regarding the contents of this electronically transmitted information is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this message.

[Jeffery-Wasty]

Hi,

Yes, sorry, I deleted my response as I realized this was a follow-up to a previous email issue that was sent to the team. Thank you for the additional information, we'll look into this and get back to you with our response.

[paragpandit123]

FYI, this is a production issue and needs a quick resolution. Parag Pandit Application Architect ***@***.******@***.***> 732.789.7326 [Logo Description automatically generated]<http://www.lumeris.com/> [Icon Description automatically generated] <https://www.linkedin.com/company/lumeris> [Logo Description automatically generated] <https://twitter.com/lumeris>

…

________________________________ From: Jeff Wasty ***@***.***> Sent: Friday, May 24, 2024 12:21 PM To: microsoft/mssql-jdbc ***@***.***> Cc: Parag Pandit ***@***.***>; Mention ***@***.***> Subject: [External] Re: [microsoft/mssql-jdbc] Slowness in MS SQL Server JDBC driver (Issue #2432) EXTERNAL Client / Partner / Business Vendor This email originated from a Lumeris affiliated entity. Please continue to exercise caution when opening links or attachments. Note: If there is a security concern, please contact the Service Desk immediately<https://lumeris.service-now.com/sp?id=sc_cat_item&sys_id=3f1dd0320a0a0b99000a53f7604a2ef9&sysparm_category=e15706fc0a0a0aa7007fc21e1ab70c2f>. Hi, Yes, sorry, I deleted my response as I realized this was a follow-up to a previous email issue that was sent to the team. Thank you for the additional information, we'll look into this and get back to you with our response. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microsoft/mssql-jdbc/issues/2432*issuecomment-2129935362__;Iw!!IKIcsDr4!WA83AhyiPMhS8t_uAdgyKHsQ0BjIRQRtH43l0yR4WhSyU6Dud2A4Y098AlRzH_InKBiX5TjdGQwoBIntKv7yuUA$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A7SAXHVZFBNH6VS5U63UOL3ZD5SJDAVCNFSM6AAAAABIHXCWCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRZHEZTKMZWGI__;!!IKIcsDr4!WA83AhyiPMhS8t_uAdgyKHsQ0BjIRQRtH43l0yR4WhSyU6Dud2A4Y098AlRzH_InKBiX5TjdGQwoBIntFosUmnw$>. You are receiving this because you were mentioned.Message ID: ***@***.***> This e-mail and any files accompanying this e-mail contain confidential information belonging to the sender. This information may contain protected health information and other material that is legally protected, privileged or proprietary. The information is intended only for the use of the individual or entity named above. The intended recipient of this e-mail is prohibited from disclosing this information to any other party unless required to do so by law or regulation and is required to delete or destroy the information after its stated need has been fulfilled. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on or regarding the contents of this electronically transmitted information is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this message.

[Jeffery-Wasty]

I read the email thread again. You are using the same connection string options and the same machines and servers in both cases, correct? You mention you are already using prepareMethod=prepare, there were 2 more questions:

• What are the encryption settings you are using (it would be best if you could share the connection string)? As mentioned in the email, even with identical settings JDBC and jTDS can have different performance based on how they handle encryption.
• Is Unicode data being used? If not, then try setting sendStringParametersAsUnicode=false

Also please enable logging (https://learn.microsoft.com/en-us/sql/connect/jdbc/tracing-driver-operation?view=sql-server-ver16) and allow logging to run for a set amount of time while waiting on the query. The timestamps will allow us to find out exactly what part of the driver is causing the long times.

[paragpandit123]

I have attached an image with connection details. I guess we need a call / working session. Which will avoid back and forth . How about sometime Monday , I am on east coast Thanks Parag Pandit Application Architect ***@***.******@***.***> 732.789.7326 [Logo Description automatically generated]<http://www.lumeris.com/> [Icon Description automatically generated] <https://www.linkedin.com/company/lumeris> [Logo Description automatically generated] <https://twitter.com/lumeris> [cid:64ca1470-9719-415e-a462-b11450997b2f]

…

________________________________ From: Jeff Wasty ***@***.***> Sent: Friday, May 24, 2024 3:28 PM To: microsoft/mssql-jdbc ***@***.***> Cc: Parag Pandit ***@***.***>; Mention ***@***.***> Subject: [External] Re: [microsoft/mssql-jdbc] Slowness in MS SQL Server JDBC driver (Issue #2432) EXTERNAL Client / Partner / Business Vendor This email originated from a Lumeris affiliated entity. Please continue to exercise caution when opening links or attachments. Note: If there is a security concern, please contact the Service Desk immediately<https://lumeris.service-now.com/sp?id=sc_cat_item&sys_id=3f1dd0320a0a0b99000a53f7604a2ef9&sysparm_category=e15706fc0a0a0aa7007fc21e1ab70c2f>. I read the email thread again. You are using the same connection string options and the same machines and servers in both cases, correct? You mention you are already using prepareMethod=prepare, there were 2 more questions: * What are the encryption settings you are using (it would be best if you could share the connection string)? As mentioned in the email, even with identical settings JDBC and jTDS can have different performance based on how they handle encryption. * Is Unicode data being used? If not, then try setting sendStringParametersAsUnicode=false Also please enable logging (https://learn.microsoft.com/en-us/sql/connect/jdbc/tracing-driver-operation?view=sql-server-ver16<https://urldefense.com/v3/__https://learn.microsoft.com/en-us/sql/connect/jdbc/tracing-driver-operation?view=sql-server-ver16__;!!IKIcsDr4!X1PcvwDW6vnR1ruL6JWmXScTfDYpTzGQyZJfNqsEwbxrLYUcfE_Qsak1aQSdsEM2BwJCREqdojMmknRLxaLr5XI$>) and allow logging to run for a set amount of time while waiting on the query. The timestamps will allow us to find out exactly what part of the driver is causing the long times. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microsoft/mssql-jdbc/issues/2432*issuecomment-2130230756__;Iw!!IKIcsDr4!X1PcvwDW6vnR1ruL6JWmXScTfDYpTzGQyZJfNqsEwbxrLYUcfE_Qsak1aQSdsEM2BwJCREqdojMmknRLN5nJCS8$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A7SAXHSM3TXZYP4XPZO74ALZD6IGJAVCNFSM6AAAAABIHXCWCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGIZTANZVGY__;!!IKIcsDr4!X1PcvwDW6vnR1ruL6JWmXScTfDYpTzGQyZJfNqsEwbxrLYUcfE_Qsak1aQSdsEM2BwJCREqdojMmknRLGd0Vyn0$>. You are receiving this because you were mentioned. This e-mail and any files accompanying this e-mail contain confidential information belonging to the sender. This information may contain protected health information and other material that is legally protected, privileged or proprietary. The information is intended only for the use of the individual or entity named above. The intended recipient of this e-mail is prohibited from disclosing this information to any other party unless required to do so by law or regulation and is required to delete or destroy the information after its stated need has been fulfilled. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on or regarding the contents of this electronically transmitted information is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this message.

[Jeffery-Wasty]

I asked for additional connection settings because I was not sure whether the image was comprehensive. For instance, you have mentioned in the email you are already using prepareMethod=prepare, but I do not see this as part of the connection options.

Prior to any meeting, I would like to see where the slowness is happening in the driver. To do that, I will need to see the logs. Please capture the logs as described in the link above and either attach it to this issue or to the email thread. If a solution is not clear from the logs, then we can have a meeting to discuss this issue further.

[paragpandit123]

I will work with DBA's and try to get that log . [cid:d9e5e708-4130-44dd-be01-cabfd109b6a0] Parag Pandit Application Architect ***@***.******@***.***> 732.789.7326 [Logo Description automatically generated]<http://www.lumeris.com/> [Icon Description automatically generated] <https://www.linkedin.com/company/lumeris> [Logo Description automatically generated] <https://twitter.com/lumeris>

…

________________________________ From: Jeff Wasty ***@***.***> Sent: Friday, May 24, 2024 3:49 PM To: microsoft/mssql-jdbc ***@***.***> Cc: Parag Pandit ***@***.***>; Mention ***@***.***> Subject: [External] Re: [microsoft/mssql-jdbc] Slowness in MS SQL Server JDBC driver (Issue #2432) EXTERNAL Client / Partner / Business Vendor This email originated from a Lumeris affiliated entity. Please continue to exercise caution when opening links or attachments. Note: If there is a security concern, please contact the Service Desk immediately<https://lumeris.service-now.com/sp?id=sc_cat_item&sys_id=3f1dd0320a0a0b99000a53f7604a2ef9&sysparm_category=e15706fc0a0a0aa7007fc21e1ab70c2f>. I asked for additional connection settings because I was not sure whether the image was comprehensive. For instance, you have mentioned in the email you are already using prepareMethod=prepare, but I do not see this as part of the connection options. Prior to any meeting, I would like to see where the slowness is happening in the driver. To do that, I will need to see the logs. Please capture the logs as described in the link above and either attach it to this issue or to the email thread. If a solution is not clear from the logs, then we can have a meeting to discuss this issue further. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microsoft/mssql-jdbc/issues/2432*issuecomment-2130255936__;Iw!!IKIcsDr4!Vlsa2qYrNr0wLgySdJ4L7Ws7WOnm8z2B_o9nNOn42qjH9idc2BywPGniYVkhR7EyPX_fl_1lKMnXUpfpjhrWdiE$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A7SAXHWNDTR3VEPQASAKHJTZD6KSZAVCNFSM6AAAAABIHXCWCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGI2TKOJTGY__;!!IKIcsDr4!Vlsa2qYrNr0wLgySdJ4L7Ws7WOnm8z2B_o9nNOn42qjH9idc2BywPGniYVkhR7EyPX_fl_1lKMnXUpfpjcDyJnI$>. You are receiving this because you were mentioned.Message ID: ***@***.***> This e-mail and any files accompanying this e-mail contain confidential information belonging to the sender. This information may contain protected health information and other material that is legally protected, privileged or proprietary. The information is intended only for the use of the individual or entity named above. The intended recipient of this e-mail is prohibited from disclosing this information to any other party unless required to do so by law or regulation and is required to delete or destroy the information after its stated need has been fulfilled. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on or regarding the contents of this electronically transmitted information is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this message.

[paragpandit123]

This log should be enabled on SQL server right ?

[paragpandit123]

I guess we need to meet to understand the log request better.

[David-Engel]

There is client-side logging for the JDBC driver. The JDBC driver uses standard Java logging that can be configured through your application or via a config in your Java home. See the link previously provided for details: https://learn.microsoft.com/en-us/sql/connect/jdbc/tracing-driver-operation?view=sql-server-ver16

[paragpandit123]

We are using Snaplogic tool to connect to SQL server. Not sure what can be done there.

[David-Engel]

Contact Snaplogic support for assistance with logging in their application.

[paragpandit123]

We are working with snaplogic and will get back

[AlBundy33]

I'm not related to the project but here are my two cents...

We used to complete our data load of around 200 tables under 2 hrs with other driver and when we moved to SQL JDBC drivers , the load is not completing even after 7 hrs.

what means "other driver" - older version of mssql-jdbc, other vendor (e.g. jTDS), ...?
have you tried different commection-settings (e.g. disabled encryption, datbase-user instead of domain-user, ...)?
have you tried to create a standalone-testcase that repeoduces the problem?
You can also try tools like JProfiler to find hotspots.

you also wrote that this is an production issue - does this mean that this issue does not exist in your development and integration-systems? 🤔