-
Notifications
You must be signed in to change notification settings - Fork 374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance issue, using odbc is faster than sqlsrv with PDO #189
Comments
@sean256 Very interesting find. Let us see what is going on here in our test labs. My hunch is that there is some conversion logic when using sqlsrv that causes the perf differences when comparing it to vanilla ODBC |
I have the same slowness with PDO_SQLSRV. see call graph. My profiling points to the PDOStatement::fetch method. OS: Ubuntu 16.04.1 (tested on CentOS 7.2 too) |
@richerlariviere Just wondering what tool did you use to create the call graph? Thanks for the info. |
@richerlariviere @jansor @sean256 We are investigating this right now. From the surface it looks like SQLSRV fetch functions compared to ODBC are very extensive and try to predict and cover more cases. This could be one of the reasons for the slowness. This is not conclusive and we should have more to share in the future. |
@ALL, Also need to check too. Edit: |
@mlhkr If I understand your edit correctly, you found that your issue was with the environment and not the driver? |
@v-mabarw exactly. Driver's query speed is amazing. |
@mlhkr Good to know! Is there any way we can help with any other performance issues you are facing? |
@meet-bhagdev I'm not sure what kind of observation are you looking for? |
@richerlariviere I'm talking about the performance issues 👍 |
The only performance issue I got is the one with the PDO driver. SQLSRV is super fast on my 2 setups. |
Any update for the performance issue? I tried with the latest build and got the same problem. |
Something my team discovered about this, increased latency seems to impact sqlsrv far more than other drivers. When testing sqlsrv against dblib on a sever very close to the DB they both perform very well. As the latency increases sqlsrv gets far worse than dblib. A set of quires on the "near by" server (latency < 5ms) they would average ~400ms for both dblib and sqlsrv. From our office where we dev (latency of 29ms) dblib would hit ~500ms but sqlsrv is averaging at 1.2 seconds. |
Do you use dblib extension with PHP7 ? |
@sean256 We have not tested latency scenarios. What do the results look like when both sqlsrv and dblib are very close to the db? Also does the aforementioned repro script still stand for this scenario? |
I no longer have the environment to test this so I can't give you solid numbers, sorry about that. When they were close to the DB they both performed very well. |
Trying to figure out why our Drupal projects run slower than that of our MySql friends, I've setup a test script that shows PDO_SQLSRV being x4 to x5 times slower than the PDO_MYSQL driver. Either my test script is super flawed, or we have serious issues with the PDO driver for Microsoft SQL Server. Both MySQL and MSSQL servers are on local machine during tests, using PHP 7.0.14 x64 with ODBC 13.1. This is the output of the test script:
Inserting the data took x6 longer on MySQL (probably due to me having no clue on to how properly setup MySQL). The issue is when retrieving data. SQL Server PDO shows a x2 improvement in using buffered queries vs not using them (great!). But when compared to MySQL, SQL Server PDO fetches data x5 times slower. To make thing matter worse, if in the sample I comment out the following line:
(So that everything is fetched as a string), then the PDO numbers get even worse, like x3 slower than the original: sqlsrv:server=localhost;Database=php_d7test Actually in this case non buffered perform better than buffered. And in the best case, they are x16 slower than MySql. Either my test/setup is totally flawed, the MSSQL PDO driver has very bad performance or a combination of both. Worth link into. |
@david-garcia-garcia We are looking into this. Were you able to test this with different datasets? |
@sean256 if your SQL Server is remote then disabling MARS may work. Set The caveat is that disabling MARS means you can only have one active statement/result at a time. So depending on how your PHP code is structured you could encounter errors when executing a second statement or accessing a result. There are examples on MSDN that explain MARS. They maybe in other languages but the same concepts apply to PHP. You should review the MSDN documentation so you understand how disabling MARS could affect your code. @david-garcia-garcia we are looking at the performance difference with PDO ODBC as a baseline instead of MySQL. PDO_SQLSRV is still slower than PDO ODBC and we are investigating how to close that gap. |
@v-dareck @meet-bhagdev It is great news to hear you are starting to take the performance characteristics seriously. With such a performance critical product, benchmark metrics should be publicised with every release and the team should be aspiring to beat the competition. |
That helped. A little. "If your SQL Server is remote". When is the SQL Server not remote except on a development environment?
If using client buffered queries, disabling MARS should not have any impact, because result set is fetched into PHP the moment the pdo statement is executed. Tested on the Rackspace Cloud with Virtual Servers, disabling MARS yields +25% better speed on a real world application (Drupal 7) for the PDOStatement::execute coverage. I.e. an execution path that runs 152 statements spend average 201ms in PDOStatement::execute with MARS enabled, and 150ms with MARS disabled. In this specific scenario 50ms are irrelevant to the complete execution path. I've got execution paths with > 16,000 statements run (pure crap code, but nothing to do about it - it's open source implementation that won't change). In that specific case 95 seconds spent in PDOStatement::execute went down to ~70 seconds. So... summarizing the different results I've been seeing: Disabling MARS + Using client buffered queries + Enabling numeric fetches seem the best performance wise combination. |
@Joe-U-Questionmark Point taken on the benchmark metrics. We are not there yet but we will work towards it. We are currently focusing on improving the performance by addressing issues such as this one. @david-garcia-garcia thanks for testing the recommendation. We will continue the testing here and document it clearly in our docs once we are fully satisfied with the perf improvements. |
@sean256 @david-puglielli
@dhazelett |
@rax1606 that is a fair point if that is a requirement, I'd not recommend it. |
I have something new to point here about the performance issue I have been facing. The issue that I was having seems to be there only for docker containers. I could find one ubuntu box in my company,with SQLSRV-4.0.7 and the setup works perfectly fine. The db performance is very nice there. But even after setting the SQLSRV version to 4.0.7 on my docker container, I couldn't get it work fine. The response time increase by about 6 times to be specific, when I run my application in docker. I was wondering if I am missing something in my docker configuration? Or is there anything else that I should be taking care while devising a docker image for php and sqlsrv. I have already uploaded my docker file sript and the way I am connecting to the db's. Please guide me any ways you can. I really need my application working in docker and I seem to have hit a dead end at this point. Thanks. |
@rax1606 Looking at your docker script:
You mentioned your code is hitting a rest api. Is the rest api hosted in the docker container or is that external, too? Have you checked the basics? Specifically, are you allocating enough CPU and memory to docker to support what you are running? Are you pulling a lot of data (inadvertently maybe?) from SQL Server into the rest api and then into PHP for processing? It doesn't seem like it should require much, but without seeing any code, it's possible you are doing things that require more memory and/or CPU than you are allocating to docker. David |
@david-puglielli here's a funny one. 5.2.0, both ODBC 13 & 17 exhibited same behavior. Here we found a problem with a query's performance, where odbc/freetds had sub ms response times This one is from a backend laravel app, in order:
same as above, but removing the quotes from the parameter:I'm going to try to get some repo stuff on monday for you guys. |
After passing more than a year dealing with this performance issue, to me it's comes down to choice at this point. Or I choose to stay with SCROLL cursors with the BUFFERED option, despite consuming more RAM, or I choose ODBC and miss some new settings and more importantly encoding support. Most of PHP frameworks will use PDO and expect support for |
hi @codecrafting-net I suppose you are comparing our PDO_SQLSRV driver to PDO ODBC? Note that you do not have to use BUFFERED option to get row count. For example, there exists static cursor on the server side that lets you get rowCount(). Buffered queries are only recommended for small to medium result sets. Using AdventureWorks DB for example, you can do the following:
Hope this clarifies things. If not, please create a new issue with more details. |
Thanks for respond @yitam, but that's the problem. If I do not use SQLSRV_CURSOR_BUFFERED all other cursors have a TERRIBLE performance, at least on my environment. Just as example: $query = "SELECT TOP 1000 * FROM reports";
$stmt = $conn->prepare($query, array(PDO::ATTR_CURSOR => PDO::CURSOR_SCROLL, PDO::SQLSRV_ATTR_CURSOR_SCROLL_TYPE => PDO::SQLSRV_CURSOR_STATIC));
$stmt->execute();
echo count($stmt->fetchAll()); Takes 1.1 to 1.3 seconds and if I change to SQLSRV_CURSOR_BUFFERED takes 80 to 70 ms, so a abysmal difference. I am using Just to clarify things, follows my environment settings:
This can't be a CPU, network thing since all tests are done with the same environment For last I just not creating a new issue beacause I still think is about performance in ODBC comparison, but if it is better I can create a new one. |
Hi @codecrafting-net Just so you know, when using buffered cursors, our drivers store all results in memory when executing the query, so when you do fetching, the results are already in memory. On the other hand, without cursors, we do make a trip to the server to get the results. That's why buffered cursors are recommended for small to medium result sets. You might want to compare the time taken for a combination of execute and fetch, and the performance with or without cursors should not be that significantly different. Otherwise, please create a new issue. Thank you. |
@yitam Ok I get this, but why PDO without cursors or even just ODBC is as fast as buffered, but consuming way fewer resources? Look, to me I saying that the difference with or without cursors is abysmal, this |
hi @codecrafting-net Many thanks for your inputs, and it would certainly help us to improve if you can provide details, say a more concrete repro scenario in a new issue. Your contributions will be much appreciated. Thank you. |
Same here, except with Laravel. I can't change (or rather, don't know how to) the scroll type. Queries take 1~4s to execute for no reason. |
I can add to the laravel thing too, in some queries simply not escaping the query gains a bit of performance too. For instance we had a where in like so:
Returned results in 30s or more, just removing them using DB::raw() it dropped to 1s |
Looks like rolling back to pdo_sqlsrv 4.3.0 and ODBC Driver 13.0.1 fixes the lag... or something. No idea what's going on. |
Hi @Raymonf and @dhazelett As I said to @codecrafting-net, please create a new issue and provide details and examples (table schemas, sample queries, etc.) to help us investigate and/or reproduce the problems you're experiencing. Thank you for your contributions. |
hi @Raymonf Concerning the performance difference between 4.3.0 and 5.2.0, we have fixed one of the problems that might contribute to degrading performance in our latest 5.2.1-preview. Would you be able to give it a try? |
Sorry for the late reply. I don't want to mess something working up right now so I'll create a Docker container of my site or something. As for reproduction, even a simple |
@Raymonf please provide us more details in a new issue, in particular your environment, SQL Server version, etc. What I mentioned about the changes from 5.2.0 to the latest 5.2.1-preview mainly affected how the drivers bind parameters. Nonetheless, I just did a quick test comparing 5.2.0 and the latest 5.2.1-preview with ODBC 17.1 in diff platforms. I do not use Docker in any of them. The following table shows the time taken to run a simple query
|
@yitam It looks like 5.2.1-preview fixed whatever problem 5.2.0 had with horrible performance. Nevertheless, the SQL Server version is 14.0 running on Windows Server 2016 (not 1709). Ping between servers is <2ms, so that was definitely not the issue. The original server (which had the issue before downgrading) is running Ubuntu 16.04 x64, and the Docker container (which does not have the issue) runs Debian Stretch. Thanks! |
Hi @Raymonf, your prompt reply and details are appreciated. Thank you for letting us know that 5.2.1-preview has shown improvement. Again, please feel free to create a new issue if you want us to investigate further. |
Hello everyone! Thanks for all of the great chatter about this issue, I have read through all of the comments to make sure I'm up to speed. We have a very busy intranet only Drupal site currently running PHP 5.3 with a version 2.5 of the SQL Server PDO extension on Server 2008 R2 (Very old, I know). We are attempting to upgrade the site to PHP 7.2 and SQL Server PDO extension 5.3, but unfortunately the performance issues render the site unusable. The web server jumps to 100% CPU immediately, and stays there permanently. Running on 5.3 with extension version 2.5 we hover around 40% with a much more responsive site. We also tried PHP 5.6 with extension version 3.2 and the issue was exactly the same. I noticed today in the release notes for version 3.2 there is a performance issue for machines running Windows 7 or Server 2008 R2. Is it safe to say this is our problem since it looks like the performance issues weer fixed in the 5.2.1-preview and beyond? Thanks for the help! |
hi @bwhiteford, we would like to check if you have any updates for us. Just so you know, performance improvement is of high priority, and any inputs or suggestions are welcome to help us proceed further. |
I was having really bad performance here and the solution on https://stackoverflow.com/questions/45598786/slow-sqlsrv-pdo-query-performance-with-parameterized-queries helped me a lot. Basically, add |
@pedrorocha-net We're glad it helped. FYI, the latest preview release of the drivers includes additional optimisations for handling UTF-8 data, so that may help with performance as well. |
Also read this. Disabled due to load times. Simple page 1.7 Seconds Changed this and the same page load times went down to .04 Seconds |
Hi @sean256 and others I can't seem to reproduce the original issue with the latest pdo_sqlsrv and ODBC drivers with PHP 7.4. If any of you still experience similar problems please provide a repro script or scenario. Otherwise, we will close this issue due to inactivity. |
Closing this issue due to inactivity or lack of repro steps. |
For others who bump into this issue while looking into performance issues: There's more info on this on the wiki in this very repo: https://github.com/Microsoft/msphpsql/wiki/Recommendations-for-improving-the-performance-of-PDO_SQLSRV-and-SQLSRV |
While screwing around with trying to get connection pooling working I discovered that sqlsrv is much slower during execute than using odbc.
Test code (in milliseconds)
Output:
The text was updated successfully, but these errors were encountered: