Support For Big Endian Systems

[saitama951]

The Microsoft SqlClient driver crashes on the s390x. This error stems mostly from the use of BitConverter function used in method definitions across the code and improper byte ordering for big endian systems.

The BitConverter class does conversions according to the endianess of the system it is operating on which causes incorrect reads of little-endian packets sent by the server.

[codecov]

Codecov Report

Attention: 21 lines in your changes are missing coverage. Please review.

Comparison is base (4319661) 72.48% compared to head (0333a90) 71.30%.
Report is 1 commits behind head on main.

❗ Current head 0333a90 differs from pull request most recent head 499bae3. Consider uploading reports for the commit 499bae3 to get more accurate results

Files	Patch %	Lines
.../netcore/src/Microsoft/Data/SqlClient/TdsParser.cs	22.22%	14 Missing ⚠️
...icrosoft/Data/SqlClient/SqlSequentialTextReader.cs	20.00%	4 Missing ⚠️
...c/Microsoft/Data/SqlClient/TdsParserStateObject.cs	33.33%	2 Missing ⚠️
...oft/Data/SqlClient/TdsParserStateObject.netcore.cs	93.75%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2170      +/-   ##
==========================================
- Coverage   72.48%   71.30%   -1.18%     
==========================================
  Files         309      306       -3     
  Lines       61959    61892      -67     
==========================================
- Hits        44911    44135     -776     
- Misses      17048    17757     +709     

Flag	Coverage Δ
addons	92.88% <ø> (ø)
netcore	73.73% <67.18%> (-2.94%)	⬇️
netfx	70.16% <25.00%> (+0.26%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[saitama951]

Hi, @Wraith2 @David-Engel can you please review this?
Thanks

[Wraith2]

Is there a way to do this using the span we already have instead of allocating a new byte[]? If an array is needed it could be stackalloc-ed since we know it is fixed size.

[DavoudEshtehari]

@saitama951 any answer to this query?

[saitama951]

@Wraith2 Thank you for the suggestions, I have made the changes. not sure why the CI is failing, specifically win 2022 one's (Failed to start Sql browser)

[JRahnama]

@saitama951 failures are not related to your changes. Somehow starting SqlBrowser task fails on the pipelines. We will look into them.

[Wraith2]

I'm a little concerned about the number of preprocessor regions. If there's any way to extract common patterns into functions so we have fewer ifdefs it'd be nice. Otherwise without the ability to test i'll go with your changes as long as you've got the tests working.

Out of interest are you using managed networking? or is there a big endian native sni.dll ?

[saitama951]

I'm a little concerned about the number of preprocessor regions. If there's any way to extract common patterns into functions so we have fewer ifdefs it'd be nice. Otherwise without the ability to test i'll go with your changes as long as you've got the tests working.

Out of interest are you using managed networking? or is there a big endian native sni.dll ?

I agree with you the if defs are unnecessarily populating the code, I do see the csproj includes a package System.Memory which supports BinaryPrimitives for all the targets but some functions are not supported with netstandard2.0 so I have added a file with the compatibility support, let me know your thoughts on this.
to answer your second question it uses the managed networking dll.

[DavoudEshtehari]

My concern is how this change is going to impact the performance by impacting the hot paths. Does anybody have a perf test to compare it with before the change?

[saitama951]

@DavoudEshtehari
I'm looking into the perf tests.

[saitama951]

@DavoudEshtehari
Hi,
I did run the performance tests and I'am seeing some noise and inconsistency in the results with the patch and without the patch for all the various number of times I run the tests. any guidance on this would help.
Thanks

[Wraith2]

They're quite noisy. Unless the differences are very large I wouldn't worry about it too much.

[DavoudEshtehari]

@saitama951 Make sure there is no heavy background process running while your test is running.

[H-Yeo]

Did we truncate a word here/spelling mistake? Are you trying to say Compatible/Compact? I think it's okay to make it more clear if you need to say Compatible.

[saitama951]

@H-Yeo Yes, This was meant to be Compatible (since certain functions of bitconverter were not supported in older frameworks). I will make the necessary change .Thanks

[DavoudEshtehari]

@saitama951 May I ask you to share the perf project and the result from your perf test run?

[saitama951]

@DavoudEshtehari Sorry for the delay in response, I'am running the performance tests present in the SqlClient and was using the ResultComparer tool to compare the results with and without the patch. Initially I was facing a lot of issue in generating a consistent set of data with various runs. As of now I'am able to pertain the consistency and see no reproducible regression, here are the perf results from the various runs.
run-1.xlsx
run-2.xlsx
run-3.xlsx
run-4.xlsx

[saitama951]

In Addition to the files, here is a summary output

[omen ResultsComparer]# dotnet run -c Release --base /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/without_change/result-1/results/ --diff /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/with_change/result-1/results/ --threshold 10% --full-id -f net7.0
summary:
better: 1, geomean: 1.420
total diff: 1

No Slower results for the provided threshold = 10% and noise filter = 0.3 ns.

| Faster                                                                   | base/diff | Base Median (ns) | Diff Median (ns) | Modality|
| ------------------------------------------------------------------------ | ---------:| ----------------:| ----------------:| --------:|
| Microsoft.Data.SqlClient.PerformanceTests.SqlCommandRunner.ExecuteScalar |      1.42 |      49766159.25 |      35054435.75 |         |


[omen ResultsComparer]# dotnet run -c Release --base /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/without_change/result-2/results/ --diff /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/with_change/result-2/results/ --threshold 10% --full-id -f net7.0
**No differences found between the benchmark results with threshold 10%.**

[omen ResultsComparer]# dotnet run -c Release --base /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/without_change/result-3/results/ --diff /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/with_change/result-3/results/ --threshold 10% --full-id -f net7.0
**No differences found between the benchmark results with threshold 10%.**

[omen ResultsComparer]# dotnet run -c Release --base /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/without_change/result-4/results/ --diff /home/manjaro/sqlclient-patch/SqlClient/src/Microsoft.Data.SqlClient/tests/PerformanceTests/with_change/result-4/results/ --threshold 10% --full-id -f net7.0
**No differences found between the benchmark results with threshold 10%.**

[omen ResultsComparer]#

Specs:

BenchmarkDotNet=v0.13.2, OS=manjaro 
AMD Ryzen 7 5800H with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores
.NET SDK=7.0.111
  [Host] : .NET 7.0.11 (7.0.1123.46301), X64 RyuJIT AVX2

[JRahnama]

@saitama951 can you address the conflict so we can proceed with the preview release today?

[panoskj]

You have reverted some changes I introduced with #2168 . Note this file is shared between netcore and netfx versions. I highlighted two examples, but there are more lines.

[panoskj]

@saitama951 The last 3 commits look good now. I haven't reviewed the merge commit at all, it will take me some time. By the way I would squash these commits since reviewing them individually is harder.