Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plans to open source Microsoft.SqlServer.Management.SqlParser.dll #623

Open
clement911 opened this issue May 28, 2018 · 64 comments
Open

Plans to open source Microsoft.SqlServer.Management.SqlParser.dll #623

clement911 opened this issue May 28, 2018 · 64 comments

Comments

@clement911
Copy link

Apologies if this is the wrong place to ask, but is it possible to also open source Microsoft.SqlServer.Management.SqlParser.dll ?
It would seem like the right repo for it?

@clement911
Copy link
Author

Not going to happen? We use the parser in our app but it's always scary to grab a dll from the GAC and just copy it to our app.
If it was properly open sourced, it would give us confidence that this library is not going to be removed under us!

@kburtram
Copy link
Member

kburtram commented Jun 5, 2018

@clement911 SQLParser is the primary components used for SSMS, SQL Ops Studio and vscode-mssql language services. It is safe to assume the parser will continue to be maintained for the foreseeable future. There should be a redist you can use which is the recommend way to install this component.

The SQL Parser component is currently build as part of the SSMS internal repository. SSMS (and therefore SQL Parser) is built using a custom build environment that we'd need to update to a more standard environment prior to open sourcing. Also, there are various internal procedures we need to figure out, such as how best to handle vNext changes prior to public disclosure, etc.

We have backlog items for open-sourcing Parser, SMO, DacFx and other tools components, but there are quite a bit of details we need to work through prior to publishing these components that weren't originally developed as OSS. Unfortunately, I don't have an ETA for this work.

@clement911
Copy link
Author

I see.
Well I hope that the relevant team can get the ball rolling in this direction.
I think it would open up a number of very interesting scenario that would empower the sql server community.
For us, we want to pre-process our scripts with macros and also format consistently but I imagine there would be other cool scenarios such as linters and code analyzers.

@clement911
Copy link
Author

There should be a redist you can use
Which redist do you mean? The dll in the sql server folder?

@jzabroski
Copy link

@clement911 Microsoft has started publishing the SqlParser as a nuget package, which is a step in the right direction: https://www.nuget.org/packages/Microsoft.SqlServer.SqlParser/140.17279.0-xplat

@clement911
Copy link
Author

Great to hear.
Here is my user voice entry if more people would like to vote:
https://feedback.azure.com/forums/908035-sql-server/suggestions/34741105-open-source-microsoft-sqlserver-sqlparser-dll-re

@clement911
Copy link
Author

@jzabroski I found a few bugs in the parser from Microsoft.SqlServer.SqlParser.
OVER clauses and NULLIF are not parsed correctly.
Do you know where I can report bugs?

@jzabroski
Copy link

jzabroski commented Aug 14, 2018

@kburtram works for Microsoft and would be better to answer that. Unless it's open source I doubt these issues get resolved. I anecdotally believe a lot of bugs have crept into SqlParser over the last 3 releases. I have done very odd things where I reverse the string manually in the text, assign it to @sqlcommand, reverse it again in memory in TSQL, and execute it using EXEC (@sqlcommand). Then it executes fine. Most of the issues have to do with privileged commands like sp_dropuser. I posted about it on StackOverflow.

Maybe try the same trick..

@clement911
Copy link
Author

Thanks @jzabroski , although I'm not really sure what you mean by 'reversing the string'.
We're just trying to parse some sql code.

@kburtram I'm calling on to you again then :)
We're taking a pretty big dependency on the sql parser nuget package. We're really glad it is now on nuget!
We are relying on the parser to provide us with an AST that we can visit to inspect various things about
our TSQL code and even change on the fly via pseudo macros.
It is very powerful and pretty cool but the problem is that the AST is often not full fidelity to the code given. It seems that certain code paths are not fully parsed, such as OVER bodies, NULLIF parameters, and more.
Is there anything we can do to facilitate the open sourcing a parser?
I'm sure it would open up some really cool possibilities such as linters, analyzer, editors, macros, etc...
It's pretty frustrating because it looks like the parsing works 95% great, but there are just a few issues.

If this is a dead end, are there any other TSQL parser out there you know of? We need a parser capable of producing a full fidelity AST...

@jzabroski
Copy link

jzabroski commented Aug 15, 2018 via email

@kburtram
Copy link
Member

@clement911 I don't think we have plans to open source the parser within the next few months. We'd probably want to do that as part of opening a larger body of code, such as SMO, which will take a while.

@shueybubbles is the owner of the Parser Nuget package (and SMO) and may have additional context. If you provide example SQL that the parser isn't correctly handling I can open an internal bug against the SQL Parser component.

In the near-term it's possible something like the parser that @jzabroski mentioned may work better for you. Though we certainly would want to fix any significant bugs in our language service SQL parser (and eventually open-up that codebase).

@clement911
Copy link
Author

I haven't looked at that other parser no. If we get desperate we may have to look at it...

I will get some minimal bug repros and post them here shortly.

@clement911
Copy link
Author

So here are the 2 bugs I found so far.
I parse and inspect the AST with Parser.Parse(sqlQuery).Script.Xml

Bug 1)
Compare the two ASTs for the two following queries.

Query 1: SELECT NULLIF([col], 0) FROM [Table]
Query 2: SELECT ISNULL([col], 0) FROM [Table]

The queries are identical except that one is using NULLIF and the other is using ISNULL.
Yet, the AST of the query with ISNULL includes the parameters within the function call whereas the AST of the query with NULLIF doesn't.

nullif_bug

Bug 2)

The OVER() clause seems to be swallowed up in the SUM aggregate function call and is not present at all in the AST.

over_bug

@jzabroski
Copy link

jzabroski commented Aug 15, 2018 via email

@clement911
Copy link
Author

Of course we know that there is at least one parser that is comprehensive, and that is the parser used by Sql Server itself. If only that was available...

On the programming language side, the situation has become ideal, with C# and Typescript both coming with a language service as a first class citizen, which is powered by the same parser that the compilers use. The tooling has become much better thanks to that...

@clement911
Copy link
Author

I just came across another parser.

https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/
https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/dd194286(v=vs.100)

This just gives me hope.
I will investigate further but for now I'm confused.
What is the relationship between this project and Microsoft.SqlServer.SqlParser?

@kburtram
Copy link
Member

@clement911 we use Microsoft.SqlServer.Management.SqlParser for our language service, so it's optimized around those types of tasks (e.g., IntelliSense suggestions, QuickInfo tooltips, diagnostic errors).

I think ScriptDom is more around working with the AST, which may work better for your scenario. You may also need to bring in DacFx to work with ScriptDom, and it's not available on .Net Core AFAIK.

@kevcunnane do you have additional context on ScriptDom vs. SqlParser?

@clement911
Copy link
Author

Good news!
Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser is able to parse those test cases I posted above!
This is looking very promising for us. The API is different but at least the AST covers the full syntax.

@kburtram I see what you're saying, although it seems sharing a single parser would be worthwhile given the complexity of the task...

FYI it is also available as a nuget package and the only dependency is .Net 4.0.
https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/

@FlorianGrimm
Copy link

Is it possible to publish your ANTLR definition?
I'm using currently ASP.NET Core with .NET Framework. And for ASP.Net Core 3.0 ....

@clement911
Copy link
Author

Hi @FlorianGrimm ,
Could you share your scenario? What are you trying to achieve?

I'm not sure if you checked out Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser but for us it worked much better. As I understand it, it is the same parser as the one used by Sql server itself, but exposed as a .net dll.

@FlorianGrimm
Copy link

I currently testing Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser, but it doesn't have a a .Net Core assembly.
I saw the ANTLR classes in it. ANTLR.org has a grammer file, but I don't know how much close it is.
I have to do some type analysis.
Basically I want to know if i call a SQL Statement what will the result look like Name and Type - for more than one SELECT Statement. No GUI no external exe.
It tried some libraries. Even for "simple" things I want behave like the server e.g. nested comments /* \n--*/ \n */.
I guess I saw 3 different SQL Parser from Microsoft. TsqlParser works well, but it's only available for .Net 4.x.
The ANTRL 4 runtime now has a return value for visitors which can be really useful - IMHO.
That's why I asked for the grammer files

@clement911
Copy link
Author

Got it.
I see your points. It would be great if the team could update Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser to be available on .net core.
Asp.net core 3.0 won't be available on .net so we will definitely have to upgrade to .net core and we do use TsqlParser...
It's frustrating, there seems to be no way to reach the relevant teams...

With respect to your scenario have you considered calling this stored proc:
https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-describe-first-result-set-transact-sql?view=sql-server-2017

@MarcusKohnert
Copy link

@FlorianGrimm This pre-release package is working well for our scenarios on .NET Core.

@clement911
Copy link
Author

@MarcusKohnert , is there a difference between this package and https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/ ?

@MarcusKohnert
Copy link

@clement911 I don't know your referenced package so I don't know if there are any differences. Microsoft.SqlServer.SqlParser/150.18068.0-xplat is packaged by Microsoft at least which is a plus for me. ;)

@srutzky
Copy link

srutzky commented Apr 18, 2019

Just FYI: Another parsing bug that is still affecting SSMS (and likely also SQLCMD) is related to the odd (and probably not well known) ability of SQL Server to allow nested block comments. Splitting up batches based on the GO separator has issues with nested block comments, which I detailed in the following ticket:

"GO" in 2nd half of nested block comments breaks batch parsing in SSMS and SQLCMD

@dmarkle
Copy link

dmarkle commented May 23, 2019

Ugh, the pre-release package has been unlisted from nuget:

"The owner has unlisted this package. This could mean that the package is deprecated or shouldn't be used anymore."

Does anyone see if there is an update, maybe if MS is shipping this somewhere else now?

@clement911
Copy link
Author

@shueybubbles
Copy link
Contributor

shueybubbles commented May 23, 2019 via email

@dmarkle
Copy link

dmarkle commented May 25, 2019

Oh. That's awesome. I've been waiting for that to go on Nuget forever. It's like a dream come true, and SqlParser from SMO works great (except SMO doesn't really like installing on my dev workstation with tons of other assemblies already installed). But I can get around that.... Next thing you know, they'll be porting SQL AMO to .Net Core (ha)

@clement911
Copy link
Author

See below. Only one target framework moniker, for net40

image

@ststeiger
Copy link

ststeiger commented Sep 16, 2019

NetStandard 2.0
Show NetStandard 2.0

Build output
Show build output

.NET Core 2.1
Show .NET Core

And it's running:
ScriptDom on NetCore

@ststeiger
Copy link

ststeiger commented Sep 16, 2019

What you're doing wrong is taking the DLL from nuget, instead of taking the one that AzureDataStudio uses. (AzureDataStudio runs on Linux)

https://github.com/microsoft/azuredatastudio

Download the zip archive, and get the NetStandard2.0 dlls from subfolder
resources\app\extensions\mssql\sqltoolsservice\Windows\<VERSION>

With respect to "correct grammar", you only need the grammar file if you build your own parser with ANTLR. If you use the SqlParser provided by MS, you don't need any grammar file at all - the corresponding code has already been created and is part of ScriptDom/SqlParser.

@clement911
Copy link
Author

Oh I see. Great.
Are there any plans to publish this Net standard dll in Nuget, or update the existing nuget package to use this dll?
It would mean we can easily update to the latest version moving forward, as opposed to referencing the dll manually.

@shueybubbles
Copy link
Contributor

The Microsoft.SqlServer.DacFx nuget package has an appropriate license for ScriptDom, which it includes.

@clement911
Copy link
Author

@shueybubbles Microsoft.SqlServer.DacFx target Net framework though. I want to use .net core.

@ststeiger
Copy link

@clement911: I have no idea what license Microsoft.SqlServer.TransactSql.ScriptDom comes with, so I'd rather not publish it.

@clement911
Copy link
Author

Referencing the dll from Azure Data Studio works. Great!
Hopefully it can be packaged in nuget.

@clement911
Copy link
Author

Me neither. I guess if the right people at Microsoft could get together maybe they could sort this out.
At least there is a work around for now.

@shueybubbles
Copy link
Contributor

The SqlToolsService used by ADS is itself open source. It gets all its dependencies from nuget.
Checking its PackageReference (see https://github.com/microsoft/sqltoolsservice/blob/master/src/Microsoft.SqlTools.ServiceLayer/Microsoft.SqlTools.ServiceLayer.csproj) for DacFx, we find:

<PackageReference Include="Microsoft.SqlServer.DacFx" Version="150.4534.2-preview" />

Installing that package, we find the following netstandard2.0 DLLs:

 Directory of C:\Users\username\.nuget\packages\microsoft.sqlserver.dacfx\150.4534.2-preview\lib\netstandard2.0

09/11/2019  02:09 PM    <DIR>          .
09/11/2019  02:09 PM    <DIR>          ..
09/05/2019  09:05 PM        11,212,920 Microsoft.Data.Tools.Schema.Sql.dll
09/05/2019  09:04 PM           257,144 Microsoft.Data.Tools.Utilities.dll
09/05/2019  09:04 PM           329,848 Microsoft.SqlServer.Dac.dll
09/05/2019  09:04 PM           752,760 Microsoft.SqlServer.Dac.Extensions.dll
09/05/2019  09:06 PM         4,559,128 Microsoft.SqlServer.TransactSql.ScriptDom.dll
09/05/2019  09:07 PM            22,296 Microsoft.SqlServer.Types.dll

@jzabroski
Copy link

Thanks, David Shiflet! @shueybubbles

@ststeiger
Copy link

What do you use Dac for ?

Anybody knows what Microsoft.Data.Tools.Schema.Sql.dll and Microsoft.Data.Tools.Utilities.dll can do ?

@clement911
Copy link
Author

@shueybubbles oh I see, so it's a preview package.
I was looking at Microsoft.SqlServer.DacFx.x64...

I don't want to be a pain here, but in our project, we only need Microsoft.SqlServer.TransactSql.ScriptDom.dll
The package Microsoft.SqlServer.DacFx has a LOT of dependencies and we'd love to see small package for Microsoft.SqlServer.TransactSql.ScriptDom.dll

@sherland
Copy link

This is rather anoying ;-)
https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/ should be netstandard compatilbe, and Microsoft.SqlServer.DacFx should have a dependency on the nuget, instead of embedding its own copy.
As an example: Which dll will be used in projects that reference both Microsoft.SqlServer.TransactSql.ScriptDom (could be indirecty) and Microsoft.SqlServer.DacFx?

@kevcunnane
Copy link
Contributor

@sherland that's not an official Microsoft package. The only official, NetStandard-compatible way to get ScriptDom is to reference the DacFx nuget package. I hear you on it being inconvenient to pull in all the DLLs if you just need ScriptDom - @shueybubbles or @pensivebrian can likely comment on priority of separating it into its own package.

@duncansmart
Copy link

I've just stumbled across this. Sorry about the Microsoft.SqlServer.TransactSql.ScriptDom nuget package that I started uploading years ago causing all the confusion recently, but I'm struggling to see how I could have made things much clearer 🤷‍♂
image

Good to see .NET Core/netstandard support in Microsoft.SqlServer.DacFx!

@jzabroski
Copy link

@duncansmart The "Microsoft." prefix is supposed to be reserved for Microsoft projects. On top of that, "Microsoft.SqlServer.Management" is literally the prefix for Smo. I know you feel you made things obvious, but how do you think developers were confused?

@duncansmart
Copy link

@jzabroski I've been getting emails from people asking for a .NET Core version of that package and people here seem to think it's an official release. I added it years before the official DacFx packages were added. Naming it after the assembly it packaged seemed sensible in allowing people to be able to find it, indeed this was pretty common practice in the early days of NuGet when some parts of Microsoft were slow in releasing official NuGet packages.

@shueybubbles
Copy link
Contributor

shueybubbles commented Nov 6, 2019 via email

@jzabroski
Copy link

jzabroski commented Nov 11, 2019

@duncansmart Either way, your naming of the NuGet package isn't desirable today given Nuget package prefix id's, specified here: https://github.com/NuGet/Home/wiki/NuGet-Package-Identity-Verification#what-does-a-reserved-package-id-prefix-do - This RFC went into effect in 2017. In general, since the 1990s and Java engineers using "reverse domain name" for namespaces, e.g. com.java.*, it's well-established not to use somebody else's company name as a prefix for a code artifact, whether it be an assembly name, package name or namespace.

You're in a strange spot right now, because permanently deleting a package is not supported by Nuget, yet that would be the right thing to do.

@WeihanLi
Copy link

Any progress?

@clement911
Copy link
Author

Coming back to the original question. Still no progress on open sourcing the parser?

@clement911
Copy link
Author

Any progress on open sourcing the parser Microsoft.SqlServer.Management.SqlParser?
What's the real blocker here?

@jzabroski
Copy link

Just a guess: $$$ :)

@clement911
Copy link
Author

This would probably be an investment and cost initially but it would probably pay off in the long term.
@kburtram any thoughts?
We've been waiting patiently for 3.5 years 😢

@FlorianGrimm
Copy link

The VS SQL Analyzer uses an modified / improved version.
Please give use the code and so the opportunity to build tools around the t-sql code.

@masonwheeler
Copy link

masonwheeler commented Jan 5, 2024

This thread seems to be the closest thing we have to a place to report bugs in Microsoft.SqlServer.Management.SqlParser for the moment. (If there's a better place, please let me know.)

I just ran across an annoying problem in the current NuGet build: calls to string_agg are parsed as a SqlBuiltinScalarFunctionCallExpression node, but string_agg is not a scalar function; it's an aggregate function. So this call should be represented by a SqlAggregateFunctionCallExpression node.

@Charles-Gagnon
Copy link
Contributor

@masonwheeler Could you please open a new issue? This repo is good enough for now, but finding specific issues buried in unrelated threads is difficult 😄

@masonwheeler
Copy link

@Charles-Gagnon Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests