Enable feature lightup on json strings (classification, bracematching, diagnostics). #59034

CyrusNajmabadi · 2022-01-23T19:27:20Z

Looks like this:

Dark mode:

TODO:

Support new System.Text.Json apis.
Support new System.Text.Json options api.
Ensure full testing.

CyrusNajmabadi · 2022-01-29T21:08:20Z

src/EditorFeatures/CSharpTest/EmbeddedLanguages/Json/CSharpJsonParserTests_BasicTests.cs

@@ -1171,7 +1171,7 @@ public void TestMissingColon()
  <Diagnostic Message=""Only properties allowed in an object"" Start=""20"" Length=""3"" />
 </Diagnostics>",
        @"<Diagnostics>
-  <Diagnostic Message=""Only properties allowed in an object"" Start=""20"" Length=""3"" />
+  <Diagnostic Message=""Strings must start with &quot; not '"" Start=""12"" Length=""1"" />


some tests changed as we now always report the earlier error in a json string.

jasonmalinowski

, although a few questions still pending. I think to me the main ones are:

The parser that deals with figuring out options specified by the enum seems a bit strange, since it seems I can pretty easily make combos of flags that don't make sense?
The brace matcher optimization to avoid decending into parts of the trees that aren't needed I think is still pending too, unless I missed where it got added. (GitHub is definitely getting slow with a scale of this PR, so hard to say.)

jasonmalinowski · 2022-01-31T22:53:16Z

src/EditorFeatures/CSharpTest/EmbeddedLanguages/Json/CSharpJsonParserTests_NstTests.cs

@@ -13,7 +14,7 @@ public partial class CSharpJsonParserNstTests : CSharpJsonParserTests
        private void TestNST(


Covered in an email i just sent to @jinujoseph and you. An open source suite of json tests.

jasonmalinowski · 2022-01-31T23:00:46Z

src/EditorFeatures/Core/EmbeddedLanguages/Json/JsonEmbeddedBraceMatcher.cs

+                if (child.IsNode)
+                {
+                    var result = FindObjectOrArrayNode(child.Node, ch);


@CyrusNajmabadi Did this happen or no? (it's possible it snuck into some other place somewhere?)

jasonmalinowski · 2022-01-31T23:03:21Z

src/Features/Core/Portable/EmbeddedLanguages/AbstractLanguageDetector.cs

+                // Stop walking up once we hit a statement.  We don't need/want statements higher up the parent chain to
+                // have any impact on this token.
+                if (syntaxFacts.IsStatement(node))
+                    break;


(I recognized this may have moved, asking anyways):

Does this need some check for lambdas, or OK with something outside a lambda impacting inside of it? I think there's a reasonable argument for it, this just my general caution any time I see check similar to this.

yeah. i think this is acceptable for now. We could always grow this later if users find this undesirable.

jasonmalinowski · 2022-01-31T23:06:21Z

...Core/Portable/EmbeddedLanguages/RegularExpressions/LanguageServices/RegexLanguageDetector.cs

+                var name = GetNameOfType(typeNode, syntaxFacts);
+                if (name != null)
+                {
+                    if (syntaxFacts.StringComparer.Compare(nameof(Regex), name) == 0)


We're just ignoring aliases here?

We've had discussions around source generators too where a "is the type here being created" helper that does this might be helpful, not sure if we should extract this pattern. At least that way the ignoring of aliases can be in one place so we can fix it everywhere later?

Yes we do. Note: this is going to go semantic in a followup PR this week to support StringSyntaxAttribute. So this is just about detecting this when older frameworks.

jasonmalinowski · 2022-01-31T23:07:45Z

...Core/Portable/EmbeddedLanguages/RegularExpressions/LanguageServices/RegexLanguageDetector.cs

+    /// </summary>
+    internal sealed class RegexLanguageDetector : AbstractLanguageDetector<RegexOptions, RegexTree>
+    {
+        private const string _patternName = "pattern";


Naming convention isn't PatternName here?

will fix in followup :)

jasonmalinowski · 2022-01-31T23:08:42Z

...Core/Portable/EmbeddedLanguages/RegularExpressions/LanguageServices/RegexLanguageDetector.cs

+
+        public RegexLanguageDetector(
+            EmbeddedLanguageInfo info,
+            INamedTypeSymbol? regexType,


We need this type to be createable even if we don't have a regex type, because there's still other comments we are looking for as well?

currect! we would still allow regex highlighting if you had // lang=regex on something.

jasonmalinowski · 2022-01-31T23:17:31Z

src/Workspaces/Core/Portable/EmbeddedLanguages/LanguageServices/LanguageCommentDetector.cs

+    /// </summary>
+    internal struct LanguageCommentDetector<TOptions> where TOptions : struct, Enum
+    {
+        private static readonly Dictionary<string, TOptions> s_nameToOption =


can do in followup.

jasonmalinowski · 2022-01-31T23:19:19Z

src/Workspaces/Core/Portable/EmbeddedLanguages/LanguageServices/LanguageCommentDetector.cs

+    /// </summary>
+    internal struct LanguageCommentDetector<TOptions> where TOptions : struct, Enum
+    {
+        private static readonly Dictionary<string, TOptions> s_nameToOption =


The assumption here is TOptions is a flags enum? Does something need to enforce that?

sure. i can add that in followup.

jasonmalinowski · 2022-01-31T23:27:28Z

src/Features/Core/Portable/EmbeddedLanguages/Json/JsonOptions.cs

+    /// <summary>
+    /// Parse using loose rules.  This is very relaxed and allows lots of features not allowed by standard IETF 8259
+    /// (like comments, and trailing commas).  This also allows all the additional constructs added by Json.net, like
+    /// constructors and string that use single-quotes instead of double-quotes.
+    /// </summary> 
+    Loose = 0,


Should this be structured that "Loose" is equal to Comments | TrailingCommas and strict = 0? Otherwise, it seems odd that some bits mean "be more strict" and some mean "be more loose"? I guess though this has to work this way because this is how the comment detector is looking for these particular names?

(in general this generic parser seems a bit janky here, since I guess a user could write loose,strict and we'd just go with that?)

so... there's two issues here. First: i want // lang=json to be as permissive as possible. Which means the 0 state should be that (though i suppose we could also have a way to pass the 'default' that a detector should pick if no other information is provided).

Second, there are actually two separate groups. The first choice is: "are we going json.net parsing, or .net parsing?" Second, if we're doing .net parsing, is it in strict mode (teh default), or does it allow these two variants.

Note that Loose is still a superset of that, allowing a broad set of other craziness that is not controllable.

I don't have a great way to model this IMO, but i'm open to ideas.

CyrusNajmabadi · 2022-02-01T06:02:51Z

The brace matcher optimization to avoid decending into parts of the trees that aren't needed I think is still pending too, unless I missed where it got added. (GitHub is definitely getting slow with a scale of this PR, so hard to say.)

It hasn't been done. I want to do it as part of a larger refactoring of the EmbeddedParsing system so that, like with normal Roslyn parsing, information abotu positions/widths flow up to the parent nodes. Then we can use that to narrow the walking down.

CyrusNajmabadi added 30 commits April 9, 2018 15:21

Merge branch 'jsonParsing' into jsonFeatures

87de235

Merge branch 'embeddedLanguageServices2' into jsonFeatures

f0faba4

Update resources.

bd1ad78

Restore loc strings.

e69f0f4

Restore loc strings.

303be3d

Remove regex test.

8d4a2fb

revert these files.

931b182

Merge branch 'jsonParsing' into jsonFeatures

350dea8

Merge branch 'jsonParsing' into jsonFeatures

74e778a

Merge branch 'jsonParsing' into jsonFeatures

67c7aa0

Support full interface

d898a7d

Merge branch 'jsonParsing' into jsonFeatures

b0d9cad

Merge branch 'jsonParsing' into jsonFeatures

e12cdd8

Merge branch 'jsonParsing' into jsonFeatures

4173a36

Merge branch 'jsonParsing' into jsonFeatures

bc4993d

Merge branch 'jsonParsing' into jsonFeatures

fb68b52

Merge branch 'jsonParsing' into jsonFeatures

7fd1ca6

Merge branch 'jsonParsing' into jsonFeatures

ef54c93

Merge branch 'jsonParsing' into jsonFeatures

4cf1724

Merge branch 'jsonParsing' into jsonFeatures

484522d

Merge branch 'jsonParsing' into jsonFeatures

9a79d59

Update downstream.

c3687ba

Merge branch 'jsonParsing' into jsonFeatures

432dad7

Merge branch 'jsonParsing' into jsonFeatures

0622e9a

Merge branch 'jsonParsing' into jsonFeatures

6607e08

Merge branch 'jsonParsing' into jsonFeatures

b60fa2b

Merge branch 'jsonParsing' into jsonFeatures

7aa255f

Merge branch 'regexAndJsonParsing' into regexAndJsonFeatures

47f244b

Merge branch 'jsonFeatures' into regexAndJsonFeatures

8e087e7

Merge branch 'regexFeatures' into regexAndJsonFeatures

6de24cd

CyrusNajmabadi added 10 commits January 29, 2022 01:21

Add vb tests

264d476

Add comments

43d35bc

NRT

bd9006b

Simplify

739da25

Simplify

3a7d4a9

Simplify

9fe4e37

Nameing

5598afc

Export with name

e50dd39

Fix test

6f23704

Always report earliest diagnostics

054768d

CyrusNajmabadi requested a review from davidwengier January 29, 2022 21:07

CyrusNajmabadi commented Jan 29, 2022

View reviewed changes

CyrusNajmabadi added 3 commits January 29, 2022 18:01

Update options search strings

3e126ff

Rename enum member

a89230f

Update comments

8f56e0a

runfoapp bot mentioned this pull request Jan 31, 2022

Test failure in Microsoft.CodeAnalysis.CSharp.Emit.UnitTests.dll #57560

Closed

rename strings

a328198

jasonmalinowski approved these changes Jan 31, 2022

View reviewed changes

CyrusNajmabadi enabled auto-merge February 1, 2022 05:59

runfoapp bot mentioned this pull request Feb 1, 2022

Timeout on Microsoft.VisualStudio.LanguageServices.CSharp.UnitTests.dll tests #55638

Open

Narrow the range of nodes to search when doign json brace matching

8a4af89

CyrusNajmabadi disabled auto-merge February 2, 2022 18:17

CyrusNajmabadi merged commit 29dfd6e into dotnet:main Feb 2, 2022

CyrusNajmabadi deleted the jsonFeatures branch February 2, 2022 18:17

ghost added this to the Next milestone Feb 2, 2022

RikkiGibson modified the milestones: Next, 17.2.P1 Feb 4, 2022

ryzngard added UX Review Not Required UX Review Not Required and removed Needs UX Triage labels Apr 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable feature lightup on json strings (classification, bracematching, diagnostics). #59034

Enable feature lightup on json strings (classification, bracematching, diagnostics). #59034

CyrusNajmabadi commented Jan 23, 2022 •

edited

CyrusNajmabadi Jan 29, 2022

jasonmalinowski left a comment

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

jasonmalinowski Jan 31, 2022

CyrusNajmabadi Feb 1, 2022

CyrusNajmabadi commented Feb 1, 2022

		@@ -13,7 +14,7 @@ public partial class CSharpJsonParserNstTests : CSharpJsonParserTests
		private void TestNST(

Enable feature lightup on json strings (classification, bracematching, diagnostics). #59034

Enable feature lightup on json strings (classification, bracematching, diagnostics). #59034

Conversation

CyrusNajmabadi commented Jan 23, 2022 • edited

Choose a reason for hiding this comment

jasonmalinowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CyrusNajmabadi commented Feb 1, 2022

CyrusNajmabadi commented Jan 23, 2022 •

edited