C# - font lock breaks after an interpolated string or terminating simple string

[sebasmonia]

After an interpolated string font lock break until I re-open the buffer.

var a = 25;
var b = $"This is value of the variable: {a}";
// Everything from here on is not correctly font-locked until I reopen the buffer. 
// Can't indent code correctly either.

[josteink]

Hey there and thanks for the report.

Sorry about the delay in looking into this, but unfortunately I'm not able to reproduce the issue:

Could you be more specific about which version of Emacs and what version of csharp-mode you are using?

[sebasmonia]

I'm using Emacs 27 and latest csharp-mode from MELPA.
Here you can see it all seems ok:

After adding ; and new line then:

The colors fix after I start typing below, however if I add something, the new line doesn't indent, and when I hit TAB nothing happens:

But after closing and reopening the buffer, I can go back to that line and it indents just fine.

The whole Program.cs code is:

using System;

namespace testapp
{
    class Program
    {
        private static int a_number;

        static void Main(string[] args)
        {

            System.Diagnostics.Debugger.Launch();
            System.Diagnostics.Debugger.Break();
            Console.WriteLine("Hello World!");
            System.Diagnostics.Debugger.Break();
            Console.WriteLine("Bye  World!");
        }
    }
}

UPDATE: I missed the leading $ when taking the screenshots but it doesn't change the outcome of the test!

[josteink]

Thanks for very much taking the time to document your issue this thoroughly. And sorry for the slow response. Been down with some illness.

Sadly though, looking at it now, I still can't reproduce the issue. 😞

Have you tried deleting csharp-mode from your ~/.emacs.d/elpa-folder and then reinstalling?

csharp-mode depends on cc-mode from Emacs core at compile time, so sometimes upgrading Emacs can cause issues with csharp-mode, unless it gets reinstalled and recompiled against the newer cc-mode-version.

[phoenixanimations]

@josteink I can confirm this breaks for me on emacs 27. Another example:
emacs -Q

(package-initialize)
(require 'csharp-mode)

C-x b csharp-example
M-x csharp-mode

using System;
using System.Collections.Generic;

public class Csharp
{
    static public void Main ()
    {
	String example = "another line of text";
	Console.WriteLine(example + "and some more");
    }
}

1. Goto the end of String example = "another line of text";.
2. Press return.
3. Type (do not copy and paste) String foo = "bar";
4. Press return.

using System;
using System.Collections.Generic;

public class Csharp
{
    static public void Main ()
    {
	String example = "another line of text";
	String foo = "bar";
	
	Console.WriteLine(example + "and some more");
    }
}

Everything from Console.Writeline down is messed up. From there:

1. delete the first " in "bar".
2. reenter a ".
3. go to the end of the line and press return.

Tab indentation is now messed up as well. I reinstalled csharp-mode I think something in cc-mode broke csharp. I'll see if sometime today I can investigate further, this didn't happen to me on Emacs 26.

GNU Emacs 27.0.50

[josteink]

Thanks for the reports, guys.

Please note “Emacs 27” just means a currently unreleased version built from source. It doesn’t say what source revision.

Are you guys both on git master? That’s what I built when testing and couldn’t reproduce.

[sebasmonia]

Confirmed it happens after deleting C# mode.

In my case I built from source last week or so:
GNU Emacs 27.0.50 (build 1, x86_64-w64-mingw32) of 2019-11-04

[phoenixanimations]

@josteink I use emacs-plus to build: brew install emacs-plus --with-ctags --with-jansson --HEAD my emacs build is:
GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin17.7.0, NS appkit-1561.60 Version 10.13.6 (Build 17G65)) of 2019-11-12.

Oh the variable c-version is 5.34.

[josteink]

I really can't reproduce this in any respect.

Would one of you be able to setup something super-simple, like a step-by-step recipie, dockerfile or anything which can reproduce this error in a clean environment?

If I can't reproduce, I will definitely not be able to debug this or track down what causes it.

If you guys feel like contributing (since you can reproduce and everything!), you could check out the source locally and try to check out older revisions to see if this ever working properly.

If it used to work properly and stopped working at some point, you can use a git bisect to track down where the error was introduced.

Does that sound like something you guys could do?

[sebasmonia]

Will give it a try comparing with older versions, when I get a chance. Thanks for looking into this!

[snippins]

@sebasmonia I have the same problem, a workaround without having to close and reopen the buffer is to put the cursor behind the closing quote, then press spacebar (insert a space) and backspace. After that everything is font-locked correctly again.

[phoenixanimations]

Additionally another workaround is doing M-x csharp-mode. When I'm able to I'll try to reproduce it in a clean environment, or at the very least diff between the old/new emacs' cc-mode files.

[josteink]

When I'm able to I'll try to reproduce it in a clean environment, or at the very least diff between the old/new emacs' cc-mode files.

If you could do that, that would be awesome. I really haven't found any way to reproduce this issue myself.

[ronnieholm]

With Emacs compiled from master and the latest csharp-mode installed from Melpa, font locking still has issues.

Starting with this code:

All it takes is removing and adding the last " characters of the string and code after that position is incorrectly font locked:

According to font-lock-studio, the "if" token is correctly identified as font-lock-keyword-face, just like before the edit, and single stepping with font-lock-studio restores proper font locking to the buffer:

"\\<\\(\\(?:a\\(?:bstract\\|dd\\|s\\(?:cending\\|ync\\)?\\|wait\\)\\|b\\(?:ase\\|reak\\)\\|c\\(?:a\\(?:se\\|tch\\)\\|hecked\\|lass\\|on\\(?:st\\|tinue\\)\\)\\|d\\(?:e\\(?:fault\\|legate\\|scending\\)\\|o\\)\\|e\\(?:lse\\|num\\|vent\\|x\\(?:plicit\\|te\\(?:nds\\|rn\\)\\)\\)\\|f\\(?:i\\(?:nally\\|xed\\)\\|or\\(?:each\\)?\\|rom\\)\\|g\\(?:et\\|oto\\)\\|i\\(?:mp\\(?:l\\(?:ements\\|icit\\)\\|ort\\)\\|nter\\(?:face\\|nal\\)\\|[fns]\\)\\|l\\(?:et\\|ock\\)\\|n\\(?:amespace\\|ew\\)\\|o\\(?:perator\\|rderby\\|ut\\|verride\\)\\|p\\(?:a\\(?:ckage\\|r\\(?:ams\\|tial\\)\\)\\|r\\(?:ivate\\|otected\\)\\|ublic\\)\\|re\\(?:adonly\\|f\\|move\\|turn\\)\\|s\\(?:e\\(?:aled\\|\\(?:lec\\)?t\\)\\|izeof\\|t\\(?:atic\\|ruct\\)\\|uper\\|witch\\)\\|t\\(?:h\\(?:is\\|rows?\\)\\|ry\\|ypeof\\)\\|u\\(?:n\\(?:checked\\|safe\\)\\|sing\\)\\|v\\(?:ar\\|irtual\\|olatile\\)\\|wh\\(?:\\(?:er\\|il\\)e\\)\\|yield\\)\\)\\([^[:alnum:]_]\\|$\\)"
  (1 font-lock-keyword-face)

Without running font-lock-studio, as with the previous reports, if I switch the buffer to another major mode and then back to csharp-mode, correct font locking is restored. As if resetting the mode resets stale state.

I've made the same edit to the C# code but with the buffer in C or C++ mode. Font locking remains correct.

Any idea how to narrow down the issue further? To me nothings stands out in csharp-mode.el.

[josteink]

Thanks @ronnieholm.

I've tested and can reproduce with the minimal sample provided here. Clearly a bug.

I know in the past we've had issues with initial fontification not always being consistent with how fontification is done with live-editing, but I don't think I've ever seen a case this obvious.

Unfortunately font-lock-studio only assists debugging initial fontification.

I think this may be a case of Emacs gradually deprecating the support for a syntax-propertize function, used in many major-modes, csharp-mode included.

I'm not sure about the best way forwards , but knowing which single edit triggers the bug, it should be possible to instrument/edebug the syntax-propertize-function (in our case csharp-mode-syntax-propertize-function) before initiating that one keystroke which triggers the bug and then:

1. see if it gets triggered, and
2. if or where it goes wrong

I don't have much spare time to dig into this right now, so any help is appreciated.