Allow plugins to change foreground text color through manually adding regions

[DanielRosenwasser]

The TypeScript Language Service has an API for getting "classification ranges" of a source file, which are intended to be used for colorization in editors.

A mapping of classifications to Sublime regions seems to be fairly trivial, and so it would appear that the Sublime API would make this easy; however, the foreground color of the text does not seem to change when using add_regions.

I'm not sure if this is an intentional limitation (using build 3083), but it would allow us to focus on other features for our Sublime plugin instead of maintaining a separate grammar file.

[FichteFoll]

Just for the record: There is a workaround for that, which is as follows (inspired by https://github.com/whitequark/rainbowth).

1. Read the color scheme file and determine background color.
2. Artificially determine a new background color that is (marginally) different from the original one, e.g. '#%06x' % max(1, (int(bg[1:], 16) - 1)), with #000001 for when background is black. Otherwise ST will ignore it and behave as if no background color was set.
3. Create a new scope in the color scheme with the above color as background and any color as foreground.
4. Either overwrite existing color scheme (bad) or save new color scheme to a different place and adjust color_scheme setting accordingly.
5. Use the new scope name with view.add_regions.

Just be careful with the color_scheme setting, i.e. consider the user changing it during runtime and restore the original one when the plugin is disabled/removed. I had to do similar but even more complex things here: https://github.com/SublimeText/InactivePanes/

[DanielRosenwasser]

Thanks for the response @FichteFoll! I think this is definitely something we may look into. We'll try to keep you posted on how well it works out with the work-around.

@steveluc @billti @mhegazy

[donysukardi]

Any update on this? I was looking into styling ANSI colors with regions just to realize that I couldn't get Sublime to set just the foreground color. It would end up being reversed as the background as mentioned.

[FichteFoll]

@donysukardi No update.

You might want to check out https://packagecontrol.io/packages/ANSIescape though.

[rwols]

I will need this for LSP's semantic highlighting. Or whatever alternative API.

[eerohele]

My use case is that I need to disable syntax highlighting for a set of regions. It's currently rather unwieldy.

[rchl]

One concrete example use case for LSP (apart from semantic highlighting) is fading the color of arguments/variables that are unused. So that means that it would ideally consider the current color of the region and dynamically create a less opaque variant.

[rwols]

To make things clear, I think there are two requests here:

1. Allow foreground color changes with some fixed color chosen by a plugin (rgb(100, 123, 240) for instance).
2. Allow foreground color changes with a scope (from https://www.sublimetext.com/docs/3/scope_naming.html for example)

I'm asking specifically for (2) with regards to semantic highlighting. I think @DanielRosenwasser is too. But yeah, maybe add_regions is too overused for this. SublimeHQ may want to think about allowing plugins to change the scopes dynamically but in another way. IMO it should be the same kind of set up as querying for completions: ST decides when it's a good time to ask all plugins if they know some semantic scopes for the view and can combine all results from all plugins this way. Such an API should be async and plugins should be allowed to return a promise object.

class MyListener(sublime_plugin.ViewEventListener):
    def on_query_highlighting_async(self, promise: HighlightingPromise) -> None:
        # this imaginary method doesn't exist (yet?)
        promise.set_result([...])

Option (1) would be more suited for things like:

• Marking related entities under the caret,
• Fading out unused arguments of a function,
• Marking deprecated functions in yellowish.

[wbond]

I would be strongly opposed to arbitrary colors since that will just result in plugins doing ugly things. Users pick colors schemes because they like the harmonious colors. Allowing plugin authors to drop rgb(255, 0, 0) into a pastel color scheme should be a crime. 😄 We have the concept of the "ish" colors, so any approach would ideally use those.

Having plugins create crappy, laggy syntax highlighting it something I'd also like to generally avoid.

For other uses, such as augmenting a syntax with things that are syntax-agnostic like todos, or dimming regions, I'd be more inclined to use scopes, or some sort of blending of scopes, as these should work more harmoniously with color schemes.

[rchl]

To make it clear, the intention of semantic highlighting from LSP is to augment the syntax, not replace it.
Of course, the augmentations will appear somewhat laggy but it will be a switch from syntax-highlighted code, not from plain text.

[wbond]

I'm aware, I'm just not keen on the editing experience of LSP where the color of tokens changes as you scroll - it feels very janky.

[rchl]

That wouldn't happen on scrolling as the semantic tokens are reported for the whole document at once on the initial open of the file. And from then the extra information are just some smaller deltas.

If you want more details then the proposal is here https://github.com/kittaakos/vscode-languageserver-node/blob/semantic-highlighting-proposal/protocol/src/protocol.semanticHighlighting.proposed.md

[wbond]

If LSP servers take hundreds or more milliseconds to respond (which they anecdotally do quite frequently), it will definitely be laggy/janky. No getting around that.

[wbond]

Either way, at the highest level, Sublime Text has always kept syntax highlighting distinct from annotating regions with extra info. Allowing regions to change the foreground (highlighting) feels like it would be confusing, and if the highlight changes after a delay introduced by a background server would introduce a laggy feel. My understanding is we've always explicitly prevented changing the foreground color because we think that is a bad idea.

I'm just not keen on making things fragmented and slower. At least right now it is reasonably easy to identify when plugins are messing with views since they can't affect the foreground and users can visually identify squiggly underlines, or boxes, and those make it pretty obvious a plugin is doing something incorrectly.

[rchl]

Personally I don't care about semantic highlighting much but I would love to be able to for example fade-out an argument or an import when the server reports it as being unused.

And yes, that would never be lag-free either but it wouldn't be worse than what is done currently for reporting diagnostic errors and triggering "squiggles" after the document is loaded or edited. Those appear after a delay due to a document needing to be validated first.

[FichteFoll]

I would love to be able to for example fade-out an argument or an import when the server reports it as being unused.

How about adding a new region marker type to dim or highlight text instead of allowing foreground color overrides? If this is what you intend to do, then I believe ST can provide you with the necessary tools instead of a swiss army knife.

[rchl]

There are different use cases. Dimming would certainly help with one of them.

[FichteFoll]

Maybe you could elaborate on those so we can better understand what kind of operation you want and whether or how ST could help with that.

[rchl]

Aren't previous comments descriptive enough? :)

The other use case is semantic highlighting which is basically about assigning arbitrary colors to arbitrary variables/properties. Basically like hashed highlighting (I think) but using server knowledge to make more accurate matches.

[eerohele]

For what its worth, here's a screenshot of my use case:

I'm working on a package that lets users evaluate bits of Clojure directly from within Sublime Text (like SublimeREPL, but bespoke for Clojure).

Evaluating (map inc (range 10)) yields a value ((1 2 3 4 5 6 7 8 9 10)). It should use syntax highlighting.

Evaluating (println "Hello, world!") yields a line in stdout (Hello, world!). It should not use syntax highlighting.

Evaluating (throw ,,,) yields an exception message. It should use the redish color of the current color scheme.

So, in brief, for my use case, it would be sufficient if I could:

• Disable syntax highlighting for a region one way or another (e.g. via setting the text color of a region to the foreground color of the current scheme)
• Set the text color of a region to one of the "ish" colors without affecting the background color

[rwols]

@FichteFoll

How about adding a new region marker type to dim or highlight text instead of allowing foreground color overrides? If this is what you intend to do, then I believe ST can provide you with the necessary tools instead of a swiss army knife.

I believe that would be a solution for things like:

• Marking related entities under the caret,
• Fading out unused arguments of a function,
• Marking deprecated functions in yellowish.

Perhaps this can be considered as well in sublimehq/Packages#1036
Related issue: sublimelsp/LSP#1227

Some names that I'm thinking of off the top of my head:

• Related entities under caret: markup.smart.related.text, markup.smart.related.read, markup.smart.related.write
• Fading out unused arguments: markup.smart.unnecessary
• Marking deprecated function calls: markup.smart.deprecated

(I don't really care what these scope names are as long as they end up in the scope name guide and end up in color schemes)

The above ideas/proposals do not solve the semantic highlighting API request.

---

@wbond

if the highlight changes after a delay introduced by a background server would introduce a laggy feel.

I don't know how you're coping with Visual Studio C++ then. That program also has delayed highlighting with respect to resolving function argument names, and resolving "active" preprocessor "#if #else" branches. These are just two examples that are impossible with the current state of the ST parser. Another example: multiple starting heredocs on a single line.

Allowing regions to change the foreground (highlighting) feels like it would be confusing

A fair point. Though with a pull model like my above proposal which I'll repeat here:

class MyListener(sublime_plugin.ViewEventListener):
    def on_query_additional_highlighting_async(self, promise: HighlightingPromise) -> None:
        # this imaginary method doesn't exist (yet?)
        promise.set_result([...])

The intention of this callback is clear and it is hard to abuse it. Because ST decides when to call it just like on_query_completions.

---

@eerohele

Your example use-case is again different from the above two use-cases. You want to take over the job of highlighting completely, but only for certain kind of "console" / "terminal" views. You seem to want to re-use the existing Clojure syntax to highlight only parts of the view.

Evaluating (println "Hello, world!") yields a line in stdout (Hello, world!). It should not use syntax highlighting.

You can use an invisible unicode character for that as the first character of the line and match it in an extended Clojure syntax that you create.

IMO it should be possible to do what you want with a custom .sublime-syntax today. Use a variety of invisible characters to differentiate between the different outputs.

That said, there are certainly use cases to completely "take over" highlighting for a view. But ST should restrict that possibility to special kinds of views that have some property set in their settings. Just thinking out loud: if the view.element() is of type "terminal" then it's allowed for a plugin to change the foreground colors.

---

In summary people are asking for three different things:

• More markup scopes
• Semantic highlighting on top of the regular ST highlighting
• Completely take over highlighting in certain output/terminal views

[eerohele]

Your example use-case is again different from the above two use-cases. You want to take over the job of highlighting completely, but only for certain kind of "console" / "terminal" views. You seem to want to re-use the existing Clojure syntax to highlight only parts of the view.

Right, more or less.

You can use an invisible unicode character for that as the first character of the line and match it in an extended Clojure syntax that you create.

IMO it should be possible to do what you want with a custom .sublime-syntax today. Use a variety of invisible characters to differentiate between the different outputs.

I considered that, but littering the view with invisible unicode characters has the potential to cause confusing situations when copy-pasting things from the view into other programs.

That said, there are certainly use cases to completely "take over" highlighting for a view. But ST should restrict that possibility to special kinds of views that have some property set in their settings. Just thinking out loud: if the view.element() is of type "terminal" then it's allowed for a plugin to change the foreground colors.

That would work for me. 👍

[pushqrdx]

My case is implementing Vim Quickscope like package in sublime, it highlights chars with shortest jump distances from the cursor. I tried all the currently available APIs and it's either hard to read or clashes with existing underline decorations resulting in my chars not being underlined.. or underlined and the line is in between other underlined chars making it impossible to see.

If you are concerned about performance checkout Vim's textprop, All to say, if properly implemented we can have both good performance and solve multiple useful use cases. Besides if a plugin behaves badly or abuses one feature, it's just like any other plugin that already abusing other Sublime APIs

[jason5122]

My use case for this would be implementing tree-sitter's "local variables" feature, which can draw all variables in a local scope the same color. This would require the use of changing foreground colors of the variables' text regions once parsed by tree-sitter.

[jpochyla]

It's sad to see the Sublime Text developers don't care about proper language experience, and are unwilling to provide basic features for the LSP to properly work, to be on par with other editors :/

[deathaxe]

Actually they do. It's just the strategy and values which differ. LSP can't provide speed and experience which can be considdered "working well enough" to satisfy least expectations.

With all the complains about claimed "poor" performance just because something takes 50ms longer than in ST3 here and there, it's pretty clear what happened once this feature was added and LSP starts highlighting things 3 seconds after manipulating tokens.

I personally find all the diagnostics quite annoying already as those often paint way too late and thus look off. I wouldn't want something like that interfering with syntax highlighting.

[rchl]

LSP can't provide speed and experience which can be considdered "working well enough" to satisfy least expectations.

I will refer to this very old comment a couple years later and try to make an argument again (if there isn't enough already)...

LSP semantic highlighting (as well as the foreground color hack) is here and it's not going anywhere so ST might as well make the experience better rather than being stubborn and having packages hack stuff.

Semantic highlighting provides a lot of value in terms of readability since it can color arguments and its uses consistently. For example with the params argument here:

This is pure ST without semantic highlighting:

Also it can differentiate types vs. instances as seen above with Typescript.

The problem with the current hack is that it makes the tokens that have semantic highlighting have non-transparent background so it can be problematic when other region decorations are applied also and it just looks weird with active lines.

[detly]

I encountered this last week trying to get ANSI highlighting to work a bit better. I was in fact hacking on the SublimeANSI package to fix up what I thought was an issue with that.

Just in case it helps anyone else, here's the code I used to demonstrate/test the behaviour. It's like a minimal repro for anyone who's stared at plugin code for long enough to question their grip on reality.

First, put this syntax highlighting scheme in Packages/User/test-background-thing.sublime-color-scheme:

{
    "author": "Test scheme",
    "globals": {
        "background": "#000000",
    },
    "name": "Test scheme",
    "rules": [
        {
            "background": "#000000",
            "foreground": "#c7c7c7",
            "scope": "test_scope_for_color"
        },
    ]
}

Then open a new file, and type and select some text. Open the console (usually ctrl+`) and paste this code:

test_view = sublime.active_window().active_view()
test_view.settings().set("color_scheme", "Packages/User/test-background-thing.sublime-color-scheme")
test_region = test_view.sel()
test_key = "test_region_key"
test_view.erase_regions(test_key)
test_view.add_regions(test_key, test_region, scope="test_scope_for_color", flags=sublime.DRAW_NO_OUTLINE | sublime.PERSISTENT)

The text hello will have its background changed to #c7c7c7. If, instead, you set the rule's background to #010000, it will have its foreground changed to #c7c7c7.

---

So, first request: if this is deliberate behaviour to dissuade plugins from arbitrarily changing colours outside of the scope/scheme abstractions, could it please be documented? (Maybe it is, but I looked at a few of the documentation pages relating to customisation and didn't see anything about it.) I wasted a few hours on this, because my starting point was not the simple scheme I posted above; that was reached by working backwards from the net effect of all the different sources of customisation. It's a surprising thing, and it wasn't remotely obvious that it wasn't simply me misconfiguring or misreading something. I sympathise with the intent (sort of), but just own it, document it, and you might save plugin developers some time.

Second request: can you clarify how this is meant to be done? I understand not wanting plugins to just inject #ff0000 literals into the view, but to loop back to this example:

You might want to check out https://packagecontrol.io/packages/ANSIescape though.

That package has the same problem! They have to tweak the background colour for coloured text to get around this. But as far as I can tell, they're doing the right thing... aren't they?

If you dig into how this plugin works, it assigns scopes to regions determined by ANSI escape codes. The regions are named for the combinations of the ANSI colour designations eg. red, red_light, red_light_bg_black. These scope names correspond to scopes used in a syntax scheme Packages/User/ANSIEscape/ansi.sublime-color-scheme. The user (or syntax scheme author) can customise that file to override things in a semantic way — except for this jarring, distracting maze of discolouration!

(You can disagree that red_light_bg_black is truly semantic, in general, but it's the maximum amount of semantic... ism...? that ANSI colour codes allow. We play the Unix legacy hand we're dealt.)

Third request is can we just have it please.