Implementation of a semantic tokens reconciler

[rubenporras]

Implementation of a semantic tokens reconciler which supports only SemanticTokensFull.

The reconciler uses the theme defined for TM4E so that the same styles are used for the same token types.

In addition to the tokens defined for TM4E, also the deprecated modifier is supported by marking striking out the regions.

[mickaelistria]

Do you think you can add tests about it?
How does it coexist with syntax highlighting provided by TM4E "static" grammars?

[mickaelistria]

As mentioned earlier, do you think you can add some tests about this change?

[rubenporras]

Yes, I will look into writing some test. I have marked the PR as a draft in the meantime

[rubenporras]

@mickaelistria , I must admit I am clueless about how to test this reconciler. I have seen that eclipse/tm4e#33 has been open since long. Can we actually write a test for a reconciler?

[rubenporras]

@mickaelistria , the commit cannot be built because
[ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-repository-plugin:3.0.0:assemble-repository (default-assemble-repository) on project repository: Execution default-assemble-repository of goal org.eclipse.tycho:tycho-p2-repository-plugin:3.0.0:assemble-repository failed: An API incompatibility was encountered while executing org.eclipse.tycho:tycho-p2-repository-plugin:3.0.0:assemble-repository: java.lang.NoSuchMethodError: 'void org.eclipse.equinox.internal.p2.repository.helpers.ChecksumProducer.(java.lang.String, java.lang.String, java.lang.String)'

do you know what that is?

Thanks

[mickaelistria]

Please try using latest Tycho release (this can fit in a dedicated PR)

[rubenporras]

Please try using latest Tycho release (this can fit in a dedicated PR)

Thanks, that fixed the issue.

[rubenporras]

Hi @mickaelistria , I have refactored the code so that I could write tests of most parts involved in the reconciler. The code is not building currently but that looks like a problem with the build system, and not the PR.

I have also tested this code (a very similar one because I did some minor refactoring today when writing the unit tests) in our setup and it works really nice.

So I am putting it again for review, if you think more tests are needed, I am happy to do them if they can be done without testing a full fledge editor setup, I am still not sure how to do that.

Regards

[rubenporras]

@mickaelistria , I have now also written a unit test that shows end-to-end that opening a file in the editor triggers the semantic reconciler and that this one updates the styles in the buffer to the values which are set in the mocked server.

I think the code is good, but just in case I have also added a store configuration to disable the reconciler as a workaround, if it would prove to be faulty in such way and makes working with the editor hard, until we can fix it.

Is it ok with you to merge it now? I believe it is a very nice feature.

[rubenporras]

testCheckIfOtherAnnotationsRemains failed, but that fails constantly in the jobs, I am sure it is not this PR's fault :)

[mickaelistria]

Thanks!

[laeubi]

@mickaelistria @rubenporras I'm currently try to use this but nothing seem to visually change (but I also don't get an error), my method is called and I'm currently just return a single dummy like this:

SemanticTokens tokens = new SemanticTokens(new ArrayList<>());
data.add(1); // line 1
data.add(1); // startChar 1
data.add(5); // length 5
data.add(0); // token 0
data.add(0); // no modifiers
return tokens;

I use version:
3116 ACTIVE org.eclipse.lsp4e_0.18.0.202310181943
3117 RESOLVED org.eclipse.lsp4j_0.21.1.v20230829-0012

code: eclipse-pde/eclipse.pde#821

So I seem to miss something ... or is it required to configure something here?

[deanmaster]

hello i'm testing LSP within Eclipse. The semantic highlighting in VSCode is working and lsp4j already working on this topic.
I see the PR is merged ? Can I enable it somehow in Eclipse ? It has lsp4e installed already.

[laeubi]

hello i'm testing LSP within Eclipse. The semantic highlighting in VSCode is working and lsp4j already working on this topic. I see the PR is merged ? Can I enable it somehow in Eclipse ? It has lsp4e installed already.

I can't make this work in Eclipse either so there so it seems I must be something I did wrong, it is not implemented "visually" or some magic setting is required :-)

[rubenporras]

Hello,

In addition to having TM4E installed, it should be configured for the editor where semantic highlight is needed. At the very least the stylesheet it uses must contain an style for the token the language server returns.

Have you done that?

Regards

[laeubi]

Have you done that?

Likely not as it is nowhere documented (or I didn't found that) :-)

At the very least the stylesheet it uses must contain an style for the token the language server returns

I wounder if there could be some defaults? If I read it correctly here there are some default tokens so at least for them there should be some "fallback" if nothing else is defined?

[rubenporras]

Have you done that?

Likely not as it is nowhere documented (or I didn't found that) :-)

Fair enough. Where would you expect such documentation?

At the very least the stylesheet it uses must contain an style for the token the language server returns

I wounder if there could be some defaults? If I read it correctly here there are some default tokens so at least for them there should be some "fallback" if nothing else is defined?

I would not know what that would be. How can we know the color (or other style) of a token?

[laeubi]

Fair enough. Where would you expect such documentation?

Good question I searched for LSP4J SemanticTokens and this issue was the only real hit, then I used this PR code and some specification reads to make something that at least make not the server / client complain ...

I would not know what that would be. How can we know the color (or other style) of a token?

I think it must not be perfect, but for example make some of the bold, others italic or even some random colors what seems suitable to have a different visual appearance. Right now it just stays all the same so I don't even know if my above example has ever worked.