Limited tracing instrumentation makes profiling and performance analysis harder

[Draggu]

It seems that the Expert currently has limited tracing support, which makes it challenging to perform detailed profiling and performance analysis.
This is related to #235.

Having additional tracing instrumentation would make it easier to understand where time is being spent, for example:

• processing individual LSP requests (such as completion or hover)
• indexing the project
• building the engine

One possible approach could be to make this tracing optional, to avoid overhead in normal usage.
It could be enabled via:

• ENV var like EXPERT_PROFILE
• config checkbox in editor like Enable profiling
• cli argument like --profile

This would produce .expert/profile_<timestamp>.json file.

For reference, below is an example screenshot of a profile file generated by cairols and viewed in perfetto, which illustrates the kind of insights that tracing can provide.

Open question is what would be best for row descriptions?

[doorgan]

Adding this would be really nice, hooking into a running expert and tracing things is a big pain point right now, and we have a bunch of performance related issues on large projects that would benefit greatly from this

config checkbox in editor like Enable profiling

I leaning towards this

[mhanberg]

In Next LS, I had open telemetry working at one point. GenLSP emits telemetry events, so can be used.

Also to note, just because its in your example, things like building the engine or compiling users code are not really going to be helpful to profile, as they only will go as fast as the compiler itself, which is independent to the project.

[mkaput]

Also to note, just because its in your example, things like building the engine or compiling users code are not really going to be helpful to profile, as they only will go as fast as the compiler itself, which is independent to the project.

You will still learn how long it took and how much work was stuck waiting for this to finish