VS2017 RC - Colorization causing slow editing in large files

[dsyme]

If you open a large file (e.g. TypeChecker.fs or CompileOps.fs) then character-by-character editing is noticably slow, much slower than in VS2012-15.

The underlying problem is large CPU time usage by the routine AddSyntacticClassificationsAsync. Here are the top entries for a profiling run for 20 seconds of editing (this was using a debug build of Visual F# Tools, but the same results should apply to a release build )

Function Name	Total CPU (ms)	Total CPU (%)	Self CPU (ms)	Self CPU (%)	Module
 - <StartupCode$FSharp-Editor>.$ColorizationService+Microsoft-CodeAnalysis-Editor-IEditorClassificationService-AddSyntacticClassificationsAsync@138::Invoke	14055	48.06%	0	0.00%	FSharp.Editor.dll
| - Microsoft.VisualStudio.FSharp.Editor.FSharpColorizationService::GetColorizationData	14011	47.91%	139	0.48%	FSharp.Editor.dll
|| - Microsoft.VisualStudio.FSharp.Editor.FSharpColorizationService::scanSourceLine	12726	43.52%	140	0.48%	FSharp.Editor.dll
||| - <StartupCode$FSharp-Editor>.$ColorizationService+scanAndColorNextToken@62::Invoke	11465	39.20%	219	0.75%	FSharp.Editor.dll
|||| - Microsoft.FSharp.Compiler.SourceCodeServices.FSharpLineTokenizer::ScanToken	10861	37.14%	527	1.80%	FSharp.LanguageService.Compiler.dll
||||| + <StartupCode$FSharp-LanguageService-Compiler>.$ServiceLexing+GetTokenWithPosition@569::Invoke	4487	15.34%	349	1.19%	FSharp.LanguageService.Compiler.dll
||||| + <StartupCode$FSharp-LanguageService-Compiler>.$ServiceLexing+FinalState@675::Invoke	2141	7.32%	11	0.04%	FSharp.LanguageService.Compiler.dll
||||| + Microsoft.FSharp.Compiler.SourceCodeServices.LexerStateEncoding::decodeLexInt	1025	3.50%	170	0.58%	FSharp.LanguageService.Compiler.dll

Repro steps

1. Open VisualFSharp.sln and go to COmpileOps.fs ot TYpeChecker,fs

2. Randomly delete characters and revert changes

3. Repeat this

• at the start of large files in projects
• at the end of large files in projects
• at the start of large scripts
• at the end of large scripts
• at the start of large files out of projects
• at the end of large files out of projects

Expected behavior

Fast editing

Actual behavior

Slow editing

Known workarounds

Use small code files :)

[dsyme]

@OmarTawfik I have a fix for this. Colorization is slow when the document text is changing in large documents. My understanding is that the table holding the FSharpColorization SourceTextData should be keyed by document or documentID (which doesn't change), rather than by document source text (which changes)

[dsyme]

Fix ready as part of #1829

[Pilchie]

FSharpColorization SourceTextData should be keyed by document or documentID (which doesn't change), rather than by document source text (which changes)

For reference Document does change - it is the immutable snapshot of the file. DocumentId is the stable identifier that lets you look up Documents in different snapshots.

[dsyme]

@Pilchie Thanks for reviewing these post-hoc. Can confirm we switched to DocumentID here https://github.com/Microsoft/visualfsharp/pull/1829/files#diff-de5bdafb119bed995860757094ee328eR61