Find remaining performance bottlenecks for large files

[maxbrunsfeld]

This list is a work in progress.

• Memory Usage:
  • Marker snapshots (Speed up marker snapshotting by allowing markers to opt out of it text-buffer#71)
  • Tokenized lines eat tons of memory (Use 70% less memory for TokenizedLines by using a different representation #6757)
• Scrolling:
  • Garbage collection during DisplayBuffer::getDecorations (🐎 Avoid repeated array allocation in ::getDecorations #6694)
• Typing:
  • Generally
    • Updating symbol store index takes ~4ms
  • With an active find and replace query:
    • MarkerManager::handleBufferChange (Use MarkerIndex from text-document text-buffer#58)
    • Marker::notifyObservers (Optimize marker observation #7023)
    • Find-and-replace recreates all markers, not just those affected by change (Re-scan only changed region when buffer changes find-and-replace#394)
    • TextEditorPresenter::updateDecorations()
• Switching tabs:
  • Marker::destroy > TextEditorPresenter::observeDecoration > CompositeDisposable::remove. (Removing from a large CompositeDisposable is slow event-kit#7)
• Initially loading files
  • Screen lines are computed twice (Improve performance of opening files #6777)
  • We query for folds on every line when building the display buffer instead of batching (Only perform a single marker query for folds when updating screen lines #6879)
  • Creating placeholder tokenized lines blocks UI thread for multiple seconds
  • Creating initial symbol store index blocks UI thread for multiple seconds
• Moving the cursor
  • Bracket matcher takes a long time scanning for the matching bracket (Supply getRange to scanInRange iterator text-buffer#64)
  • Move forward by word hangs in huge files (Optimize buffer search text-buffer#72)
  • Bracket matcher still seems to be hanging in huge files (Optimize buffer search text-buffer#72)
• Rendering
  • Beyond about 700k lines, lines are no longer rendered (Render lines via tiles #6733)
  • Beyond about 700k lines, gutter is no longer rendered (Tiled Line Numbers #7101)

Other ideas:

• Disable tokenization above a certain size (Load large files with fewer features but much better performance #7130)
• Disable soft wrap above a certain size (Load large files with fewer features but much better performance #7130)

[alexandernst]

By "remaining" you mean aside from the text-document repo, right? (which on the other hand hasn't seen any activity in almost 2 weeks. Is that still a thing?)

[maxbrunsfeld]

@alexandernst we have decided to immediately extract some code and ideas from text-document into the relevant parts of Atom itself (see atom/text-buffer#58 and #6757), rather that continuing to develop text-document as a rewrite.

The plan is still to pursue the remaining ideas from text-document in some form, but we have made a tactical decision not to continue with the the text-document repository.

[alexandernst]

Hell, Atom's core-ideas-roadmap-guide-things change every day :)
I'm curious. is refactoring text-buffer better than a start-from-scratch text-document? And what made you change your mind and re-start text-buffer development and stop text-document?

[nathansobo]

@alexandernst Just a growing sense that we were taking on too big a piece. We still learned some really valuable things getting to think with a clean slate. But we decided it will be more effective to break them up and incrementally add them. We've already had a couple big wins performance-wise from doing this and I'd say about 85% of the code we wrote there is either in Atom or on its way to being in.

[as-cii]

Beyond about 700k lines, lines are no longer rendered (could probably be fixed with @as-cii's tiling)

A quick test confirms that tiling allows 1 million lines to be rendered correctly! 🎆 🎆 🎆

We still need tiling for the gutter, which stops displaying information as soon as we reach ~750k lines:

I am going to mark the related task as completed and add another one where we can keep track of gutter tiling. 🔥

[v3ss0n]

Thats awesome , i also made a list of test i've made against over here #6733 (comment)

[benogle]

Closing this as we will create specific issues for more improvements.

[lock]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you can still reproduce this issue in Safe Mode then please open a new issue and fill out the entire issue template to ensure that we have enough information to address your issue. Thanks!