Large file support #307

Closed
watsonian opened this Issue Feb 25, 2013 · 140 comments

Comments

Projects
None yet
@watsonian
Contributor

watsonian commented Feb 25, 2013

So, I use my text editor for all kinds of things (writing code, reading readmes, editing dotfiles, etc). One of the things I do most often during any given day is open up large log files to troubleshoot Enterprise problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately -- the file selector didn't change and the entire window went completely white 4 seconds after I tried viewing the file. I could still close the top level window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice progress indicator while it was loading:

Screen Shot 2013-02-24 at 4 50 56 PM

Once it loaded it was as responsive as any other file I load (scrolled fast, normal text highlight performance, etc). It would be nice if we set this as a baseline for expected behavior (progress bar + responsive after loading).

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Feb 25, 2013

Contributor

Yeah, we currently do a blocking read to load files. We need to switch to
something event-driven. I also wonder if we can load the file progressively
off the disk as needed so we don't have to seek the whole thing into
memory. But that's super-advanced. Just avoiding blocking the UI thread
would be a good start. Yet another thing I'd like to do with Node.

On Sun, Feb 24, 2013 at 5:54 PM, Joel Watson notifications@github.comwrote:

So, I use my text editor for all kinds of things (writing code, reading
readmes, editing dotfiles, etc). One of the things I do most often during
any given day is open up large log files to troubleshoot Enterprise
problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately --
the file selector didn't change and the entire window went completely white
4 seconds after I tried viewing the file. I could still close the top level
window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice
progress indicator while it was loading:

[image: Screen Shot 2013-02-24 at 4 50 56 PM]https://f.cloud.github.com/assets/244/190379/c0abc27e-7ee5-11e2-8ec4-4205cdee0ad0.png

Once it loaded it was as responsive as any other file I load (scrolled
fast, normal text highlight performance, etc). It would be nice if we set
this as a baseline for expected behavior (progress bar + responsive after
loading).


Reply to this email directly or view it on GitHubhttps://github.com/github/atom/issues/307.

Contributor

nathansobo commented Feb 25, 2013

Yeah, we currently do a blocking read to load files. We need to switch to
something event-driven. I also wonder if we can load the file progressively
off the disk as needed so we don't have to seek the whole thing into
memory. But that's super-advanced. Just avoiding blocking the UI thread
would be a good start. Yet another thing I'd like to do with Node.

On Sun, Feb 24, 2013 at 5:54 PM, Joel Watson notifications@github.comwrote:

So, I use my text editor for all kinds of things (writing code, reading
readmes, editing dotfiles, etc). One of the things I do most often during
any given day is open up large log files to troubleshoot Enterprise
problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately --
the file selector didn't change and the entire window went completely white
4 seconds after I tried viewing the file. I could still close the top level
window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice
progress indicator while it was loading:

[image: Screen Shot 2013-02-24 at 4 50 56 PM]https://f.cloud.github.com/assets/244/190379/c0abc27e-7ee5-11e2-8ec4-4205cdee0ad0.png

Once it loaded it was as responsive as any other file I load (scrolled
fast, normal text highlight performance, etc). It would be nice if we set
this as a baseline for expected behavior (progress bar + responsive after
loading).


Reply to this email directly or view it on GitHubhttps://github.com/github/atom/issues/307.

@watsonian

This comment has been minimized.

Show comment
Hide comment
@watsonian

watsonian Mar 2, 2013

Contributor

I just tried opening a 124MB JSON file directly from the CLI and got this:

RangeError: Maximum call stack size exceeded
    at BufferChangeOperation.module.exports.BufferChangeOperation.changeBuffer (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:99:17)
    at BufferChangeOperation.module.exports.BufferChangeOperation.do (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:37:19)
    at Buffer.module.exports.Buffer.pushOperation (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:348:31)
    at Buffer.module.exports.Buffer.change (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:314:20)
    at Buffer.module.exports.Buffer.setText (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:174:19)
    at Buffer.module.exports.Buffer.reload (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:126:12)
    at new Buffer (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:65:14)
    at Project.module.exports.Project.buildBuffer (/Applications/Atom.app/Contents/Resources/src/app/project.js:270:16)
    at Project.module.exports.Project.bufferForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:259:33)
    at Project.module.exports.Project.buildEditSessionForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:180:41) index.html:23
window.onload
Contributor

watsonian commented Mar 2, 2013

I just tried opening a 124MB JSON file directly from the CLI and got this:

RangeError: Maximum call stack size exceeded
    at BufferChangeOperation.module.exports.BufferChangeOperation.changeBuffer (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:99:17)
    at BufferChangeOperation.module.exports.BufferChangeOperation.do (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:37:19)
    at Buffer.module.exports.Buffer.pushOperation (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:348:31)
    at Buffer.module.exports.Buffer.change (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:314:20)
    at Buffer.module.exports.Buffer.setText (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:174:19)
    at Buffer.module.exports.Buffer.reload (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:126:12)
    at new Buffer (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:65:14)
    at Project.module.exports.Project.buildBuffer (/Applications/Atom.app/Contents/Resources/src/app/project.js:270:16)
    at Project.module.exports.Project.bufferForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:259:33)
    at Project.module.exports.Project.buildEditSessionForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:180:41) index.html:23
window.onload
@probablycorey

This comment has been minimized.

Show comment
Hide comment
@probablycorey

probablycorey Oct 17, 2013

Member

This moved forward after #939. But there is still some telepath work to get the rest working. Adding single-user mode to telepath that eliminates the need for each character to have a location may be a solution.

Member

probablycorey commented Oct 17, 2013

This moved forward after #939. But there is still some telepath work to get the rest working. Adding single-user mode to telepath that eliminates the need for each character to have a location may be a solution.

@stuart-warren

This comment has been minimized.

Show comment
Hide comment
@stuart-warren

stuart-warren Jun 19, 2014

I'm failing to open up a 1.4MB HTML file with the same RangeError: Maximum call stack exceeded

Running Version 0.105
Has anything happened on this issue since October?

Full error:

Window load time: 761ms 
/Applications/Atom.app/Contents/Resources/app/src/window-bootstrap.js:18
RangeError: Maximum call stack size exceeded
  at DisplayBuffer.module.exports.DisplayBuffer.updateScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:1246:17)
  at DisplayBuffer.module.exports.DisplayBuffer.updateAllScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:179:19)
  at new DisplayBuffer (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:94:12)
  at new Editor (/Applications/Atom.app/Contents/Resources/app/src/editor.js:76:30)
  at Project.module.exports.Project.buildEditorForBuffer (/Applications/Atom.app/Contents/Resources/app/src/project.js:430:16)
  at /Applications/Atom.app/Contents/Resources/app/src/project.js:192:24
  at _fulfilled (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:787:54)
  at self.promiseDispatch.done (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:816:30)
  at Promise.promise.promiseDispatch (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:749:13)
  at /Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:557:44
  at flush (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:108:17)
  at process._tickCallback (node.js:605:11)

Thanks

I'm failing to open up a 1.4MB HTML file with the same RangeError: Maximum call stack exceeded

Running Version 0.105
Has anything happened on this issue since October?

Full error:

Window load time: 761ms 
/Applications/Atom.app/Contents/Resources/app/src/window-bootstrap.js:18
RangeError: Maximum call stack size exceeded
  at DisplayBuffer.module.exports.DisplayBuffer.updateScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:1246:17)
  at DisplayBuffer.module.exports.DisplayBuffer.updateAllScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:179:19)
  at new DisplayBuffer (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:94:12)
  at new Editor (/Applications/Atom.app/Contents/Resources/app/src/editor.js:76:30)
  at Project.module.exports.Project.buildEditorForBuffer (/Applications/Atom.app/Contents/Resources/app/src/project.js:430:16)
  at /Applications/Atom.app/Contents/Resources/app/src/project.js:192:24
  at _fulfilled (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:787:54)
  at self.promiseDispatch.done (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:816:30)
  at Promise.promise.promiseDispatch (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:749:13)
  at /Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:557:44
  at flush (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:108:17)
  at process._tickCallback (node.js:605:11)

Thanks

@izuzak izuzak added the performance label Jul 9, 2014

@probablycorey

This comment has been minimized.

Show comment
Hide comment
@probablycorey

probablycorey Jul 9, 2014

Member

Opening files larger than 2MB is an edge case that most users don't run into, but I wanted to investigate this.

How to create a large file

Download this file and run it generate-text five-megabytes-of-words.txt 5

Good news

Atom handles 5MB without changing anything.

Bad news

Atom hangs when opening a a file >10MB.

Atom eats up ~300MB or RAM when opening large files.

activity monitor all processes

Member

probablycorey commented Jul 9, 2014

Opening files larger than 2MB is an edge case that most users don't run into, but I wanted to investigate this.

How to create a large file

Download this file and run it generate-text five-megabytes-of-words.txt 5

Good news

Atom handles 5MB without changing anything.

Bad news

Atom hangs when opening a a file >10MB.

Atom eats up ~300MB or RAM when opening large files.

activity monitor all processes

@probablycorey

This comment has been minimized.

Show comment
Hide comment
@probablycorey

probablycorey Jul 10, 2014

Member

I modified window-bootstrap to test out some memory assumptions. Here is the memory footprint of Atom using a file size of 40MB

Baseline (window-bootstrap.coffee does nothing): 50MB
Baseline + file via fs: 96MB
Baseline + file via File: 181MB
Baseline + TextBuffer: 538MB

Member

probablycorey commented Jul 10, 2014

I modified window-bootstrap to test out some memory assumptions. Here is the memory footprint of Atom using a file size of 40MB

Baseline (window-bootstrap.coffee does nothing): 50MB
Baseline + file via fs: 96MB
Baseline + file via File: 181MB
Baseline + TextBuffer: 538MB

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Jul 10, 2014

Contributor

I'm thinking we probably need to store the TextBuffer data off the v8 heap using a C++ extension, otherwise it hoses GC performance.

Contributor

nathansobo commented Jul 10, 2014

I'm thinking we probably need to store the TextBuffer data off the v8 heap using a C++ extension, otherwise it hoses GC performance.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Jul 10, 2014

Contributor

But there's also a ton of other data such as tokenization etc.

Contributor

nathansobo commented Jul 10, 2014

But there's also a ton of other data such as tokenization etc.

@zcbenz

This comment has been minimized.

Show comment
Hide comment
@zcbenz

zcbenz Jul 11, 2014

Member

Node's Buffer stores data in C++ heap. But if we want to support really big files we have to avoid reading the whole file in memory.

Member

zcbenz commented Jul 11, 2014

Node's Buffer stores data in C++ heap. But if we want to support really big files we have to avoid reading the whole file in memory.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Jul 11, 2014

Contributor

Interested in any ideas you have about the design of that @zcbenz. The hardest part might be a bunch of APIs might need to go async to support loading more of the file for certain operations, which could make for more awkward scripting.

Contributor

nathansobo commented Jul 11, 2014

Interested in any ideas you have about the design of that @zcbenz. The hardest part might be a bunch of APIs might need to go async to support loading more of the file for certain operations, which could make for more awkward scripting.

@zcbenz

This comment has been minimized.

Show comment
Hide comment
@zcbenz

zcbenz Jul 11, 2014

Member

The basic idea is to only draw the file on needed, but it will need a good caching strategy to make things smooth since drawing file would rely on disk IO, many native apps use memory-mapped files to simplify it, a rough design is:

when opening a large file

  1. quickly scans the whole file to count there are how many lines and how long is each line.
  2. the editor view computes and draws a virtual scroller bar.
  3. the editor view decides which part of file should be read into memory according to current position of scroller.
  4. the editor view reads that part of the file and draws them.

when user scrolls the editor view
repeat 3 - 4

when the editor view is resized
repeat 2 - 4

when the file is changed outside
repeat 1 - 4

  • step 1 and step4 could be asynchronous.
  • the step 1 will need to take the responsibility of tokenization.
  • when file is really large that even scanning it takes time we may give up scanning and tokenizing it, the editor view can just show scroller bar with a minimal drag handle, and when user drags the scroller the editor view just roughly computes a start position to read file.
Member

zcbenz commented Jul 11, 2014

The basic idea is to only draw the file on needed, but it will need a good caching strategy to make things smooth since drawing file would rely on disk IO, many native apps use memory-mapped files to simplify it, a rough design is:

when opening a large file

  1. quickly scans the whole file to count there are how many lines and how long is each line.
  2. the editor view computes and draws a virtual scroller bar.
  3. the editor view decides which part of file should be read into memory according to current position of scroller.
  4. the editor view reads that part of the file and draws them.

when user scrolls the editor view
repeat 3 - 4

when the editor view is resized
repeat 2 - 4

when the file is changed outside
repeat 1 - 4

  • step 1 and step4 could be asynchronous.
  • the step 1 will need to take the responsibility of tokenization.
  • when file is really large that even scanning it takes time we may give up scanning and tokenizing it, the editor view can just show scroller bar with a minimal drag handle, and when user drags the scroller the editor view just roughly computes a start position to read file.
@i4004

This comment has been minimized.

Show comment
Hide comment
@i4004

i4004 Jul 25, 2014

I tried to open ~8gb text file, but nothing just happened :)

i4004 commented Jul 25, 2014

I tried to open ~8gb text file, but nothing just happened :)

@batjko

This comment has been minimized.

Show comment
Hide comment
@batjko

batjko Jul 25, 2014

Contributor

I'm not sure how they do it, but Baretail is a pretty impressive log file viewer in that regard.
It doesn't care how big your file is and only seems to load portions of it as needed, with very fast seek and search across the entire monster file though, including life refresh of the bottom lines (useful for log files of course) - while keeping the memory footprint quite small.

I assume that's rather difficult to implement in Atom, maybe too much effort for the rare use case?

Contributor

batjko commented Jul 25, 2014

I'm not sure how they do it, but Baretail is a pretty impressive log file viewer in that regard.
It doesn't care how big your file is and only seems to load portions of it as needed, with very fast seek and search across the entire monster file though, including life refresh of the bottom lines (useful for log files of course) - while keeping the memory footprint quite small.

I assume that's rather difficult to implement in Atom, maybe too much effort for the rare use case?

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Jul 25, 2014

Contributor

@batjko Thanks for the tip. Long term we definitely want to support files of arbitrary size, but an approach that doesn't load the whole file into memory is going to force us to redesign some of our synchronous APIs to be asynchronous due to the single-threaded nature of JavaScript. We made a pragmatic decision early on to perform editor state manipulations synchronously for a more convenient scripting experience, but we may need to revisit that decision.

Contributor

nathansobo commented Jul 25, 2014

@batjko Thanks for the tip. Long term we definitely want to support files of arbitrary size, but an approach that doesn't load the whole file into memory is going to force us to redesign some of our synchronous APIs to be asynchronous due to the single-threaded nature of JavaScript. We made a pragmatic decision early on to perform editor state manipulations synchronously for a more convenient scripting experience, but we may need to revisit that decision.

@batjko

This comment has been minimized.

Show comment
Hide comment
@batjko

batjko Jul 25, 2014

Contributor

@nathansobo Personally, I'm happy with it as it is. I have no use for extreme file sizes in an editor.

I think the huge log file scenario is not really one for editors, but rather for viewers/greppers or whatever you might call that category.

Contributor

batjko commented Jul 25, 2014

@nathansobo Personally, I'm happy with it as it is. I have no use for extreme file sizes in an editor.

I think the huge log file scenario is not really one for editors, but rather for viewers/greppers or whatever you might call that category.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Jul 25, 2014

Contributor

Yeah, I pretty much agree. I'm confident we'll get there eventually, but there's a lot of other things we consider more important right now.

Contributor

nathansobo commented Jul 25, 2014

Yeah, I pretty much agree. I'm confident we'll get there eventually, but there's a lot of other things we consider more important right now.

@simonzack

This comment has been minimized.

Show comment
Hide comment
@simonzack

simonzack Aug 3, 2014

@batjko I don't think this is a rare use case, given the amount of text editors which advertise this as a feature, and the comments so far on this issue. Log viewers and greppers are nice, but it is nicer to view and edit text in an editor. After all isn't that what a text editor is for.

@batjko I don't think this is a rare use case, given the amount of text editors which advertise this as a feature, and the comments so far on this issue. Log viewers and greppers are nice, but it is nicer to view and edit text in an editor. After all isn't that what a text editor is for.

@detly

This comment has been minimized.

Show comment
Hide comment
@detly

detly Aug 6, 2014

The other aspect to this is the failure mode while it isn't supported. If viewing and editing large text files isn't going to work for the time being, it might still be nice to have a better failure mode than locking up and forcing the user to kill Atom.

detly commented Aug 6, 2014

The other aspect to this is the failure mode while it isn't supported. If viewing and editing large text files isn't going to work for the time being, it might still be nice to have a better failure mode than locking up and forcing the user to kill Atom.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Aug 6, 2014

Contributor

The failure mode is currently an error message displayed in the console for files exceeding 2mb, which should leave Atom in a usable state. Are you talking about large files that are < 2mb?

Contributor

nathansobo commented Aug 6, 2014

The failure mode is currently an error message displayed in the console for files exceeding 2mb, which should leave Atom in a usable state. Are you talking about large files that are < 2mb?

@detly

This comment has been minimized.

Show comment
Hide comment
@detly

detly Aug 7, 2014

Yes. This is a 1.4M (generated) Python file.

detly commented Aug 7, 2014

Yes. This is a 1.4M (generated) Python file.

@izuzak

This comment has been minimized.

Show comment
Hide comment
@izuzak

izuzak Aug 12, 2014

Member

Yes. This is a 1.4M (generated) Python file.

@detly Can you share the file for which this is happening? Also, does this happen when you run atom in safe mode? Anything else special about the file (e.g. a really long line)?

Member

izuzak commented Aug 12, 2014

Yes. This is a 1.4M (generated) Python file.

@detly Can you share the file for which this is happening? Also, does this happen when you run atom in safe mode? Anything else special about the file (e.g. a really long line)?

@izuzak

This comment has been minimized.

Show comment
Hide comment
@izuzak

izuzak Aug 12, 2014

Member

Can you share the file for which this is happening?

Actually, I just tried this myself and Atom was slowing down drastically on large files like that. I thought that wasn't happening on files of that size -- I thought I tried that a while ago but I might be wrong. Sorry about that!

Member

izuzak commented Aug 12, 2014

Can you share the file for which this is happening?

Actually, I just tried this myself and Atom was slowing down drastically on large files like that. I thought that wasn't happening on files of that size -- I thought I tried that a while ago but I might be wrong. Sorry about that!

@wb-towa

This comment has been minimized.

Show comment
Hide comment
@wb-towa

wb-towa Sep 10, 2014

I can't believe this doesn't seem to have a very high priority given there have been numerous releases of atom and it still can not match the performance of editors from 20 or more years ago.

If you're going to reinvent the wheel it shouldn't be a step backwards.

wb-towa commented Sep 10, 2014

I can't believe this doesn't seem to have a very high priority given there have been numerous releases of atom and it still can not match the performance of editors from 20 or more years ago.

If you're going to reinvent the wheel it shouldn't be a step backwards.

@Daegalus

This comment has been minimized.

Show comment
Hide comment
@Daegalus

Daegalus Sep 29, 2014

Any news or progress on this? I love atom and everything, but I regularly have to open files bigger than 2MB and I keep having to revert to SublimeText to do it.

I still feel this needs a much higher priority and focus. And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

Any news or progress on this? I love atom and everything, but I regularly have to open files bigger than 2MB and I keep having to revert to SublimeText to do it.

I still feel this needs a much higher priority and focus. And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

@simonzack

This comment has been minimized.

Show comment
Hide comment
@simonzack

simonzack Sep 29, 2014

And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

I agree with this, I think that large file support will change lots of the algorithms (e.g. syntax highlighting, line wrapping, column editing, text search) as well, if they aren't built to handle large files from the start. And that can possibly mean interface changes.

And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

I agree with this, I think that large file support will change lots of the algorithms (e.g. syntax highlighting, line wrapping, column editing, text search) as well, if they aren't built to handle large files from the start. And that can possibly mean interface changes.

@mark-hahn

This comment has been minimized.

Show comment
Hide comment
@mark-hahn

mark-hahn Sep 29, 2014

Contributor

Check out view-tail-large-files. It is read-only with no syntax coloring but it opens multi-gigabyte files with no problem. It designed for log files but can open any text file. It is designed for plugins to add features.

Contributor

mark-hahn commented Sep 29, 2014

Check out view-tail-large-files. It is read-only with no syntax coloring but it opens multi-gigabyte files with no problem. It designed for log files but can open any text file. It is designed for plugins to add features.

@karloluis

This comment has been minimized.

Show comment
Hide comment
@karloluis

karloluis Oct 25, 2014

How come this is a problem on powerful computers that have the amount of RAM to retrieve this large kind of file? (32GB RAM PC)

How come this is a problem on powerful computers that have the amount of RAM to retrieve this large kind of file? (32GB RAM PC)

@simonzack

This comment has been minimized.

Show comment
Hide comment
@simonzack

simonzack Oct 25, 2014

@karloluis Because the limit should always be the harddrive, not ram. Nobody wants to use up all their ram to open up a log file. What if you want to have 4 vm's running at the same time?

Large file support also has problems other than just opening the file. Syntax highlighting, column editing also have associated algorithm complexity problems.

@karloluis Because the limit should always be the harddrive, not ram. Nobody wants to use up all their ram to open up a log file. What if you want to have 4 vm's running at the same time?

Large file support also has problems other than just opening the file. Syntax highlighting, column editing also have associated algorithm complexity problems.

@wb-towa

This comment has been minimized.

Show comment
Hide comment
@wb-towa

wb-towa Oct 25, 2014

If you're running so many VMs that memory is a concern even with 32gb then you shouldn't be using Atom full stop. It's memory use is quite high for smaller files compared to something like vim or emacs or even sublime I suspect. CPU usage is also higher than it should be for plain text which will affect your multiple VMs.

If we're being honest with ourselves Atom is reinventing the wheel and with numerous regressions.

wb-towa commented Oct 25, 2014

If you're running so many VMs that memory is a concern even with 32gb then you shouldn't be using Atom full stop. It's memory use is quite high for smaller files compared to something like vim or emacs or even sublime I suspect. CPU usage is also higher than it should be for plain text which will affect your multiple VMs.

If we're being honest with ourselves Atom is reinventing the wheel and with numerous regressions.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Mar 19, 2015

Contributor

That is the plan. I predict it will help us a lot but we may also need to continue optimizing at other layers as well. It's a big piece of the puzzle though.

Contributor

nathansobo commented Mar 19, 2015

That is the plan. I predict it will help us a lot but we may also need to continue optimizing at other layers as well. It's a big piece of the puzzle though.

@karai17

This comment has been minimized.

Show comment
Hide comment
@karai17

karai17 Mar 19, 2015

Awesome! I look very forward to this.

karai17 commented Mar 19, 2015

Awesome! I look very forward to this.

@octref

This comment has been minimized.

Show comment
Hide comment
@octref

octref Mar 19, 2015

Contributor

@karai17
Just wondering. You mentioned you need to edit 3D model files. How do those files look like?
And, as for syntax-highlighting on them, do you

  • Absolutely need it
  • Don't want it at all
  • Can live without it but wish to have it
Contributor

octref commented Mar 19, 2015

@karai17
Just wondering. You mentioned you need to edit 3D model files. How do those files look like?
And, as for syntax-highlighting on them, do you

  • Absolutely need it
  • Don't want it at all
  • Can live without it but wish to have it
@karai17

This comment has been minimized.

Show comment
Hide comment
@karai17

karai17 Mar 19, 2015

@octref

Here is an example of a several megabyte IQE 3D model file. The weird indenting is caused by Blender's IQE exporter, it is not part of the spec. Here is the IQE spec.

Since IQE is a data format and not a language format, I wouldn't really expect it to have syntax highlighting. Maybe it would be neat to write a package that did highlight it, but I don't think it would be all that useful.

There are some language files that can reach huge sizes such as SQL dumps that would benefit from syntax highlighting, though.

karai17 commented Mar 19, 2015

@octref

Here is an example of a several megabyte IQE 3D model file. The weird indenting is caused by Blender's IQE exporter, it is not part of the spec. Here is the IQE spec.

Since IQE is a data format and not a language format, I wouldn't really expect it to have syntax highlighting. Maybe it would be neat to write a package that did highlight it, but I don't think it would be all that useful.

There are some language files that can reach huge sizes such as SQL dumps that would benefit from syntax highlighting, though.

@octref

This comment has been minimized.

Show comment
Hide comment
@octref

octref Mar 19, 2015

Contributor

@karai17

Got it.
Currently Atom is using converted textmate grammar bundles. Once the support for large file is there, you should be able to write a custom syntax package and get syntax-highlighting in Atom for the 3D model files.

Contributor

octref commented Mar 19, 2015

@karai17

Got it.
Currently Atom is using converted textmate grammar bundles. Once the support for large file is there, you should be able to write a custom syntax package and get syntax-highlighting in Atom for the 3D model files.

@karai17

This comment has been minimized.

Show comment
Hide comment
@karai17

karai17 Mar 19, 2015

Awesome! One thing I noticed recently is that, at least in the language-lua package, some syntax is highlighted in a way that I don't believe it should be, and I am not sure if that is the package itself or a symptom of how Atom colours syntax.

An example would be that if I use a Lua core function such as print(), Atom highlights that text. But if I type it incorrectly, such as pirnt(), it still highlights it. What seems to be happening is the package picks up on () and highlights it as a function. With sublime I believe only core functions were highlighted, or at the very least core and user functions had separate colours.

I know that this falls outside of the scope of this particular issue, but I was wondering if it had to do with Atom's syntax highlighting or if it was just an oversight on the package itself?

karai17 commented Mar 19, 2015

Awesome! One thing I noticed recently is that, at least in the language-lua package, some syntax is highlighted in a way that I don't believe it should be, and I am not sure if that is the package itself or a symptom of how Atom colours syntax.

An example would be that if I use a Lua core function such as print(), Atom highlights that text. But if I type it incorrectly, such as pirnt(), it still highlights it. What seems to be happening is the package picks up on () and highlights it as a function. With sublime I believe only core functions were highlighted, or at the very least core and user functions had separate colours.

I know that this falls outside of the scope of this particular issue, but I was wondering if it had to do with Atom's syntax highlighting or if it was just an oversight on the package itself?

@LPGhatguy

This comment has been minimized.

Show comment
Hide comment
@LPGhatguy

LPGhatguy Mar 19, 2015

I've seen some interesting behavior with themes too as far as Lua core functions go.

Here's a side-by-side comparison of some code. Sublime Text 3 is on the left (correct behavior), while Atom is on the right, using the same theme as ST3 using the theme importer from a couple versions back.

http://i.imgur.com/V9b5GWh.png

(It doesn't help that I have each editor in a different font and font size at the moment)

I've seen some interesting behavior with themes too as far as Lua core functions go.

Here's a side-by-side comparison of some code. Sublime Text 3 is on the left (correct behavior), while Atom is on the right, using the same theme as ST3 using the theme importer from a couple versions back.

http://i.imgur.com/V9b5GWh.png

(It doesn't help that I have each editor in a different font and font size at the moment)

@octref

This comment has been minimized.

Show comment
Hide comment
@octref

octref Mar 21, 2015

Contributor

@karai17
If you write a test.lua file with the following two lines:

print()
pirnt()

Use Ctrl-Shift-I to open the inspector, and Ctrl-F find the corresponding DOM node:
For print(), it's:
<span class="support function lua">print</span>
For pirnt(), it's:
<span class="support function any-method lua">pirnt</span>
On the inspector, you can find they are all colored by the .support.function class, which corresponds to this LESS rule in Atom-Dark theme:
https://github.com/atom/atom-dark-syntax/blob/master/index.less#L218-L220
The grammar:
https://github.com/FireZenk/language-lua/blob/d1bbd1805cfc1e0cfaeb60b426133f72b442d2e3/grammars/lua.cson#L131-L133

So if you are unhappy about this grammar syntax, you can open an issue to language-lua, asking them to parse print differently.

For example, python parses print as keyword.other.python and gives it a pretty azure :-)
<span class="keyword other python">print</span>

But

@karai17 , @LPGhatguy
Let's stop going deep on syntax here. Many people are subscribed to this issue to get progress on large file support.

Contributor

octref commented Mar 21, 2015

@karai17
If you write a test.lua file with the following two lines:

print()
pirnt()

Use Ctrl-Shift-I to open the inspector, and Ctrl-F find the corresponding DOM node:
For print(), it's:
<span class="support function lua">print</span>
For pirnt(), it's:
<span class="support function any-method lua">pirnt</span>
On the inspector, you can find they are all colored by the .support.function class, which corresponds to this LESS rule in Atom-Dark theme:
https://github.com/atom/atom-dark-syntax/blob/master/index.less#L218-L220
The grammar:
https://github.com/FireZenk/language-lua/blob/d1bbd1805cfc1e0cfaeb60b426133f72b442d2e3/grammars/lua.cson#L131-L133

So if you are unhappy about this grammar syntax, you can open an issue to language-lua, asking them to parse print differently.

For example, python parses print as keyword.other.python and gives it a pretty azure :-)
<span class="keyword other python">print</span>

But

@karai17 , @LPGhatguy
Let's stop going deep on syntax here. Many people are subscribed to this issue to get progress on large file support.

@karai17 karai17 referenced this issue in FireZenk/language-lua Mar 21, 2015

Open

Lua Syntax highlighting is a bit off #9

@erikdonohoo

This comment has been minimized.

Show comment
Hide comment
@erikdonohoo

erikdonohoo Mar 23, 2015

Super happy to hear this is being worked on. Its not just large binary files that are issues here. Any js framework (like angular, ember) end up being too large for atom to handle. It doesn't seem like an edge case to occasionally need to look at the source of some 3rd party package you are using.

Super happy to hear this is being worked on. Its not just large binary files that are issues here. Any js framework (like angular, ember) end up being too large for atom to handle. It doesn't seem like an edge case to occasionally need to look at the source of some 3rd party package you are using.

@wpostma

This comment has been minimized.

Show comment
Hide comment
@wpostma

wpostma Mar 27, 2015

This editor is a toy, if it can't handle large files. I'll be watching to see if Atom matures in this area.

wpostma commented Mar 27, 2015

This editor is a toy, if it can't handle large files. I'll be watching to see if Atom matures in this area.

@murphyj

This comment has been minimized.

Show comment
Hide comment
@murphyj

murphyj Mar 30, 2015

Agree with @erikdonohoo - this is not an edge case. You also can't open some crucial files for Ghost because it's ember.js backed (hence has large binary vendor files).

murphyj commented Mar 30, 2015

Agree with @erikdonohoo - this is not an edge case. You also can't open some crucial files for Ghost because it's ember.js backed (hence has large binary vendor files).

@sduensin

This comment has been minimized.

Show comment
Hide comment
@sduensin

sduensin Apr 3, 2015

This is a HUGE problem. 2 meg is nothing these days. As much as I like Atom, this limitation makes it pretty useless for day to day work. :-(

sduensin commented Apr 3, 2015

This is a HUGE problem. 2 meg is nothing these days. As much as I like Atom, this limitation makes it pretty useless for day to day work. :-(

@sceee

This comment has been minimized.

Show comment
Hide comment
@sceee

sceee Apr 3, 2015

I can support this - many files (let it be log files etc.) have more than 2 MB in size. This always makes you need another editor to open these files beside atom.

sceee commented Apr 3, 2015

I can support this - many files (let it be log files etc.) have more than 2 MB in size. This always makes you need another editor to open these files beside atom.

@gpluta

This comment has been minimized.

Show comment
Hide comment
@gpluta

gpluta Apr 5, 2015

This limitation is really a dealbreaker when it comes to using atom... I hope this gets improved one day.

gpluta commented Apr 5, 2015

This limitation is really a dealbreaker when it comes to using atom... I hope this gets improved one day.

@mindvox

This comment has been minimized.

Show comment
Hide comment
@mindvox

mindvox Apr 7, 2015

This should be considered a standard feature for a modern text editor. I would really love to see this feature added as Atom is such an amazing text editor.

mindvox commented Apr 7, 2015

This should be considered a standard feature for a modern text editor. I would really love to see this feature added as Atom is such an amazing text editor.

@50Wliu

This comment has been minimized.

Show comment
Hide comment
@50Wliu

50Wliu Apr 8, 2015

Member

This is currently being worked on. The Atom team recognizes that it's a big problem, but it also needs a big rewrite in order to make everything work. See https://github.com/atom/text-document for more details.

Member

50Wliu commented Apr 8, 2015

This is currently being worked on. The Atom team recognizes that it's a big problem, but it also needs a big rewrite in order to make everything work. See https://github.com/atom/text-document for more details.

@mindvox

This comment has been minimized.

Show comment
Hide comment
@mindvox

mindvox Apr 8, 2015

@50Wliu I can appreciate the work involved. Looking forward to seeing this in a future release 😄

mindvox commented Apr 8, 2015

@50Wliu I can appreciate the work involved. Looking forward to seeing this in a future release 😄

@backspaces

This comment has been minimized.

Show comment
Hide comment
@backspaces

backspaces Apr 8, 2015

Could we pin this? The extraordinary atom/text-document should be shown to
all those concerned.

I say this because Atom is historic, like several JS/Html/Browser/Node
projects have been over the last few years.

Many of Atom's clients are simply looking for a coherent text editor.
That's fine. But they should also understand the JavaScript Revolution
background.

On Tue, Apr 7, 2015 at 6:19 PM, Karl Bateman notifications@github.com
wrote:

@50Wliu https://github.com/50Wliu I can appreciate the work involved.
Looking forward to seeing this in a future release [image: 😄]


Reply to this email directly or view it on GitHub
#307 (comment).

Could we pin this? The extraordinary atom/text-document should be shown to
all those concerned.

I say this because Atom is historic, like several JS/Html/Browser/Node
projects have been over the last few years.

Many of Atom's clients are simply looking for a coherent text editor.
That's fine. But they should also understand the JavaScript Revolution
background.

On Tue, Apr 7, 2015 at 6:19 PM, Karl Bateman notifications@github.com
wrote:

@50Wliu https://github.com/50Wliu I can appreciate the work involved.
Looking forward to seeing this in a future release [image: 😄]


Reply to this email directly or view it on GitHub
#307 (comment).

@Tapefabrik

This comment has been minimized.

Show comment
Hide comment

+1

@mika76

This comment has been minimized.

Show comment
Hide comment
@mika76

mika76 Apr 8, 2015

I must admit these days the only thing keeping me from using Atom is the performance and the large file non-support. I can't wait to see this working 😁

mika76 commented Apr 8, 2015

I must admit these days the only thing keeping me from using Atom is the performance and the large file non-support. I can't wait to see this working 😁

@xpepermint

This comment has been minimized.

Show comment
Hide comment

+1

@theksmith

This comment has been minimized.

Show comment
Hide comment

+100

@pixelchutes

This comment has been minimized.

Show comment
Hide comment
@pixelchutes

pixelchutes Apr 10, 2015

+2.0001 (MB) ...wait for it... 😉

UPDATE:
@thedaniel is right guys. Sorry for the spam / bad joke.

+2.0001 (MB) ...wait for it... 😉

UPDATE:
@thedaniel is right guys. Sorry for the spam / bad joke.

@thedaniel

This comment has been minimized.

Show comment
Hide comment
@thedaniel

thedaniel Apr 10, 2015

Contributor

This is actively being worked on and is a high priority, no reason to post +1 comments anymore - it just spams everyone watching the issue.

Contributor

thedaniel commented Apr 10, 2015

This is actively being worked on and is a high priority, no reason to post +1 comments anymore - it just spams everyone watching the issue.

@iamstarkov

This comment has been minimized.

Show comment
Hide comment
@iamstarkov

iamstarkov Apr 10, 2015

@thedaniel issues can be closed for commenting

@thedaniel issues can be closed for commenting

@ilanbiala

This comment has been minimized.

Show comment
Hide comment
@ilanbiala

ilanbiala Apr 11, 2015

@thedaniel I think it would help if any other possible progress updates regarding perf. improvements were provided. If there aren't any, it would just be nice to know that it's happening by @nathansobo or you commenting in addition to adding the in progress label.

@thedaniel I think it would help if any other possible progress updates regarding perf. improvements were provided. If there aren't any, it would just be nice to know that it's happening by @nathansobo or you commenting in addition to adding the in progress label.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Apr 11, 2015

Contributor

It's happening. Check out the text-document repo. Out of the country for a conference which is consuming some attention, but it's been almost my sole focus. Not much more to say than the progress and write-ups you can observe there. I empathize with everyone who wants this because I want it too, and it's been challenging but we're definitely getting there.

Contributor

nathansobo commented Apr 11, 2015

It's happening. Check out the text-document repo. Out of the country for a conference which is consuming some attention, but it's been almost my sole focus. Not much more to say than the progress and write-ups you can observe there. I empathize with everyone who wants this because I want it too, and it's been challenging but we're definitely getting there.

@robhawkes

This comment has been minimized.

Show comment
Hide comment
@robhawkes

robhawkes Apr 14, 2015

@thedaniel @matmuchrapna: Yup, it's simple to lock the comments while still allowing contributors to discuss things.

@thedaniel @matmuchrapna: Yup, it's simple to lock the comments while still allowing contributors to discuss things.

@kylegoetz

This comment has been minimized.

Show comment
Hide comment
@kylegoetz

kylegoetz Apr 14, 2015

I disagree this is an edge case. There are popular third-party JS libraries that are greater than this limit. What motivates my finding this bug is that Telerik's Kendo library is too big to open in Atom unless you open the minified version, which is, of course, not useful at all.

I'm having to read the library in Firefox's debugger since I can't open it in Atom to read it. You can argue philosophy about whether I should be opening a file I don't intend to edit in Atom, but I might edit it if I need to and just consider it no longer a proper third party lib but rather a first party, but can't edit in Atom.

I disagree this is an edge case. There are popular third-party JS libraries that are greater than this limit. What motivates my finding this bug is that Telerik's Kendo library is too big to open in Atom unless you open the minified version, which is, of course, not useful at all.

I'm having to read the library in Firefox's debugger since I can't open it in Atom to read it. You can argue philosophy about whether I should be opening a file I don't intend to edit in Atom, but I might edit it if I need to and just consider it no longer a proper third party lib but rather a first party, but can't edit in Atom.

@atom atom locked and limited conversation to collaborators Apr 14, 2015

@benogle

This comment has been minimized.

Show comment
Hide comment
@benogle

benogle Apr 14, 2015

Contributor

I've locked this. We do believe this is an important case to handle. We are currently working on this. Please visit https://github.com/atom/text-document for more information.

Contributor

benogle commented Apr 14, 2015

I've locked this. We do believe this is an important case to handle. We are currently working on this. Please visit https://github.com/atom/text-document for more information.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo May 14, 2015

Contributor

Just an update here... in the interest of time we've decided to refocus our efforts on more incremental improvements.

  • One large chunk of work from text-document, the MarkerIndex, is in the process of being integrated into our existing document model. This will improve Atom's performance in the presence of large numbers of markers, such as when visualizing numerous search results in large files. The pull request should be merged in the next few days.
  • We're also working to reduce the memory consumption of our tokenized line representation.

We'll be searching for several more smaller wins like this, then redirect our attention back to some of the bigger changes researched in text-document.

Some ideas for other areas we plan on investigating in the next couple weeks:

  • Looking for low hanging fruit in the initial load of a document.
  • Looking for low hanging fruit in parser performance.
  • Exploring batching display buffer updates for changes within a single transaction.

Making Atom performant for files of all sizes remains a top priority. We'll continue to post updates here.

Contributor

nathansobo commented May 14, 2015

Just an update here... in the interest of time we've decided to refocus our efforts on more incremental improvements.

  • One large chunk of work from text-document, the MarkerIndex, is in the process of being integrated into our existing document model. This will improve Atom's performance in the presence of large numbers of markers, such as when visualizing numerous search results in large files. The pull request should be merged in the next few days.
  • We're also working to reduce the memory consumption of our tokenized line representation.

We'll be searching for several more smaller wins like this, then redirect our attention back to some of the bigger changes researched in text-document.

Some ideas for other areas we plan on investigating in the next couple weeks:

  • Looking for low hanging fruit in the initial load of a document.
  • Looking for low hanging fruit in parser performance.
  • Exploring batching display buffer updates for changes within a single transaction.

Making Atom performant for files of all sizes remains a top priority. We'll continue to post updates here.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Jun 8, 2015

Contributor

There's still more to do here, but a basic fix for the 80% case of loading large files is now on master. Before 1.0 I'd like to fix a few performance hiccups when moving the cursor in huge files. Post 1.0 I'd like to:

  • Syntax highlight in a background process so we can highlight huge files without hiccups.
  • Show a progress bar while performing I/O and computing initial metadata while loading a large file.
  • Drop the memory overhead another order of magnitude.
  • Support folds in large files

We can create separate issues for all of those, but I'm going to close this one. Atom can now load and edit large files. Don't worry, we'll keep working on refining it.

Contributor

nathansobo commented Jun 8, 2015

There's still more to do here, but a basic fix for the 80% case of loading large files is now on master. Before 1.0 I'd like to fix a few performance hiccups when moving the cursor in huge files. Post 1.0 I'd like to:

  • Syntax highlight in a background process so we can highlight huge files without hiccups.
  • Show a progress bar while performing I/O and computing initial metadata while loading a large file.
  • Drop the memory overhead another order of magnitude.
  • Support folds in large files

We can create separate issues for all of those, but I'm going to close this one. Atom can now load and edit large files. Don't worry, we'll keep working on refining it.

@nathansobo nathansobo closed this Jun 8, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.