Large file support

[watsonian]

So, I use my text editor for all kinds of things (writing code, reading readmes, editing dotfiles, etc). One of the things I do most often during any given day is open up large log files to troubleshoot Enterprise problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately -- the file selector didn't change and the entire window went completely white 4 seconds after I tried viewing the file. I could still close the top level window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice progress indicator while it was loading:

Once it loaded it was as responsive as any other file I load (scrolled fast, normal text highlight performance, etc). It would be nice if we set this as a baseline for expected behavior (progress bar + responsive after loading).

[nathansobo]

Yeah, we currently do a blocking read to load files. We need to switch to
something event-driven. I also wonder if we can load the file progressively
off the disk as needed so we don't have to seek the whole thing into
memory. But that's super-advanced. Just avoiding blocking the UI thread
would be a good start. Yet another thing I'd like to do with Node.

On Sun, Feb 24, 2013 at 5:54 PM, Joel Watson notifications@github.comwrote:

So, I use my text editor for all kinds of things (writing code, reading
readmes, editing dotfiles, etc). One of the things I do most often during
any given day is open up large log files to troubleshoot Enterprise
problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately --
the file selector didn't change and the entire window went completely white
4 seconds after I tried viewing the file. I could still close the top level
window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice
progress indicator while it was loading:

[image: Screen Shot 2013-02-24 at 4 50 56 PM]https://f.cloud.github.com/assets/244/190379/c0abc27e-7ee5-11e2-8ec4-4205cdee0ad0.png

Once it loaded it was as responsive as any other file I load (scrolled
fast, normal text highlight performance, etc). It would be nice if we set
this as a baseline for expected behavior (progress bar + responsive after
loading).

—
Reply to this email directly or view it on GitHubhttps://github.com/github/atom/issues/307.

[nathansobo]

Yeah, we currently do a blocking read to load files. We need to switch to
something event-driven. I also wonder if we can load the file progressively
off the disk as needed so we don't have to seek the whole thing into
memory. But that's super-advanced. Just avoiding blocking the UI thread
would be a good start. Yet another thing I'd like to do with Node.

On Sun, Feb 24, 2013 at 5:54 PM, Joel Watson notifications@github.comwrote:

So, I use my text editor for all kinds of things (writing code, reading
readmes, editing dotfiles, etc). One of the things I do most often during
any given day is open up large log files to troubleshoot Enterprise
problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately --
the file selector didn't change and the entire window went completely white
4 seconds after I tried viewing the file. I could still close the top level
window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice
progress indicator while it was loading:

[image: Screen Shot 2013-02-24 at 4 50 56 PM]https://f.cloud.github.com/assets/244/190379/c0abc27e-7ee5-11e2-8ec4-4205cdee0ad0.png

Once it loaded it was as responsive as any other file I load (scrolled
fast, normal text highlight performance, etc). It would be nice if we set
this as a baseline for expected behavior (progress bar + responsive after
loading).

—
Reply to this email directly or view it on GitHubhttps://github.com/github/atom/issues/307.

[watsonian]

I just tried opening a 124MB JSON file directly from the CLI and got this:

RangeError: Maximum call stack size exceeded
    at BufferChangeOperation.module.exports.BufferChangeOperation.changeBuffer (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:99:17)
    at BufferChangeOperation.module.exports.BufferChangeOperation.do (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:37:19)
    at Buffer.module.exports.Buffer.pushOperation (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:348:31)
    at Buffer.module.exports.Buffer.change (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:314:20)
    at Buffer.module.exports.Buffer.setText (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:174:19)
    at Buffer.module.exports.Buffer.reload (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:126:12)
    at new Buffer (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:65:14)
    at Project.module.exports.Project.buildBuffer (/Applications/Atom.app/Contents/Resources/src/app/project.js:270:16)
    at Project.module.exports.Project.bufferForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:259:33)
    at Project.module.exports.Project.buildEditSessionForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:180:41) index.html:23
window.onload

[watsonian]

I just tried opening a 124MB JSON file directly from the CLI and got this:

RangeError: Maximum call stack size exceeded
    at BufferChangeOperation.module.exports.BufferChangeOperation.changeBuffer (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:99:17)
    at BufferChangeOperation.module.exports.BufferChangeOperation.do (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:37:19)
    at Buffer.module.exports.Buffer.pushOperation (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:348:31)
    at Buffer.module.exports.Buffer.change (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:314:20)
    at Buffer.module.exports.Buffer.setText (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:174:19)
    at Buffer.module.exports.Buffer.reload (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:126:12)
    at new Buffer (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:65:14)
    at Project.module.exports.Project.buildBuffer (/Applications/Atom.app/Contents/Resources/src/app/project.js:270:16)
    at Project.module.exports.Project.bufferForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:259:33)
    at Project.module.exports.Project.buildEditSessionForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:180:41) index.html:23
window.onload

[probablycorey]

This moved forward after #939. But there is still some telepath work to get the rest working. Adding single-user mode to telepath that eliminates the need for each character to have a location may be a solution.

[probablycorey]

This moved forward after #939. But there is still some telepath work to get the rest working. Adding single-user mode to telepath that eliminates the need for each character to have a location may be a solution.

[stuart-warren]

I'm failing to open up a 1.4MB HTML file with the same RangeError: Maximum call stack exceeded

Running Version 0.105
Has anything happened on this issue since October?

Full error:

Window load time: 761ms 
/Applications/Atom.app/Contents/Resources/app/src/window-bootstrap.js:18
RangeError: Maximum call stack size exceeded
  at DisplayBuffer.module.exports.DisplayBuffer.updateScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:1246:17)
  at DisplayBuffer.module.exports.DisplayBuffer.updateAllScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:179:19)
  at new DisplayBuffer (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:94:12)
  at new Editor (/Applications/Atom.app/Contents/Resources/app/src/editor.js:76:30)
  at Project.module.exports.Project.buildEditorForBuffer (/Applications/Atom.app/Contents/Resources/app/src/project.js:430:16)
  at /Applications/Atom.app/Contents/Resources/app/src/project.js:192:24
  at _fulfilled (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:787:54)
  at self.promiseDispatch.done (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:816:30)
  at Promise.promise.promiseDispatch (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:749:13)
  at /Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:557:44
  at flush (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:108:17)
  at process._tickCallback (node.js:605:11)

Thanks

[stuart-warren]

I'm failing to open up a 1.4MB HTML file with the same RangeError: Maximum call stack exceeded

Running Version 0.105
Has anything happened on this issue since October?

Full error:

Window load time: 761ms 
/Applications/Atom.app/Contents/Resources/app/src/window-bootstrap.js:18
RangeError: Maximum call stack size exceeded
  at DisplayBuffer.module.exports.DisplayBuffer.updateScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:1246:17)
  at DisplayBuffer.module.exports.DisplayBuffer.updateAllScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:179:19)
  at new DisplayBuffer (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:94:12)
  at new Editor (/Applications/Atom.app/Contents/Resources/app/src/editor.js:76:30)
  at Project.module.exports.Project.buildEditorForBuffer (/Applications/Atom.app/Contents/Resources/app/src/project.js:430:16)
  at /Applications/Atom.app/Contents/Resources/app/src/project.js:192:24
  at _fulfilled (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:787:54)
  at self.promiseDispatch.done (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:816:30)
  at Promise.promise.promiseDispatch (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:749:13)
  at /Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:557:44
  at flush (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:108:17)
  at process._tickCallback (node.js:605:11)

Thanks

[probablycorey]

Opening files larger than 2MB is an edge case that most users don't run into, but I wanted to investigate this.

How to create a large file

Download this file and run it generate-text five-megabytes-of-words.txt 5

Good news

Atom handles 5MB without changing anything.

Bad news

Atom hangs when opening a a file >10MB.

Atom eats up ~300MB or RAM when opening large files.

[probablycorey]

Opening files larger than 2MB is an edge case that most users don't run into, but I wanted to investigate this.

How to create a large file

Download this file and run it generate-text five-megabytes-of-words.txt 5

Good news

Atom handles 5MB without changing anything.

Bad news

Atom hangs when opening a a file >10MB.

Atom eats up ~300MB or RAM when opening large files.

[probablycorey]

I modified window-bootstrap to test out some memory assumptions. Here is the memory footprint of Atom using a file size of 40MB

Baseline (window-bootstrap.coffee does nothing): 50MB
Baseline + file via fs: 96MB
Baseline + file via File: 181MB
Baseline + TextBuffer: 538MB

[probablycorey]

I modified window-bootstrap to test out some memory assumptions. Here is the memory footprint of Atom using a file size of 40MB

Baseline (window-bootstrap.coffee does nothing): 50MB
Baseline + file via fs: 96MB
Baseline + file via File: 181MB
Baseline + TextBuffer: 538MB

[nathansobo]

I'm thinking we probably need to store the TextBuffer data off the v8 heap using a C++ extension, otherwise it hoses GC performance.

[nathansobo]

I'm thinking we probably need to store the TextBuffer data off the v8 heap using a C++ extension, otherwise it hoses GC performance.

[nathansobo]

But there's also a ton of other data such as tokenization etc.

[nathansobo]

But there's also a ton of other data such as tokenization etc.

[zcbenz]

Node's Buffer stores data in C++ heap. But if we want to support really big files we have to avoid reading the whole file in memory.

[zcbenz]

Node's Buffer stores data in C++ heap. But if we want to support really big files we have to avoid reading the whole file in memory.

[nathansobo]

Interested in any ideas you have about the design of that @zcbenz. The hardest part might be a bunch of APIs might need to go async to support loading more of the file for certain operations, which could make for more awkward scripting.

[nathansobo]

Interested in any ideas you have about the design of that @zcbenz. The hardest part might be a bunch of APIs might need to go async to support loading more of the file for certain operations, which could make for more awkward scripting.

[zcbenz]

The basic idea is to only draw the file on needed, but it will need a good caching strategy to make things smooth since drawing file would rely on disk IO, many native apps use memory-mapped files to simplify it, a rough design is:

when opening a large file

1. quickly scans the whole file to count there are how many lines and how long is each line.
2. the editor view computes and draws a virtual scroller bar.
3. the editor view decides which part of file should be read into memory according to current position of scroller.
4. the editor view reads that part of the file and draws them.

when user scrolls the editor view
repeat 3 - 4

when the editor view is resized
repeat 2 - 4

when the file is changed outside
repeat 1 - 4

• step 1 and step4 could be asynchronous.
• the step 1 will need to take the responsibility of tokenization.
• when file is really large that even scanning it takes time we may give up scanning and tokenizing it, the editor view can just show scroller bar with a minimal drag handle, and when user drags the scroller the editor view just roughly computes a start position to read file.

[zcbenz]

The basic idea is to only draw the file on needed, but it will need a good caching strategy to make things smooth since drawing file would rely on disk IO, many native apps use memory-mapped files to simplify it, a rough design is:

when opening a large file

1. quickly scans the whole file to count there are how many lines and how long is each line.
2. the editor view computes and draws a virtual scroller bar.
3. the editor view decides which part of file should be read into memory according to current position of scroller.
4. the editor view reads that part of the file and draws them.

when user scrolls the editor view
repeat 3 - 4

when the editor view is resized
repeat 2 - 4

when the file is changed outside
repeat 1 - 4

• step 1 and step4 could be asynchronous.
• the step 1 will need to take the responsibility of tokenization.
• when file is really large that even scanning it takes time we may give up scanning and tokenizing it, the editor view can just show scroller bar with a minimal drag handle, and when user drags the scroller the editor view just roughly computes a start position to read file.

[i4004]

I tried to open ~8gb text file, but nothing just happened :)

[i4004]

I tried to open ~8gb text file, but nothing just happened :)

[batjko]

I'm not sure how they do it, but Baretail is a pretty impressive log file viewer in that regard.
It doesn't care how big your file is and only seems to load portions of it as needed, with very fast seek and search across the entire monster file though, including life refresh of the bottom lines (useful for log files of course) - while keeping the memory footprint quite small.

I assume that's rather difficult to implement in Atom, maybe too much effort for the rare use case?

[batjko]

I'm not sure how they do it, but Baretail is a pretty impressive log file viewer in that regard.
It doesn't care how big your file is and only seems to load portions of it as needed, with very fast seek and search across the entire monster file though, including life refresh of the bottom lines (useful for log files of course) - while keeping the memory footprint quite small.

I assume that's rather difficult to implement in Atom, maybe too much effort for the rare use case?

[nathansobo]

@batjko Thanks for the tip. Long term we definitely want to support files of arbitrary size, but an approach that doesn't load the whole file into memory is going to force us to redesign some of our synchronous APIs to be asynchronous due to the single-threaded nature of JavaScript. We made a pragmatic decision early on to perform editor state manipulations synchronously for a more convenient scripting experience, but we may need to revisit that decision.

[nathansobo]

@batjko Thanks for the tip. Long term we definitely want to support files of arbitrary size, but an approach that doesn't load the whole file into memory is going to force us to redesign some of our synchronous APIs to be asynchronous due to the single-threaded nature of JavaScript. We made a pragmatic decision early on to perform editor state manipulations synchronously for a more convenient scripting experience, but we may need to revisit that decision.

[batjko]

@nathansobo Personally, I'm happy with it as it is. I have no use for extreme file sizes in an editor.

I think the huge log file scenario is not really one for editors, but rather for viewers/greppers or whatever you might call that category.

[batjko]

@nathansobo Personally, I'm happy with it as it is. I have no use for extreme file sizes in an editor.

I think the huge log file scenario is not really one for editors, but rather for viewers/greppers or whatever you might call that category.

[nathansobo]

Yeah, I pretty much agree. I'm confident we'll get there eventually, but there's a lot of other things we consider more important right now.

[nathansobo]

Yeah, I pretty much agree. I'm confident we'll get there eventually, but there's a lot of other things we consider more important right now.

[simonzack]

@batjko I don't think this is a rare use case, given the amount of text editors which advertise this as a feature, and the comments so far on this issue. Log viewers and greppers are nice, but it is nicer to view and edit text in an editor. After all isn't that what a text editor is for.

[simonzack]

@batjko I don't think this is a rare use case, given the amount of text editors which advertise this as a feature, and the comments so far on this issue. Log viewers and greppers are nice, but it is nicer to view and edit text in an editor. After all isn't that what a text editor is for.

[detly]

The other aspect to this is the failure mode while it isn't supported. If viewing and editing large text files isn't going to work for the time being, it might still be nice to have a better failure mode than locking up and forcing the user to kill Atom.

[detly]

The other aspect to this is the failure mode while it isn't supported. If viewing and editing large text files isn't going to work for the time being, it might still be nice to have a better failure mode than locking up and forcing the user to kill Atom.

[nathansobo]

The failure mode is currently an error message displayed in the console for files exceeding 2mb, which should leave Atom in a usable state. Are you talking about large files that are < 2mb?

[nathansobo]

The failure mode is currently an error message displayed in the console for files exceeding 2mb, which should leave Atom in a usable state. Are you talking about large files that are < 2mb?

[detly]

Yes. This is a 1.4M (generated) Python file.

[detly]

Yes. This is a 1.4M (generated) Python file.

[izuzak]

Yes. This is a 1.4M (generated) Python file.

@detly Can you share the file for which this is happening? Also, does this happen when you run atom in safe mode? Anything else special about the file (e.g. a really long line)?

[izuzak]

Yes. This is a 1.4M (generated) Python file.

@detly Can you share the file for which this is happening? Also, does this happen when you run atom in safe mode? Anything else special about the file (e.g. a really long line)?

[izuzak]

Can you share the file for which this is happening?

Actually, I just tried this myself and Atom was slowing down drastically on large files like that. I thought that wasn't happening on files of that size -- I thought I tried that a while ago but I might be wrong. Sorry about that!

[izuzak]

Can you share the file for which this is happening?

Actually, I just tried this myself and Atom was slowing down drastically on large files like that. I thought that wasn't happening on files of that size -- I thought I tried that a while ago but I might be wrong. Sorry about that!

[wb-towa]

I can't believe this doesn't seem to have a very high priority given there have been numerous releases of atom and it still can not match the performance of editors from 20 or more years ago.

If you're going to reinvent the wheel it shouldn't be a step backwards.

[wb-towa]

I can't believe this doesn't seem to have a very high priority given there have been numerous releases of atom and it still can not match the performance of editors from 20 or more years ago.

If you're going to reinvent the wheel it shouldn't be a step backwards.

[Daegalus]

Any news or progress on this? I love atom and everything, but I regularly have to open files bigger than 2MB and I keep having to revert to SublimeText to do it.

I still feel this needs a much higher priority and focus. And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

[Daegalus]

Any news or progress on this? I love atom and everything, but I regularly have to open files bigger than 2MB and I keep having to revert to SublimeText to do it.

I still feel this needs a much higher priority and focus. And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

[simonzack]

And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

I agree with this, I think that large file support will change lots of the algorithms (e.g. syntax highlighting, line wrapping, column editing, text search) as well, if they aren't built to handle large files from the start. And that can possibly mean interface changes.

[simonzack]

And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

I agree with this, I think that large file support will change lots of the algorithms (e.g. syntax highlighting, line wrapping, column editing, text search) as well, if they aren't built to handle large files from the start. And that can possibly mean interface changes.

[mark-hahn]

Check out view-tail-large-files. It is read-only with no syntax coloring but it opens multi-gigabyte files with no problem. It designed for log files but can open any text file. It is designed for plugins to add features.

[mark-hahn]

Check out view-tail-large-files. It is read-only with no syntax coloring but it opens multi-gigabyte files with no problem. It designed for log files but can open any text file. It is designed for plugins to add features.

[karloluis]

How come this is a problem on powerful computers that have the amount of RAM to retrieve this large kind of file? (32GB RAM PC)

[karloluis]

How come this is a problem on powerful computers that have the amount of RAM to retrieve this large kind of file? (32GB RAM PC)

[simonzack]

@karloluis Because the limit should always be the harddrive, not ram. Nobody wants to use up all their ram to open up a log file. What if you want to have 4 vm's running at the same time?

Large file support also has problems other than just opening the file. Syntax highlighting, column editing also have associated algorithm complexity problems.

[simonzack]

@karloluis Because the limit should always be the harddrive, not ram. Nobody wants to use up all their ram to open up a log file. What if you want to have 4 vm's running at the same time?

Large file support also has problems other than just opening the file. Syntax highlighting, column editing also have associated algorithm complexity problems.

[wb-towa]

If you're running so many VMs that memory is a concern even with 32gb then you shouldn't be using Atom full stop. It's memory use is quite high for smaller files compared to something like vim or emacs or even sublime I suspect. CPU usage is also higher than it should be for plain text which will affect your multiple VMs.

If we're being honest with ourselves Atom is reinventing the wheel and with numerous regressions.

[wb-towa]

If you're running so many VMs that memory is a concern even with 32gb then you shouldn't be using Atom full stop. It's memory use is quite high for smaller files compared to something like vim or emacs or even sublime I suspect. CPU usage is also higher than it should be for plain text which will affect your multiple VMs.

If we're being honest with ourselves Atom is reinventing the wheel and with numerous regressions.

[nathansobo]

That is the plan. I predict it will help us a lot but we may also need to continue optimizing at other layers as well. It's a big piece of the puzzle though.

[nathansobo]

That is the plan. I predict it will help us a lot but we may also need to continue optimizing at other layers as well. It's a big piece of the puzzle though.

[karai17]

Awesome! I look very forward to this.

[karai17]

Awesome! I look very forward to this.

[octref]

@karai17
Just wondering. You mentioned you need to edit 3D model files. How do those files look like?
And, as for syntax-highlighting on them, do you

• Absolutely need it
• Don't want it at all
• Can live without it but wish to have it

[octref]

@karai17
Just wondering. You mentioned you need to edit 3D model files. How do those files look like?
And, as for syntax-highlighting on them, do you

• Absolutely need it
• Don't want it at all
• Can live without it but wish to have it

[karai17]

@octref

Here is an example of a several megabyte IQE 3D model file. The weird indenting is caused by Blender's IQE exporter, it is not part of the spec. Here is the IQE spec.

Since IQE is a data format and not a language format, I wouldn't really expect it to have syntax highlighting. Maybe it would be neat to write a package that did highlight it, but I don't think it would be all that useful.

There are some language files that can reach huge sizes such as SQL dumps that would benefit from syntax highlighting, though.

[karai17]

@octref

Here is an example of a several megabyte IQE 3D model file. The weird indenting is caused by Blender's IQE exporter, it is not part of the spec. Here is the IQE spec.

Since IQE is a data format and not a language format, I wouldn't really expect it to have syntax highlighting. Maybe it would be neat to write a package that did highlight it, but I don't think it would be all that useful.

There are some language files that can reach huge sizes such as SQL dumps that would benefit from syntax highlighting, though.

[octref]

@karai17

Got it.
Currently Atom is using converted textmate grammar bundles. Once the support for large file is there, you should be able to write a custom syntax package and get syntax-highlighting in Atom for the 3D model files.

[octref]

@karai17

Got it.
Currently Atom is using converted textmate grammar bundles. Once the support for large file is there, you should be able to write a custom syntax package and get syntax-highlighting in Atom for the 3D model files.

[karai17]

Awesome! One thing I noticed recently is that, at least in the language-lua package, some syntax is highlighted in a way that I don't believe it should be, and I am not sure if that is the package itself or a symptom of how Atom colours syntax.

An example would be that if I use a Lua core function such as print(), Atom highlights that text. But if I type it incorrectly, such as pirnt(), it still highlights it. What seems to be happening is the package picks up on () and highlights it as a function. With sublime I believe only core functions were highlighted, or at the very least core and user functions had separate colours.

I know that this falls outside of the scope of this particular issue, but I was wondering if it had to do with Atom's syntax highlighting or if it was just an oversight on the package itself?

[karai17]

Awesome! One thing I noticed recently is that, at least in the language-lua package, some syntax is highlighted in a way that I don't believe it should be, and I am not sure if that is the package itself or a symptom of how Atom colours syntax.

An example would be that if I use a Lua core function such as print(), Atom highlights that text. But if I type it incorrectly, such as pirnt(), it still highlights it. What seems to be happening is the package picks up on () and highlights it as a function. With sublime I believe only core functions were highlighted, or at the very least core and user functions had separate colours.

I know that this falls outside of the scope of this particular issue, but I was wondering if it had to do with Atom's syntax highlighting or if it was just an oversight on the package itself?

[LPGhatguy]

I've seen some interesting behavior with themes too as far as Lua core functions go.

Here's a side-by-side comparison of some code. Sublime Text 3 is on the left (correct behavior), while Atom is on the right, using the same theme as ST3 using the theme importer from a couple versions back.

http://i.imgur.com/V9b5GWh.png

(It doesn't help that I have each editor in a different font and font size at the moment)

[LPGhatguy]

I've seen some interesting behavior with themes too as far as Lua core functions go.

Here's a side-by-side comparison of some code. Sublime Text 3 is on the left (correct behavior), while Atom is on the right, using the same theme as ST3 using the theme importer from a couple versions back.

http://i.imgur.com/V9b5GWh.png

(It doesn't help that I have each editor in a different font and font size at the moment)

[octref]

@karai17
If you write a test.lua file with the following two lines:

print()
pirnt()

Use Ctrl-Shift-I to open the inspector, and Ctrl-F find the corresponding DOM node:
For print(), it's:
<span class="support function lua">print</span>
For pirnt(), it's:
<span class="support function any-method lua">pirnt</span>
On the inspector, you can find they are all colored by the .support.function class, which corresponds to this LESS rule in Atom-Dark theme:
https://github.com/atom/atom-dark-syntax/blob/master/index.less#L218-L220
The grammar:
https://github.com/FireZenk/language-lua/blob/d1bbd1805cfc1e0cfaeb60b426133f72b442d2e3/grammars/lua.cson#L131-L133

So if you are unhappy about this grammar syntax, you can open an issue to language-lua, asking them to parse print differently.

For example, python parses print as keyword.other.python and gives it a pretty azure :-)
<span class="keyword other python">print</span>

But

@karai17 , @LPGhatguy
Let's stop going deep on syntax here. Many people are subscribed to this issue to get progress on large file support.

[octref]

@karai17
If you write a test.lua file with the following two lines:

print()
pirnt()

Use Ctrl-Shift-I to open the inspector, and Ctrl-F find the corresponding DOM node:
For print(), it's:
<span class="support function lua">print</span>
For pirnt(), it's:
<span class="support function any-method lua">pirnt</span>
On the inspector, you can find they are all colored by the .support.function class, which corresponds to this LESS rule in Atom-Dark theme:
https://github.com/atom/atom-dark-syntax/blob/master/index.less#L218-L220
The grammar:
https://github.com/FireZenk/language-lua/blob/d1bbd1805cfc1e0cfaeb60b426133f72b442d2e3/grammars/lua.cson#L131-L133

So if you are unhappy about this grammar syntax, you can open an issue to language-lua, asking them to parse print differently.

For example, python parses print as keyword.other.python and gives it a pretty azure :-)
<span class="keyword other python">print</span>

But

@karai17 , @LPGhatguy
Let's stop going deep on syntax here. Many people are subscribed to this issue to get progress on large file support.

[erikdonohoo]

Super happy to hear this is being worked on. Its not just large binary files that are issues here. Any js framework (like angular, ember) end up being too large for atom to handle. It doesn't seem like an edge case to occasionally need to look at the source of some 3rd party package you are using.

[erikdonohoo]

Super happy to hear this is being worked on. Its not just large binary files that are issues here. Any js framework (like angular, ember) end up being too large for atom to handle. It doesn't seem like an edge case to occasionally need to look at the source of some 3rd party package you are using.

[wpostma]

This editor is a toy, if it can't handle large files. I'll be watching to see if Atom matures in this area.

[wpostma]

This editor is a toy, if it can't handle large files. I'll be watching to see if Atom matures in this area.

[murphyj]

Agree with @erikdonohoo - this is not an edge case. You also can't open some crucial files for Ghost because it's ember.js backed (hence has large binary vendor files).

[murphyj]

Agree with @erikdonohoo - this is not an edge case. You also can't open some crucial files for Ghost because it's ember.js backed (hence has large binary vendor files).

[sduensin]

This is a HUGE problem. 2 meg is nothing these days. As much as I like Atom, this limitation makes it pretty useless for day to day work. :-(

[sduensin]

This is a HUGE problem. 2 meg is nothing these days. As much as I like Atom, this limitation makes it pretty useless for day to day work. :-(

[sceee]

I can support this - many files (let it be log files etc.) have more than 2 MB in size. This always makes you need another editor to open these files beside atom.

[sceee]

I can support this - many files (let it be log files etc.) have more than 2 MB in size. This always makes you need another editor to open these files beside atom.

[gpluta]

This limitation is really a dealbreaker when it comes to using atom... I hope this gets improved one day.

[gpluta]

This limitation is really a dealbreaker when it comes to using atom... I hope this gets improved one day.

[mindvox]

This should be considered a standard feature for a modern text editor. I would really love to see this feature added as Atom is such an amazing text editor.

[mindvox]

This should be considered a standard feature for a modern text editor. I would really love to see this feature added as Atom is such an amazing text editor.

[50Wliu]

This is currently being worked on. The Atom team recognizes that it's a big problem, but it also needs a big rewrite in order to make everything work. See https://github.com/atom/text-document for more details.

[50Wliu]

This is currently being worked on. The Atom team recognizes that it's a big problem, but it also needs a big rewrite in order to make everything work. See https://github.com/atom/text-document for more details.

[mindvox]

@50Wliu I can appreciate the work involved. Looking forward to seeing this in a future release 😄

[mindvox]

@50Wliu I can appreciate the work involved. Looking forward to seeing this in a future release 😄

[backspaces]

Could we pin this? The extraordinary atom/text-document should be shown to
all those concerned.

I say this because Atom is historic, like several JS/Html/Browser/Node
projects have been over the last few years.

Many of Atom's clients are simply looking for a coherent text editor.
That's fine. But they should also understand the JavaScript Revolution
background.

On Tue, Apr 7, 2015 at 6:19 PM, Karl Bateman notifications@github.com
wrote:

@50Wliu https://github.com/50Wliu I can appreciate the work involved.
Looking forward to seeing this in a future release [image: 😄]

—
Reply to this email directly or view it on GitHub
#307 (comment).

[backspaces]

Could we pin this? The extraordinary atom/text-document should be shown to
all those concerned.

I say this because Atom is historic, like several JS/Html/Browser/Node
projects have been over the last few years.

Many of Atom's clients are simply looking for a coherent text editor.
That's fine. But they should also understand the JavaScript Revolution
background.

On Tue, Apr 7, 2015 at 6:19 PM, Karl Bateman notifications@github.com
wrote:

@50Wliu https://github.com/50Wliu I can appreciate the work involved.
Looking forward to seeing this in a future release [image: 😄]

—
Reply to this email directly or view it on GitHub
#307 (comment).

[Tapefabrik]

+1

[Tapefabrik]

+1

[mika76]

I must admit these days the only thing keeping me from using Atom is the performance and the large file non-support. I can't wait to see this working 😁

[mika76]

I must admit these days the only thing keeping me from using Atom is the performance and the large file non-support. I can't wait to see this working 😁

[xpepermint]

+1

[xpepermint]

+1

[theksmith]

+100

[theksmith]

+100

[pixelchutes]

+2.0001 (MB) ...wait for it... 😉

UPDATE:
@thedaniel is right guys. Sorry for the spam / bad joke.

[pixelchutes]

+2.0001 (MB) ...wait for it... 😉

UPDATE:
@thedaniel is right guys. Sorry for the spam / bad joke.

[thedaniel]

This is actively being worked on and is a high priority, no reason to post +1 comments anymore - it just spams everyone watching the issue.

[thedaniel]

This is actively being worked on and is a high priority, no reason to post +1 comments anymore - it just spams everyone watching the issue.

[iamstarkov]

@thedaniel issues can be closed for commenting

[iamstarkov]

@thedaniel issues can be closed for commenting

[ilanbiala]

@thedaniel I think it would help if any other possible progress updates regarding perf. improvements were provided. If there aren't any, it would just be nice to know that it's happening by @nathansobo or you commenting in addition to adding the in progress label.

[ilanbiala]

@thedaniel I think it would help if any other possible progress updates regarding perf. improvements were provided. If there aren't any, it would just be nice to know that it's happening by @nathansobo or you commenting in addition to adding the in progress label.

[nathansobo]

It's happening. Check out the text-document repo. Out of the country for a conference which is consuming some attention, but it's been almost my sole focus. Not much more to say than the progress and write-ups you can observe there. I empathize with everyone who wants this because I want it too, and it's been challenging but we're definitely getting there.

[nathansobo]

It's happening. Check out the text-document repo. Out of the country for a conference which is consuming some attention, but it's been almost my sole focus. Not much more to say than the progress and write-ups you can observe there. I empathize with everyone who wants this because I want it too, and it's been challenging but we're definitely getting there.

[robhawkes]

@thedaniel @matmuchrapna: Yup, it's simple to lock the comments while still allowing contributors to discuss things.

[robhawkes]

@thedaniel @matmuchrapna: Yup, it's simple to lock the comments while still allowing contributors to discuss things.

[kylegoetz]

I disagree this is an edge case. There are popular third-party JS libraries that are greater than this limit. What motivates my finding this bug is that Telerik's Kendo library is too big to open in Atom unless you open the minified version, which is, of course, not useful at all.

I'm having to read the library in Firefox's debugger since I can't open it in Atom to read it. You can argue philosophy about whether I should be opening a file I don't intend to edit in Atom, but I might edit it if I need to and just consider it no longer a proper third party lib but rather a first party, but can't edit in Atom.

[kylegoetz]

I disagree this is an edge case. There are popular third-party JS libraries that are greater than this limit. What motivates my finding this bug is that Telerik's Kendo library is too big to open in Atom unless you open the minified version, which is, of course, not useful at all.

I'm having to read the library in Firefox's debugger since I can't open it in Atom to read it. You can argue philosophy about whether I should be opening a file I don't intend to edit in Atom, but I might edit it if I need to and just consider it no longer a proper third party lib but rather a first party, but can't edit in Atom.

[benogle]

I've locked this. We do believe this is an important case to handle. We are currently working on this. Please visit https://github.com/atom/text-document for more information.

[benogle]

I've locked this. We do believe this is an important case to handle. We are currently working on this. Please visit https://github.com/atom/text-document for more information.

[nathansobo]

Just an update here... in the interest of time we've decided to refocus our efforts on more incremental improvements.

• One large chunk of work from text-document, the MarkerIndex, is in the process of being integrated into our existing document model. This will improve Atom's performance in the presence of large numbers of markers, such as when visualizing numerous search results in large files. The pull request should be merged in the next few days.
• We're also working to reduce the memory consumption of our tokenized line representation.

We'll be searching for several more smaller wins like this, then redirect our attention back to some of the bigger changes researched in text-document.

Some ideas for other areas we plan on investigating in the next couple weeks:

• Looking for low hanging fruit in the initial load of a document.
• Looking for low hanging fruit in parser performance.
• Exploring batching display buffer updates for changes within a single transaction.

Making Atom performant for files of all sizes remains a top priority. We'll continue to post updates here.

[nathansobo]

Just an update here... in the interest of time we've decided to refocus our efforts on more incremental improvements.

• One large chunk of work from text-document, the MarkerIndex, is in the process of being integrated into our existing document model. This will improve Atom's performance in the presence of large numbers of markers, such as when visualizing numerous search results in large files. The pull request should be merged in the next few days.
• We're also working to reduce the memory consumption of our tokenized line representation.

We'll be searching for several more smaller wins like this, then redirect our attention back to some of the bigger changes researched in text-document.

Some ideas for other areas we plan on investigating in the next couple weeks:

• Looking for low hanging fruit in the initial load of a document.
• Looking for low hanging fruit in parser performance.
• Exploring batching display buffer updates for changes within a single transaction.

Making Atom performant for files of all sizes remains a top priority. We'll continue to post updates here.

[nathansobo]

There's still more to do here, but a basic fix for the 80% case of loading large files is now on master. Before 1.0 I'd like to fix a few performance hiccups when moving the cursor in huge files. Post 1.0 I'd like to:

• Syntax highlight in a background process so we can highlight huge files without hiccups.
• Show a progress bar while performing I/O and computing initial metadata while loading a large file.
• Drop the memory overhead another order of magnitude.
• Support folds in large files

We can create separate issues for all of those, but I'm going to close this one. Atom can now load and edit large files. Don't worry, we'll keep working on refining it.

[nathansobo]

There's still more to do here, but a basic fix for the 80% case of loading large files is now on master. Before 1.0 I'd like to fix a few performance hiccups when moving the cursor in huge files. Post 1.0 I'd like to:

• Syntax highlight in a background process so we can highlight huge files without hiccups.
• Show a progress bar while performing I/O and computing initial metadata while loading a large file.
• Drop the memory overhead another order of magnitude.
• Support folds in large files

We can create separate issues for all of those, but I'm going to close this one. Atom can now load and edit large files. Don't worry, we'll keep working on refining it.