Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Avoid buffering entire files #58

Open
jdumas opened this Issue Apr 2, 2013 · 6 comments

Comments

Projects
None yet
3 participants

jdumas commented Apr 2, 2013

Hi!

One issue that's been slitghly annoying me when using vimpager is that whenever I try to open a big file with it, the program hangs until it has seemingly buffered the whole file and then display it. I don't know if it's a desired feature or not, but as less and vim open immediatly any large file I throw at them I suppose it's an issue to be fixed. Another inconvenience comes when we pipe stuff to vimpager, because it also hangs till the program writing to stdout terminates to display stuff (and if the said program is quite slow, it might be painful to just wait).

Steps to reproduce the issue:
dd if=/dev/urandom bs=1M count=40 | base64 | vimpager
Versus:
dd if=/dev/urandom bs=1M count=40 | base64 | less

Owner

rkitover commented Apr 2, 2013

It writes the input to a temp file before opening, because vim doesn't work very well for pipes.

I'm not sure what would be a good way to fix this, perhaps by piping the output to the file in the background and having a vim function poll the file for updates.

On Tuesday, April 2, 2013 at 6:33 PM, Dumas Jérémie wrote:

Hi!
One issue that's been slitghly annoying me when using vimpager is that whenever I try to open a big file with it, the program hangs until it has seemingly buffered the whole file and then display it. I don't know if it's a desired feature or not, but as less and vim open immediatly any large file I throw at them I suppose it's an issue to be fixed. Another inconvenience comes when we pipe stuff to vimpager, because it also hangs till the program writing to stdout terminates to display stuff (and if the said program is quite slow, it might be painful to just wait).
Steps to reproduce the issue:
dd if=/dev/urandom bs=1M count=40 | base64 | vimpager
Versus:
dd if=/dev/urandom bs=1M count=40 | base64 | less


Reply to this email directly or view it on GitHub (#58).

jdumas commented Apr 3, 2013

I thought that a simple cat bla.txt | vim - would do the job when reading something from stdin? And even when reading a regular file, I don't think creating a copy in /tmp is a good idea (imagine your file is ~50GB big, you'd be screwed).

I'd suggest to distinguish when something is piped to vimpager (and use vim - or whatever), and when reading a normal "file", where one should avoid creating a copy in /tmp.

Owner

rkitover commented Apr 3, 2013

It works, sort of. vimpager used to do this, but I switched to the /tmp directory because that works better.

You can try piping a file to vim to see how that works, let me know what you find. I will experiment with it as well.

As for /tmp, I'm not sure there's any other place to portably put this file.

On Wednesday, April 3, 2013 at 6:29 AM, Dumas Jérémie wrote:

I thought that a simple cat bla.txt | vim - would do the job when reading something from stdin? And even when reading a regular file, I don't think creating a copy in /tmp is a good idea (imagine your file is ~50GB big, you'd be screwed).
I'd suggest to distinguish when something is piped to vimpager (and use vim - or whatever), and when reading a normal "file", where one should avoid creating a copy in /tmp.


Reply to this email directly or view it on GitHub (#58 (comment)).

Collaborator

lucc commented Apr 15, 2016

What is the reason that existing files are copied to /tmp? When I execute vimpager ~/notes.mkd and then press CTRLg inside vimpager it prints "/tmp/vimpager_30576/notes.mkd" line 6 of 311 --1%-- col 1 in the status line. If there is no explicit reason I can send a PR to change that behaviour (at least for existing files).

Owner

rkitover commented Apr 15, 2016

@lucc this is on my TODO list, and should be easier to do now after all the refactoring I've been doing lately. The problem with paging STDIN is going to be much more complex however. If you want to give it a go (for the former) then by all means!

Collaborator

lucc commented Apr 22, 2016 edited

Files that can be opened from disk (e.g. if they are not compressed) should not be copied to /tmp any longer after #164.

About reading from stdin: vim itself can only read from stdin if there are no file arguments. So this can only be implemented in vimpager if you have no file arguments as well. The use case vimpager file1 - file2 to read from files and stdin will still need to buffer stdin to /tmp.

About the initial speed report: If you pipe text into vim I think it also buffers the text in memory. I don't know what less does but it is also faster then plain vim if you read from stdin:

dd if=/dev/urandom bs=1M count=40 | base64 > base-data
time vim -u NONE -R +q - < base-data
time less           +q - < base-data

This means vimpager will never be close to the speed of less until vim itself is.

@lucc lucc added this to the 2.07 milestone Apr 25, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment