Skip to content
Browse files

Git.pm: Use stream-like writing in cat_blob()

This commit fixes the issue with the handling of large files causing an
'Out of memory' perl exception. Instead of reading and writing the whole
blob at once now the blob is written in small pieces.

The problem was raised and discussed in this mail to the msysGit mailing
list: http://thread.gmane.org/gmane.comp.version-control.msysgit/12080

Signed-off-by: Gregor Uhlenheuer <kongo2002@googlemail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
  • Loading branch information...
1 parent 02f77e8 commit 06d0a8cd505b01f926ab1de21c3c57aa681c5dfe @kongo2002 kongo2002 committed with dscho Feb 18, 2011
Showing with 7 additions and 8 deletions.
  1. +7 −8 perl/Git.pm
View
15 perl/Git.pm
@@ -896,22 +896,26 @@ sub cat_blob {
}
my $size = $1;
-
- my $blob;
my $bytesRead = 0;
while (1) {
+ my $blob;
my $bytesLeft = $size - $bytesRead;
last unless $bytesLeft;
my $bytesToRead = $bytesLeft < 1024 ? $bytesLeft : 1024;
- my $read = read($in, $blob, $bytesToRead, $bytesRead);
+ my $read = read($in, $blob, $bytesToRead);
unless (defined($read)) {
$self->_close_cat_blob();
throw Error::Simple("in pipe went bad");
}
$bytesRead += $read;
+
+ unless (print $fh $blob) {
+ $self->_close_cat_blob();
+ throw Error::Simple("couldn't write to passed in filehandle");
+ }
}
# Skip past the trailing newline.
@@ -926,11 +930,6 @@ sub cat_blob {
throw Error::Simple("didn't find newline after blob");
}
- unless (print $fh $blob) {
- $self->_close_cat_blob();
- throw Error::Simple("couldn't write to passed in filehandle");
- }
-
return $size;
}

2 comments on commit 06d0a8c

@troth
troth commented on 06d0a8c Mar 30, 2012

You should try to get this committed up stream.

At work we have to use svn, but we have a git mirror of the svn repo. Someone decided it would be a good idea to commit a 7GB file in to svn which caused the 'git svn fetch' to fail on the commit with the 7GB file. Your patch used in combination with git-1.7.9.4 allowed me to get the mirror updated with the large file. I'm running on a 64 bit Ubuntu, 11.10 system, while the git mirror server is a 64 bit RHEL-5.x system.

I did a little digging in the git source and there was changes made in around 1.7.7.2 to improve large file support, so they made be conducive to accepting you patch.

@dscho
MSysGit - the development behind Git for Windows member
dscho commented on 06d0a8c Mar 30, 2012

Hey, if you want to submit it, be my guest. But note that just mentioning my name, let alone passing on my sign-off, is prone to set off all kinds of unreasonable objections (see the mailing list archive for a record).

Hence you might want to submit the original patch without my sign-off.

Please sign in to comment.
Something went wrong with that request. Please try again.