Skip to content
Commits on Oct 21, 2007
  1. @spearce

    Define compat version of mkdtemp for systems lacking it

    spearce committed Oct 20, 2007
    Solaris 9 doesn't have mkdtemp() so we need to emulate it for the
    rsync transport implementation.  Since Solaris 9 is lacking this
    function we can also reasonably assume it is not available on
    Solaris 8 either.  The new Makfile definition NO_MKDTEMP can be
    set to enable the git compat version.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Commits on Apr 25, 2007
  1. @spearce

    Actually handle some-low memory conditions

    spearce committed with Junio C Hamano Apr 25, 2007
    Tim Ansell discovered his Debian server didn't permit git-daemon to
    use as much memory as it needed to handle cloning a project with
    a 128 MiB packfile.  Filtering the strace provided by Tim of the
    rev-list child showed this gem of a sequence:
    
      open("./objects/pack/pack-*.pack", O_RDONLY|O_LARGEFILE <unfinished ...>
      <... open resumed> )              = 5
    
    OK, so the packfile is fd 5...
    
      mmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 5, 0 <unfinished ...>
       <... mmap2 resumed> )             = 0xb5e2d000
    
    and we mapped one 32 MiB window from it at position 0...
    
       mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...>
       <... mmap2 resumed> )             = -1 ENOMEM (Cannot allocate memory)
    
    And we asked for another window further into the file.  But got
    denied.  In Tim's case this was due to a resource limit on the
    git-daemon process, and its children.
    
    Now where are we in the code?  We're down inside use_pack(),
    after we have called unuse_one_window() enough times to make sure
    we stay within our allowed maximum window size.  However since we
    didn't unmap the prior window at 0xb5e2d000 we aren't exceeding
    the current limit (which probably was just the defaults).
    
    But we're actually down inside xmmap()...
    
    So we release the window we do have (by calling release_pack_memory),
    assuming there is some memory pressure...
    
       munmap(0xb5e2d000, 33554432 <unfinished ...>
       <... munmap resumed> )            = 0
       close(5 <unfinished ...>
       <... close resumed> )             = 0
    
    And that was the last window in this packfile.  So we closed it.
    Way to go us.  Our xmmap did not expect release_pack_memory to
    close the fd its about to map...
    
       mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...>
       <... mmap2 resumed> )             = -1 EBADF (Bad file descriptor)
    
    And so the Linux kernel happily tells us f' off.
    
       write(2, "fatal: ", 7 <unfinished ...>
       <... write resumed> )             = 7
       write(2, "Out of memory? mmap failed: Bad "..., 47 <unfinished ...>
       <... write resumed> )             = 47
    
    And we report the bad file descriptor error, and not the ENOMEM,
    and die, claiming we are out of memory.  But actually that mmap
    should have succeeded, as we had enough memory for that window,
    seeing as how we released the prior one.
    
    Originally when I developed the sliding window mmap feature I had
    this exact same bug in fast-import, and I dealt with it by handing
    in the struct packed_git* we want to open the new window for, as the
    caller wasn't prepared to reopen the packfile if unuse_one_window
    closed it.  The same is true here from xmmap, but the caller doesn't
    have the struct packed_git* handy.  So I'm using the file descriptor
    instead to perform the same test.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Commits on Mar 7, 2007
  1. @spearce

    Cast 64 bit off_t to 32 bit size_t

    spearce committed with Junio C Hamano Mar 6, 2007
    Some systems have sizeof(off_t) == 8 while sizeof(size_t) == 4.
    This implies that we are able to access and work on files whose
    maximum length is around 2^63-1 bytes, but we can only malloc or
    mmap somewhat less than 2^32-1 bytes of memory.
    
    On such a system an implicit conversion of off_t to size_t can cause
    the size_t to wrap, resulting in unexpected and exciting behavior.
    Right now we are working around all gcc warnings generated by the
    -Wshorten-64-to-32 option by passing the off_t through xsize_t().
    
    In the future we should make xsize_t on such problematic platforms
    detect the wrapping and die if such a file is accessed.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
  2. @spearce

    Use off_t when we really mean a file offset.

    spearce committed with Junio C Hamano Mar 6, 2007
    Not all platforms have declared 'unsigned long' to be a 64 bit value,
    but we want to support a 64 bit packfile (or close enough anyway)
    in the near future as some projects are getting large enough that
    their packed size exceeds 4 GiB.
    
    By using off_t, the POSIX type that is declared to mean an offset
    within a file, we support whatever maximum file size the underlying
    operating system will handle.  For most modern systems this is up
    around 2^60 or higher.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Commits on Jan 6, 2007
  1. @spearce

    Increase packedGit{Limit,WindowSize} on 64 bit systems.

    spearce committed with Junio C Hamano Jan 4, 2007
    If we have a 64 bit address space we can easily afford to commit
    a larger amount of virtual address space to pack file access.
    So on these platforms we should increase the default settings of
    core.packedGit{Limit,WindowSize} to something that will better
    handle very large projects.
    
    Thanks to Andy Whitcroft for pointing out that we can safely
    increase these defaults on such systems.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Commits on Dec 29, 2006
  1. @spearce

    Replace mmap with xmmap, better handling MAP_FAILED.

    spearce committed with Junio C Hamano Dec 24, 2006
    In some cases we did not even bother to check the return value of
    mmap() and just assume it worked.  This is bad, because if we are
    out of virtual address space the kernel returned MAP_FAILED and we
    would attempt to dereference that address, segfaulting without any
    real error output to the user.
    
    We are replacing all calls to mmap() with xmmap() and moving all
    MAP_FAILED checking into that single location.  If a mmap call
    fails we try to release enough least-recently-used pack windows
    to possibly succeed, then retry the mmap() attempt.  If we cannot
    mmap even after releasing pack memory then we die() as none of our
    callers have any reasonable recovery strategy for a failed mmap.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
  2. @spearce

    Release pack windows before reporting out of memory.

    spearce committed with Junio C Hamano Dec 24, 2006
    If we are about to fail because this process has run out of memory we
    should first try to automatically control our appetite for address
    space by releasing enough least-recently-used pack windows to gain
    back enough memory such that we might actually be able to meet the
    current allocation request.
    
    This should help users who have fairly large repositories but are
    working on systems with relatively small virtual address space.
    Many times we see reports on the mailing list of these users running
    out of memory during various Git operations.  Dynamically decreasing
    the amount of pack memory used when the demand for heap memory is
    increasing is an intelligent solution to this problem.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
  3. @spearce

    Default core.packdGitWindowSize to 1 MiB if NO_MMAP.

    spearce committed with Junio C Hamano Dec 24, 2006
    If the compiler has asked us to disable use of mmap() on their
    platform then we are forced to use git_mmap and its emulation via
    pread.  In this case large (e.g. 32 MiB) windows for pack access
    are simply too big as a command will wind up reading a lot more
    data than it will ever need, significantly reducing response time.
    
    To prevent a high latency when NO_MMAP has been selected we now
    use a default of 1 MiB for core.packedGitWindowSize.  Credit goes
    to Linus and Junio for recommending this more reasonable setting.
    
    [jc: upcased the name of the symbolic constant, and made another
     hardcoded constant into a symbolic constant while at it. ]
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Commits on Dec 24, 2006
  1. @spearce

    Rename gitfakemmap to git_mmap.

    spearce committed with Junio C Hamano Dec 24, 2006
    This minor cleanup was suggested by Johannes Schindelin.
    
    The mmap is still fake in the sense that we don't support PROT_WRITE
    or MAP_SHARED with external modification at all, but that hasn't
    stopped us from using mmap() thoughout the Git code.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Commits on Dec 22, 2006
  1. @spearce

    Introduce a global level warn() function.

    spearce committed with Junio C Hamano Dec 21, 2006
    Like the existing error() function the new warn() function can be
    used to describe a situation that probably should not be occuring,
    but which the user (and Git) can continue to work around without
    running into too many problems.
    
    An example situation is a bad commit SHA1 found in a reflog.
    Attempting to read this record out of the reflog isn't really an
    error as we have skipped over it in the past.
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Commits on Sep 2, 2006
  1. @spearce

    Replace uses of strdup with xstrdup.

    spearce committed with Junio C Hamano Sep 2, 2006
    Like xmalloc and xrealloc xstrdup dies with a useful message if
    the native strdup() implementation returns NULL rather than a
    valid pointer.
    
    I just tried to use xstrdup in new code and found it to be missing.
    However I expected it to be present as xmalloc and xrealloc are
    already commonly used throughout the code.
    
    [jc: removed the part that deals with last_XXX, which I am
     finding more and more dubious these days.]
    
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
    Signed-off-by: Junio C Hamano <junkio@cox.net>
Something went wrong with that request. Please try again.