Crash when untangling large amount of files #7

Closed
beatcracker opened this Issue Feb 1, 2015 · 1 comment

Comments

Projects
None yet
2 participants
@beatcracker
Contributor

beatcracker commented Feb 1, 2015

Issue

I've tried to process a heavily tangled SVN repo and svndumpfilterIN crashes at the same point with this error:

---- Working on Input Revision 1235 (Renumber Rev: 1) ----
Warning: svnlook is required to pull missing files
['svnlook', '--full-paths', '-r', '1234', 'tree', '.\\repo', 'tags']
['svnlook', '-r', '1234', 'cat', '.\\repo', 'tags/x.x.xx/src/long/path/to/source/file/csharpfile.cs']

# Skipped about 1066 similar lines

['svnlook', '-r', '1234', 'cat', '.\\repo', 'tags/x.x.xx/src/long/path/to/source/file/anothercsharpfile.cs']
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "svndumpfilter.py", line 805, in <module>
    main()
  File "svndumpfilter.py", line 800, in main
    parse_dump(input_dump, opt.output_dump, matches, include, opt)
  File "svndumpfilter.py", line 713, in parse_dump
    handle_exclude_to_include(node_seg, output_file, flags, opt)
  File "svndumpfilter.py", line 632, in handle_exclude_to_include
    node_seg.head[NODE_COPYFROM_REV], opt.repo, opt.debug)
  File "svndumpfilter.py", line 454, in handle_missing_directory
    handle_missing_file(d_file, file_from, file_dest, rev_num, repo_path, debug)
  File "svndumpfilter.py", line 427, in handle_missing_file
    file_body = run_svnlook_command("cat", rev_num, repo_path, from_path, None, debug)
  File "svndumpfilter.py", line 413, in run_svnlook_command
    process = subprocess.Popen(command_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "X:\Path\To\PyPy\lib-python\2.7\subprocess.py", line 745, in __init__
    self.stderr = os.fdopen(errread, 'rb', bufsize)
OSError: [Errno 9] Bad file descriptor

Environment

  • OS: Windows Server 2012 x64
  • PyPy.exe --version:
Python 2.7.8 (f5dcc2477b97, Sep 18 2014, 14:30:11)
[PyPy 2.4.0 with MSC v.1500 32 bit]
  • svnlook --version:
svnlook, version 1.8.11 (r1643975)
compiled Jan 30 2015, 05:10:28 on x86-microsoft-windows

Copyright (C) 2014 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/

The following repository back-end (FS) modules are available:

fs_fs : Module for working with a plain file (FSFS) repository.

Workaround

It seems that it's suffering from the known Python issue, described here: Subprocess Hanging: PIPE is your enemy.

I was able to workaround this problem by using [tempfile.TemporaryFile] as article suggests (see example below). After this, the svndumpfilterIN was able to complete filtering without crashes.

Example

Please note, that it's my first time Python code and it's probably a far cry from a proper solution. Also, be aware that I was forced to drop stderr redirection completely, so we wouldn't be able to catch svnlook error message, only exit code. A proper fix probably will involve spawning of two reading threads to avoid deadlocks, but this is a way over my head for now.

Add following import to the beginning of svndumpfilterIN.py

from tempfile import TemporaryFile

In run_svnlook_command replace this code

  process = subprocess.Popen(command_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  out, err = process.communicate()
  if err:
    raise SVNLookError(err)
  else:
    return out

with

  with TemporaryFile() as temp_file:
    process = subprocess.Popen(command_list, stdout=temp_file)
    err = process.wait()
    if err:
      raise SVNLookError(err)
    else:
      temp_file.flush()
      temp_file.seek(0)
      out = temp_file.read()
      return out
@freeamac

This comment has been minimized.

Show comment
Hide comment
@freeamac

freeamac Dec 21, 2015

Collaborator

Believe you have provided the fix in the request I just pulled in. Thanks!

Collaborator

freeamac commented Dec 21, 2015

Believe you have provided the fix in the request I just pulled in. Thanks!

@freeamac freeamac closed this Dec 21, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment