Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Crash when untangling large amount of files #7
I've tried to process a heavily tangled SVN repo and svndumpfilterIN crashes at the same point with this error:
---- Working on Input Revision 1235 (Renumber Rev: 1) ---- Warning: svnlook is required to pull missing files ['svnlook', '--full-paths', '-r', '1234', 'tree', '.\\repo', 'tags'] ['svnlook', '-r', '1234', 'cat', '.\\repo', 'tags/x.x.xx/src/long/path/to/source/file/csharpfile.cs'] # Skipped about 1066 similar lines ['svnlook', '-r', '1234', 'cat', '.\\repo', 'tags/x.x.xx/src/long/path/to/source/file/anothercsharpfile.cs'] Traceback (most recent call last): File "app_main.py", line 75, in run_toplevel File "svndumpfilter.py", line 805, in <module> main() File "svndumpfilter.py", line 800, in main parse_dump(input_dump, opt.output_dump, matches, include, opt) File "svndumpfilter.py", line 713, in parse_dump handle_exclude_to_include(node_seg, output_file, flags, opt) File "svndumpfilter.py", line 632, in handle_exclude_to_include node_seg.head[NODE_COPYFROM_REV], opt.repo, opt.debug) File "svndumpfilter.py", line 454, in handle_missing_directory handle_missing_file(d_file, file_from, file_dest, rev_num, repo_path, debug) File "svndumpfilter.py", line 427, in handle_missing_file file_body = run_svnlook_command("cat", rev_num, repo_path, from_path, None, debug) File "svndumpfilter.py", line 413, in run_svnlook_command process = subprocess.Popen(command_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE) File "X:\Path\To\PyPy\lib-python\2.7\subprocess.py", line 745, in __init__ self.stderr = os.fdopen(errread, 'rb', bufsize) OSError: [Errno 9] Bad file descriptor
It seems that it's suffering from the known Python issue, described here: Subprocess Hanging: PIPE is your enemy.
I was able to workaround this problem by using [tempfile.TemporaryFile] as article suggests (see example below). After this, the svndumpfilterIN was able to complete filtering without crashes.
Please note, that it's my first time Python code and it's probably a far cry from a proper solution. Also, be aware that I was forced to drop stderr redirection completely, so we wouldn't be able to catch svnlook error message, only exit code. A proper fix probably will involve spawning of two reading threads to avoid deadlocks, but this is a way over my head for now.
Add following import to the beginning of svndumpfilterIN.py
from tempfile import TemporaryFile
In run_svnlook_command replace this code
process = subprocess.Popen(command_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out, err = process.communicate() if err: raise SVNLookError(err) else: return out
with TemporaryFile() as temp_file: process = subprocess.Popen(command_list, stdout=temp_file) err = process.wait() if err: raise SVNLookError(err) else: temp_file.flush() temp_file.seek(0) out = temp_file.read() return out