Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lots of leaked resources and unclosed files when running tests #62

Open
rotu opened this issue Mar 11, 2020 · 1 comment
Open

Lots of leaked resources and unclosed files when running tests #62

rotu opened this issue Mar 11, 2020 · 1 comment

Comments

@rotu
Copy link

rotu commented Mar 11, 2020

Ros2 Foxy on Ubuntu Focal with Python3.8

 colcon test --packages-select osrf_pycommon                  
Starting >>> osrf_pycommon
[1.106s] ERROR:colcon.colcon_core.task.python.test:Exception in Python testing step extension 'pytest': 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/dan/.local/lib/python3.8/site-packages/colcon_core/task/python/test/__init__.py", line 54, in test
    matched = extension.match(self.context, env, setup_py_data)
  File "/home/dan/.local/lib/python3.8/site-packages/colcon_core/task/python/test/pytest.py", line 49, in match
    return has_test_dependency(setup_py_data, 'pytest')
  File "/home/dan/.local/lib/python3.8/site-packages/colcon_core/task/python/test/__init__.py", line 214, in has_test_dependency
    for d in tests_require:
TypeError: 'NoneType' object is not iterable

--- stderr: osrf_pycommon                   
test_call_prepare_arguments (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... /opt/ros/master/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:44: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  arguments, _, _, defaults = inspect.getargspec(func)
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_cli_utils/test_verb_pattern.py:69: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(ValueError, 'one or two parameters'):
/opt/ros/master/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:55: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  ', '.join(inspect.getargspec(func)[0])))
ok
test_split_arguments_by_verb (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... ok
test_async_execute_process_no_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908384 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908386 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908388 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908391 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_with_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test__execute_process_nopty (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/ros/master/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__process_incomming_lines (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty_combined (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty_combined_unbuffered (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test__execute_process_pty_unbuffered (tests.unit.test_process_utils.test_execute_process_pty.TestProcessUtilsExecuteNoPty) ... ok
test_which (tests.unit.test_process_utils.test_impl.TestProcessUtilsImpl) ... ok
test_remove_ansi_escape_sequences (tests.unit.test_terminal_color.test_ansi_re.TestTerminalColorAnsiRe) ... ok
test_remove_ansi_escape_sequences_false_positives (tests.unit.test_terminal_color.test_ansi_re.TestTerminalColorAnsiRe) ... ok
test_split_by_ansi_escape_sequence (tests.unit.test_terminal_color.test_ansi_re.TestTerminalColorAnsiRe) ... ok
test_ansi (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_enable_and_disable_ansi_color_substitution_globally (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_format_color (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_get_ansi_dict (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_sanitize (tests.unit.test_terminal_color.test_impl.TestTerminalColorImpl) ... ok
test_is_tty (tests.unit.test_terminal_utils.TestTerminalUtils) ... /usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908394 returncode=0 stdin=<_UnixWritePipeTransport closed fd=11 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908396 returncode=0 stdin=<_UnixWritePipeTransport closed fd=9 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908398 returncode=0 stdin=<_UnixWritePipeTransport closed fd=11 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/asyncio/base_subprocess.py:125: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=908401 returncode=0 stdin=<_UnixWritePipeTransport closed fd=9 closed>>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok

----------------------------------------------------------------------
Ran 29 tests in 0.261s

OK
---
Finished <<< osrf_pycommon [0.60s]
@rotu rotu changed the title Lots of leaked resources and unclosed files Lots of leaked resources and unclosed files when running tests Mar 11, 2020
@rotu
Copy link
Author

rotu commented Apr 14, 2020

still an issue https://ci.ros2.org/job/ci_osx/8271/consoleText:

osrf_pycommon
[94.681s] WARNING:colcon.colcon_core.verb.test:Ignored '--retest-until-pass' for package 'osrf_pycommon' since 'unittest' does not support the usage
test_call_prepare_arguments (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:44: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  arguments, _, _, defaults = inspect.getargspec(func)
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_cli_utils/test_verb_pattern.py:69: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(ValueError, 'one or two parameters'):
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/osrf_pycommon/cli_utils/verb_pattern.py:55: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  ', '.join(inspect.getargspec(func)[0])))
ok
test_split_arguments_by_verb (tests.unit.test_cli_utils.test_verb_pattern.TestCliUtilsVerbPattern) ... ok
test_async_execute_process_no_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59964 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59965 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59966 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_no_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py:126: ResourceWarning: unclosed transport <_UnixSubprocessTransport pid=59967 returncode=0 stdin=<_UnixWritePipeTransport closed fd=7 closed> stdout=<_UnixReadPipeTransport closed fd=8 closed>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_async_execute_process_with_emulation_shell_false (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_false_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test_async_execute_process_with_emulation_shell_true_combined (tests.unit.test_process_utils.test_async_execute_process.TestProcessUtilsAsyncExecuteProcess) ... ok
test__execute_process_nopty (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:127: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:104: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_combined_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:58: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test__execute_process_nopty_unbuffered (tests.unit.test_process_utils.test_execute_process_nopty.TestProcessUtilsExecuteNoPty) ... /Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedWriter name=7>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/osrf/jenkins-agent/workspace/ci_osx/ws/src/osrf/osrf_pycommon/tests/unit/test_process_utils/test_execute_process_nopty.py:81: ResourceWarning: unclosed file <_io.BufferedReader name=10>
  break
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok

@rotu rotu mentioned this issue May 6, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rotu