New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.11.* coverage plugin causes issues during build deleting sys modules #211

Closed
MirkoRossini opened this Issue Sep 23, 2015 · 23 comments

Comments

Projects
None yet
4 participants
@MirkoRossini
Contributor

MirkoRossini commented Sep 23, 2015

Hello,

I think the new coverage plugin should be refactored, as it's a potential source of problems.
Looks like it's deleting a bunch of global modules, sparing only the ones that it consider "essential".
We have a custom python installation, the coverage plugin gets a bit confused and considers non essential things like datetime. We end up with errors like:

Traceback (most recent call last):
File "/tmp/test/env-infra/bin/pyb", line 25, in
sys.exit(pybuilder.cli.main(*sys.argv[1:]))
File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 366, in main
end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'

Because of this issue we are unable to update to 0.11.1
I know there should be a way to fix on our side by reverse-engineering your code and put modules in strategic locations, but honestly nothing in pybuilder (nothing in anything IMHO) should play with sys.modules, as it will cause issues that are a nightmare to debug.

Downgrading to 0.10.63 fixes the issue obviously, but the install task would be nice to have for our purposes.

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 23, 2015

Contributor

Thanks for the report. We're actually specifically attempting not to touch ANY modules that belong to Python by inspecting the installation. If you have a custom installation, we're probably introspecting it in a wrong way.

but honestly nothing in pybuilder (nothing in anything IMHO) should play with sys.modules

Generally, yes, and we don't, except for coverage, where not cleaning out non-essential modules produces wrong results.

Could you please run pyb -v -X and post the logs here?

Contributor

arcivanov commented Sep 23, 2015

Thanks for the report. We're actually specifically attempting not to touch ANY modules that belong to Python by inspecting the installation. If you have a custom installation, we're probably introspecting it in a wrong way.

but honestly nothing in pybuilder (nothing in anything IMHO) should play with sys.modules

Generally, yes, and we don't, except for coverage, where not cleaning out non-essential modules produces wrong results.

Could you please run pyb -v -X and post the logs here?

@arcivanov arcivanov added the bug label Sep 23, 2015

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Sep 23, 2015

Member

This looks like the finally block is masking some other error, but it implodes because datetime is gone. I don't think -X would help because of this (traceback.print_exc would also fail then).

Another possibility (if you can spare the time) is to insert

import pdb
pdb.set_trace()

in the finally block and look at the stack from there.

Member

mriehl commented Sep 23, 2015

This looks like the finally block is masking some other error, but it implodes because datetime is gone. I don't think -X would help because of this (traceback.print_exc would also fail then).

Another possibility (if you can spare the time) is to insert

import pdb
pdb.set_trace()

in the finally block and look at the stack from there.

@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 23, 2015

Contributor

Thanks for the quick reply. Here is the log.
I still think there should be a solution that doesn't require to mess with sys. I did see in the code the part where you walk in the sys lib directory trying to find the "essential" modules, but it still seems like a dangerous approach to me.

PyBuilder version 0.11.1
Build started at 2015-09-23 13:16:04
------------------------------------------------------------
[DEBUG] Loading project module from /tmp/test/build.py
[DEBUG] Loading plugin 'python.core'
[DEBUG] Trying to load builtin plugin 'python.core'
[DEBUG] Loading plugin 'core'
[DEBUG] Trying to load builtin plugin 'core'
[DEBUG] Found builtin plugin 'pybuilder.plugins.core_plugin'
[DEBUG] Found task 'clean' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'clean'
[DEBUG] Found task 'compile_sources' with required dependencies [<function prepare at 0x7f9bb7613c80>] and optional dependencies []
[DEBUG] Registering task 'compile_sources'
[DEBUG] Registering initializer 'init'
[DEBUG] Found task 'install' with required dependencies [<function publish at 0x7f9bb7613f50>] and optional dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found task 'package' with required dependencies [<function compile_sources at 0x7f9bb7613cf8>] and optional dependencies [<function run_unit_tests at 0x7f9bb7613d70>]
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Found task 'publish' with required dependencies [<function package at 0x7f9bb7613de8>] and optional dependencies [<function verify at 0x7f9bb7613ed8>]
[DEBUG] Registering task 'publish'
[DEBUG] Found task 'run_integration_tests' with required dependencies [<function package at 0x7f9bb7613de8>] and optional dependencies []
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'run_unit_tests' with required dependencies [<function compile_sources at 0x7f9bb7613cf8>] and optional dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found task 'verify' with required dependencies [<function run_integration_tests at 0x7f9bb7613e60>] and optional dependencies []
[DEBUG] Registering task 'verify'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.core_plugin'
[DEBUG] Registering initializer 'init_python_directories'
[DEBUG] Found task 'package' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Loading plugin 'python.unittest'
[DEBUG] Trying to load builtin plugin 'python.unittest'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Registering initializer 'init_test_source_directory'
[DEBUG] Found task 'run_unit_tests' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Loading plugin 'python.install_dependencies'
[DEBUG] Trying to load builtin plugin 'python.install_dependencies'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Found action check_pip_available
[DEBUG] Registering action 'check_pip_available'
[DEBUG] Found action create_install_log_directory
[DEBUG] Registering action 'create_install_log_directory'
[DEBUG] Registering initializer 'initialize_install_dependencies_plugin'
[DEBUG] Found task 'install_build_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install_build_dependencies'
[DEBUG] Found task 'install_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install_dependencies'
[DEBUG] Found task 'install_runtime_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install_runtime_dependencies'
[DEBUG] Found task 'list_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'list_dependencies'
[DEBUG] Loading plugin 'python.flake8'
[DEBUG] Trying to load builtin plugin 'python.flake8'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Found task 'analyze' with required dependencies ['prepare'] and optional dependencies []
[DEBUG] Registering task 'analyze'
[DEBUG] Found action assert_flake8_is_executable
[DEBUG] Registering action 'assert_flake8_is_executable'
[DEBUG] Registering initializer 'initialize_flake8_plugin'
[DEBUG] Loading plugin 'python.coverage'
[DEBUG] Trying to load builtin plugin 'python.coverage'
[DEBUG] Loading plugin 'analysis'
[DEBUG] Trying to load builtin plugin 'analysis'
[DEBUG] Found builtin plugin 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found task 'analyze' with required dependencies ['run_unit_tests'] and optional dependencies []
[DEBUG] Registering task 'analyze'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Registering initializer 'init_coverage_properties'
[DEBUG] Found action verify_coverage
[DEBUG] Registering action 'verify_coverage'
[DEBUG] Loading plugin 'python.distutils'
[DEBUG] Trying to load builtin plugin 'python.distutils'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Found action build_binary_distribution
[DEBUG] Registering action 'build_binary_distribution'
[DEBUG] Registering initializer 'initialize_distutils_plugin'
[DEBUG] Found task 'install' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found action write_manifest_file
[DEBUG] Registering action 'write_manifest_file'
[DEBUG] Found action write_setup_script
[DEBUG] Registering action 'write_setup_script'
[DEBUG] Have loaded plugins python.core, core, python.unittest, python.install_dependencies, python.flake8, python.coverage, analysis, python.distutils
[DEBUG] Registering initializer 'set_properties'
[DEBUG] Adding 'prepare' as a required dependency of task 'compile_sources'
[DEBUG] Adding 'publish' as a required dependency of task 'install'
[DEBUG] Adding 'compile_sources' as a required dependency of task 'package'
[DEBUG] Adding 'run_unit_tests' as an optional dependency of task 'package'
[DEBUG] Adding 'package' as a required dependency of task 'publish'
[DEBUG] Adding 'verify' as an optional dependency of task 'publish'
[DEBUG] Adding 'package' as a required dependency of task 'run_integration_tests'
[DEBUG] Adding 'compile_sources' as a required dependency of task 'run_unit_tests'
[DEBUG] Adding 'run_integration_tests' as a required dependency of task 'verify'
[DEBUG] Adding 'prepare' as a required dependency of task 'analyze'
[DEBUG] Adding 'run_unit_tests' as a required dependency of task 'analyze'
[DEBUG] Adding after action 'check_pip_available' for task 'prepare'
[DEBUG] Adding before action 'create_install_log_directory' for task 'install_build_dependencies'
[DEBUG] Adding before action 'create_install_log_directory' for task 'install_runtime_dependencies'
[DEBUG] Adding before action 'create_install_log_directory' for task 'install_dependencies'
[DEBUG] Adding after action 'assert_flake8_is_executable' for task 'prepare'
[DEBUG] Adding after action 'verify_coverage' for task 'analyze'
[DEBUG] Adding after action 'verify_coverage' for task 'verify'
[DEBUG] Adding before action 'build_binary_distribution' for task 'publish'
[DEBUG] Adding after action 'write_manifest_file' for task 'package'
[DEBUG] Adding after action 'write_setup_script' for task 'package'
[DEBUG] Verbose output enabled.

[DEBUG] Executing initializer 'init' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing initializer 'init_python_directories' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Executing initializer 'init_test_source_directory' from 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Executing initializer 'initialize_install_dependencies_plugin' from 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Executing initializer 'initialize_flake8_plugin' from 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Executing initializer 'init_coverage_properties' from 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Executing initializer 'initialize_distutils_plugin' from 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Executing initializer 'set_properties' from 'build'
[DEBUG] Project properties: 
                                 basedir : /tmp/test
                    coverage_break_build : True
                     coverage_exceptions : []
                           coverage_fork : None
                 coverage_reload_modules : None
                 coverage_threshold_warn : 70
                                dir_dist : $dir_target/dist/test-1.0.dev0
                        dir_dist_scripts : scripts
                        dir_install_logs : $dir_logs/install_dependencies
                                dir_logs : $dir_target/logs
                             dir_reports : $dir_target/reports
                  dir_source_main_python : src/main/python
                 dir_source_main_scripts : src/main/scripts
              dir_source_unittest_python : src/unittest/python
                              dir_target : target
                   distutils_classifiers : ['Development Status :: 3 - Alpha', 'Programming Language :: Python']
                      distutils_commands : ['sdist', 'bdist_dumb']
  distutils_issue8876_workaround_enabled : False
                distutils_use_setuptools : True
                      flake8_break_build : False
                 flake8_exclude_patterns : None
                  flake8_include_scripts : False
             flake8_include_test_sources : False
                  flake8_max_line_length : 120
    install_dependencies_extra_index_url : None
          install_dependencies_index_url : None
      install_dependencies_local_mapping : {}
            install_dependencies_upgrade : False
                    unittest_file_suffix : None
                    unittest_module_glob : *_tests
                         unittest_runner : (<function <lambda> at 0x7f9bb7378050>, u'_make_result')
             unittest_test_method_prefix : None
                                 verbose : True
[DEBUG] Execution plan is prepare, compile_sources, run_unit_tests, package, run_integration_tests, verify, publish
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /tmp/test
[INFO]  Going to execute task publish
[DEBUG] Executing task 'prepare'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing action 'check_pip_available' from 'pybuilder.plugins.python.install_dependencies_plugin' before task
[DEBUG] Checking if pip is available
[DEBUG] Executing action 'assert_flake8_is_executable' from 'pybuilder.plugins.python.flake8_plugin' before task
[DEBUG] Checking if flake8 is executable.
[DEBUG] Executing task 'compile_sources'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing task 'run_unit_tests'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing subtask from pybuilder.plugins.python.unittest_plugin
[INFO]  Running unit tests
[DEBUG] Forking process to run unit tests
[INFO]  Executing unit tests from Python modules in /tmp/test/src/unittest/python
[DEBUG] Including files matching '*_tests'
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[DEBUG] Executing task 'package'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing subtask from pybuilder.plugins.python.core_plugin
[DEBUG] Removing preexisting distribution /tmp/test/target/dist/test-1.0.dev0
[DEBUG] Creating directory /tmp/test/target/dist/test-1.0.dev0
[INFO]  Building distribution in /tmp/test/target/dist/test-1.0.dev0
[INFO]  Copying scripts to /tmp/test/target/dist/test-1.0.dev0/scripts
[DEBUG] Executing action 'write_manifest_file' from 'pybuilder.plugins.python.distutils_plugin' before task
[DEBUG] No data to write into MANIFEST.in
[DEBUG] Executing action 'write_setup_script' from 'pybuilder.plugins.python.distutils_plugin' before task
[INFO]  Writing setup.py as /tmp/test/target/dist/test-1.0.dev0/setup.py
[DEBUG] Executing task 'run_integration_tests'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing task 'verify'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing action 'verify_coverage' from 'pybuilder.plugins.python.coverage_plugin' before task
[INFO]  Collecting coverage information
[DEBUG] Forking process to do coverage analysis
[DEBUG] Executing task 'prepare'
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 293, in fork_process
    p.start()
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/forking.py", line 127, in __init__
    sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'stdout'
Traceback (most recent call last):
  File "/tmp/test/env-infra/bin/pyb", line 25, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 366, in main
    end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 298, in fork_process
    raise_exception(result[1], result[2])
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 280, in instrumented_target
    send_value = (target(*args, **kwargs), None, None)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 102, in do_coverage
    reactor.execute_task(target_task)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 173, in execute_task
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 228, in execute_task
    timer = Timer.start()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 204, in start
    return Timer()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 207, in __init__
    self.start_time = time.time()
AttributeError: 'module' object has no attribute 'time'
------------------------------------------------------------
BUILD FAILED - 'module' object has no attribute 'time'
------------------------------------------------------------
Build finished at 2015-09-23 13:16:05
Build took 0 seconds (811 ms)
Contributor

MirkoRossini commented Sep 23, 2015

Thanks for the quick reply. Here is the log.
I still think there should be a solution that doesn't require to mess with sys. I did see in the code the part where you walk in the sys lib directory trying to find the "essential" modules, but it still seems like a dangerous approach to me.

PyBuilder version 0.11.1
Build started at 2015-09-23 13:16:04
------------------------------------------------------------
[DEBUG] Loading project module from /tmp/test/build.py
[DEBUG] Loading plugin 'python.core'
[DEBUG] Trying to load builtin plugin 'python.core'
[DEBUG] Loading plugin 'core'
[DEBUG] Trying to load builtin plugin 'core'
[DEBUG] Found builtin plugin 'pybuilder.plugins.core_plugin'
[DEBUG] Found task 'clean' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'clean'
[DEBUG] Found task 'compile_sources' with required dependencies [<function prepare at 0x7f9bb7613c80>] and optional dependencies []
[DEBUG] Registering task 'compile_sources'
[DEBUG] Registering initializer 'init'
[DEBUG] Found task 'install' with required dependencies [<function publish at 0x7f9bb7613f50>] and optional dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found task 'package' with required dependencies [<function compile_sources at 0x7f9bb7613cf8>] and optional dependencies [<function run_unit_tests at 0x7f9bb7613d70>]
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Found task 'publish' with required dependencies [<function package at 0x7f9bb7613de8>] and optional dependencies [<function verify at 0x7f9bb7613ed8>]
[DEBUG] Registering task 'publish'
[DEBUG] Found task 'run_integration_tests' with required dependencies [<function package at 0x7f9bb7613de8>] and optional dependencies []
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'run_unit_tests' with required dependencies [<function compile_sources at 0x7f9bb7613cf8>] and optional dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found task 'verify' with required dependencies [<function run_integration_tests at 0x7f9bb7613e60>] and optional dependencies []
[DEBUG] Registering task 'verify'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.core_plugin'
[DEBUG] Registering initializer 'init_python_directories'
[DEBUG] Found task 'package' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Loading plugin 'python.unittest'
[DEBUG] Trying to load builtin plugin 'python.unittest'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Registering initializer 'init_test_source_directory'
[DEBUG] Found task 'run_unit_tests' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Loading plugin 'python.install_dependencies'
[DEBUG] Trying to load builtin plugin 'python.install_dependencies'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Found action check_pip_available
[DEBUG] Registering action 'check_pip_available'
[DEBUG] Found action create_install_log_directory
[DEBUG] Registering action 'create_install_log_directory'
[DEBUG] Registering initializer 'initialize_install_dependencies_plugin'
[DEBUG] Found task 'install_build_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install_build_dependencies'
[DEBUG] Found task 'install_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install_dependencies'
[DEBUG] Found task 'install_runtime_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install_runtime_dependencies'
[DEBUG] Found task 'list_dependencies' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'list_dependencies'
[DEBUG] Loading plugin 'python.flake8'
[DEBUG] Trying to load builtin plugin 'python.flake8'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Found task 'analyze' with required dependencies ['prepare'] and optional dependencies []
[DEBUG] Registering task 'analyze'
[DEBUG] Found action assert_flake8_is_executable
[DEBUG] Registering action 'assert_flake8_is_executable'
[DEBUG] Registering initializer 'initialize_flake8_plugin'
[DEBUG] Loading plugin 'python.coverage'
[DEBUG] Trying to load builtin plugin 'python.coverage'
[DEBUG] Loading plugin 'analysis'
[DEBUG] Trying to load builtin plugin 'analysis'
[DEBUG] Found builtin plugin 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found task 'analyze' with required dependencies ['run_unit_tests'] and optional dependencies []
[DEBUG] Registering task 'analyze'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Registering initializer 'init_coverage_properties'
[DEBUG] Found action verify_coverage
[DEBUG] Registering action 'verify_coverage'
[DEBUG] Loading plugin 'python.distutils'
[DEBUG] Trying to load builtin plugin 'python.distutils'
[DEBUG] Found builtin plugin 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Found action build_binary_distribution
[DEBUG] Registering action 'build_binary_distribution'
[DEBUG] Registering initializer 'initialize_distutils_plugin'
[DEBUG] Found task 'install' with required dependencies [] and optional dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found action write_manifest_file
[DEBUG] Registering action 'write_manifest_file'
[DEBUG] Found action write_setup_script
[DEBUG] Registering action 'write_setup_script'
[DEBUG] Have loaded plugins python.core, core, python.unittest, python.install_dependencies, python.flake8, python.coverage, analysis, python.distutils
[DEBUG] Registering initializer 'set_properties'
[DEBUG] Adding 'prepare' as a required dependency of task 'compile_sources'
[DEBUG] Adding 'publish' as a required dependency of task 'install'
[DEBUG] Adding 'compile_sources' as a required dependency of task 'package'
[DEBUG] Adding 'run_unit_tests' as an optional dependency of task 'package'
[DEBUG] Adding 'package' as a required dependency of task 'publish'
[DEBUG] Adding 'verify' as an optional dependency of task 'publish'
[DEBUG] Adding 'package' as a required dependency of task 'run_integration_tests'
[DEBUG] Adding 'compile_sources' as a required dependency of task 'run_unit_tests'
[DEBUG] Adding 'run_integration_tests' as a required dependency of task 'verify'
[DEBUG] Adding 'prepare' as a required dependency of task 'analyze'
[DEBUG] Adding 'run_unit_tests' as a required dependency of task 'analyze'
[DEBUG] Adding after action 'check_pip_available' for task 'prepare'
[DEBUG] Adding before action 'create_install_log_directory' for task 'install_build_dependencies'
[DEBUG] Adding before action 'create_install_log_directory' for task 'install_runtime_dependencies'
[DEBUG] Adding before action 'create_install_log_directory' for task 'install_dependencies'
[DEBUG] Adding after action 'assert_flake8_is_executable' for task 'prepare'
[DEBUG] Adding after action 'verify_coverage' for task 'analyze'
[DEBUG] Adding after action 'verify_coverage' for task 'verify'
[DEBUG] Adding before action 'build_binary_distribution' for task 'publish'
[DEBUG] Adding after action 'write_manifest_file' for task 'package'
[DEBUG] Adding after action 'write_setup_script' for task 'package'
[DEBUG] Verbose output enabled.

[DEBUG] Executing initializer 'init' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing initializer 'init_python_directories' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Executing initializer 'init_test_source_directory' from 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Executing initializer 'initialize_install_dependencies_plugin' from 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Executing initializer 'initialize_flake8_plugin' from 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Executing initializer 'init_coverage_properties' from 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Executing initializer 'initialize_distutils_plugin' from 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Executing initializer 'set_properties' from 'build'
[DEBUG] Project properties: 
                                 basedir : /tmp/test
                    coverage_break_build : True
                     coverage_exceptions : []
                           coverage_fork : None
                 coverage_reload_modules : None
                 coverage_threshold_warn : 70
                                dir_dist : $dir_target/dist/test-1.0.dev0
                        dir_dist_scripts : scripts
                        dir_install_logs : $dir_logs/install_dependencies
                                dir_logs : $dir_target/logs
                             dir_reports : $dir_target/reports
                  dir_source_main_python : src/main/python
                 dir_source_main_scripts : src/main/scripts
              dir_source_unittest_python : src/unittest/python
                              dir_target : target
                   distutils_classifiers : ['Development Status :: 3 - Alpha', 'Programming Language :: Python']
                      distutils_commands : ['sdist', 'bdist_dumb']
  distutils_issue8876_workaround_enabled : False
                distutils_use_setuptools : True
                      flake8_break_build : False
                 flake8_exclude_patterns : None
                  flake8_include_scripts : False
             flake8_include_test_sources : False
                  flake8_max_line_length : 120
    install_dependencies_extra_index_url : None
          install_dependencies_index_url : None
      install_dependencies_local_mapping : {}
            install_dependencies_upgrade : False
                    unittest_file_suffix : None
                    unittest_module_glob : *_tests
                         unittest_runner : (<function <lambda> at 0x7f9bb7378050>, u'_make_result')
             unittest_test_method_prefix : None
                                 verbose : True
[DEBUG] Execution plan is prepare, compile_sources, run_unit_tests, package, run_integration_tests, verify, publish
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /tmp/test
[INFO]  Going to execute task publish
[DEBUG] Executing task 'prepare'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing action 'check_pip_available' from 'pybuilder.plugins.python.install_dependencies_plugin' before task
[DEBUG] Checking if pip is available
[DEBUG] Executing action 'assert_flake8_is_executable' from 'pybuilder.plugins.python.flake8_plugin' before task
[DEBUG] Checking if flake8 is executable.
[DEBUG] Executing task 'compile_sources'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing task 'run_unit_tests'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing subtask from pybuilder.plugins.python.unittest_plugin
[INFO]  Running unit tests
[DEBUG] Forking process to run unit tests
[INFO]  Executing unit tests from Python modules in /tmp/test/src/unittest/python
[DEBUG] Including files matching '*_tests'
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[DEBUG] Executing task 'package'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing subtask from pybuilder.plugins.python.core_plugin
[DEBUG] Removing preexisting distribution /tmp/test/target/dist/test-1.0.dev0
[DEBUG] Creating directory /tmp/test/target/dist/test-1.0.dev0
[INFO]  Building distribution in /tmp/test/target/dist/test-1.0.dev0
[INFO]  Copying scripts to /tmp/test/target/dist/test-1.0.dev0/scripts
[DEBUG] Executing action 'write_manifest_file' from 'pybuilder.plugins.python.distutils_plugin' before task
[DEBUG] No data to write into MANIFEST.in
[DEBUG] Executing action 'write_setup_script' from 'pybuilder.plugins.python.distutils_plugin' before task
[INFO]  Writing setup.py as /tmp/test/target/dist/test-1.0.dev0/setup.py
[DEBUG] Executing task 'run_integration_tests'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing task 'verify'
[DEBUG] Executing subtask from pybuilder.plugins.core_plugin
[DEBUG] Executing action 'verify_coverage' from 'pybuilder.plugins.python.coverage_plugin' before task
[INFO]  Collecting coverage information
[DEBUG] Forking process to do coverage analysis
[DEBUG] Executing task 'prepare'
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 293, in fork_process
    p.start()
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/forking.py", line 127, in __init__
    sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'stdout'
Traceback (most recent call last):
  File "/tmp/test/env-infra/bin/pyb", line 25, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 366, in main
    end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 298, in fork_process
    raise_exception(result[1], result[2])
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 280, in instrumented_target
    send_value = (target(*args, **kwargs), None, None)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 102, in do_coverage
    reactor.execute_task(target_task)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 173, in execute_task
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 228, in execute_task
    timer = Timer.start()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 204, in start
    return Timer()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 207, in __init__
    self.start_time = time.time()
AttributeError: 'module' object has no attribute 'time'
------------------------------------------------------------
BUILD FAILED - 'module' object has no attribute 'time'
------------------------------------------------------------
Build finished at 2015-09-23 13:16:05
Build took 0 seconds (811 ms)
@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 23, 2015

Contributor

I added traceback and apparently sys is also considered not essential.
I changed the code like this:

finally:
    import traceback
    traceback.print_exc()
    end = datetime.datetime.now()

and I get

PyBuilder version 0.11.1
Build started at 2015-09-23 13:28:13
------------------------------------------------------------
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /tmp/test
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /tmp/test/src/unittest/python
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[INFO]  Building distribution in /tmp/test/target/dist/test-1.0.dev0
[INFO]  Copying scripts to /tmp/test/target/dist/test-1.0.dev0/scripts
[INFO]  Writing setup.py as /tmp/test/target/dist/test-1.0.dev0/setup.py
[INFO]  Collecting coverage information
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 293, in fork_process
    p.start()
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/forking.py", line 127, in __init__
    sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'stdout'
Traceback (most recent call last):
  File "/tmp/test/env-infra/bin/pyb", line 25, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 370, in main
    end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 298, in fork_process
    raise_exception(result[1], result[2])
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 280, in instrumented_target
    send_value = (target(*args, **kwargs), None, None)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 102, in do_coverage
    reactor.execute_task(target_task)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 173, in execute_task
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 228, in execute_task
    timer = Timer.start()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 204, in start
    return Timer()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 207, in __init__
    self.start_time = time.time()
AttributeError: 'module' object has no attribute 'time'
------------------------------------------------------------
BUILD FAILED - 'module' object has no attribute 'time'
------------------------------------------------------------
Build finished at 2015-09-23 13:28:13
Build took 0 seconds (338 ms)
Contributor

MirkoRossini commented Sep 23, 2015

I added traceback and apparently sys is also considered not essential.
I changed the code like this:

finally:
    import traceback
    traceback.print_exc()
    end = datetime.datetime.now()

and I get

PyBuilder version 0.11.1
Build started at 2015-09-23 13:28:13
------------------------------------------------------------
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /tmp/test
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /tmp/test/src/unittest/python
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[INFO]  Building distribution in /tmp/test/target/dist/test-1.0.dev0
[INFO]  Copying scripts to /tmp/test/target/dist/test-1.0.dev0/scripts
[INFO]  Writing setup.py as /tmp/test/target/dist/test-1.0.dev0/setup.py
[INFO]  Collecting coverage information
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 293, in fork_process
    p.start()
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/tmp/test/env-infra/lib/python2.7/multiprocessing/forking.py", line 127, in __init__
    sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'stdout'
Traceback (most recent call last):
  File "/tmp/test/env-infra/bin/pyb", line 25, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 370, in main
    end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'
Traceback (most recent call last):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 298, in fork_process
    raise_exception(result[1], result[2])
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 280, in instrumented_target
    send_value = (target(*args, **kwargs), None, None)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 102, in do_coverage
    reactor.execute_task(target_task)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/reactor.py", line 173, in execute_task
    reactor=self)
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/execution.py", line 228, in execute_task
    timer = Timer.start()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 204, in start
    return Timer()
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/utils.py", line 207, in __init__
    self.start_time = time.time()
AttributeError: 'module' object has no attribute 'time'
------------------------------------------------------------
BUILD FAILED - 'module' object has no attribute 'time'
------------------------------------------------------------
Build finished at 2015-09-23 13:28:13
Build took 0 seconds (338 ms)
@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 23, 2015

Contributor

Could you describe your custom installation? What are the customizations?

Contributor

arcivanov commented Sep 23, 2015

Could you describe your custom installation? What are the customizations?

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 23, 2015

Contributor

@mriehl given that we're forking, maybe try to do something like this:

  • We fork, except on Windows
  • We don't touch modules unless asked (return of the reload)
  • PyB build always reloads

If there are other plugins that load test code into the pyb process without forking, not reloading modules will cause coverage instrumentation to be unreliable, but unittest does fork by default so maybe it won't be that big of a deal?

Contributor

arcivanov commented Sep 23, 2015

@mriehl given that we're forking, maybe try to do something like this:

  • We fork, except on Windows
  • We don't touch modules unless asked (return of the reload)
  • PyB build always reloads

If there are other plugins that load test code into the pyb process without forking, not reloading modules will cause coverage instrumentation to be unreliable, but unittest does fork by default so maybe it won't be that big of a deal?

@arcivanov arcivanov added this to the v0.11.2 milestone Sep 23, 2015

@arcivanov arcivanov self-assigned this Sep 23, 2015

@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 23, 2015

Contributor

I think the main difference is that our environment doesn't use any of the shared libraries under /usr/lib64/python2.7
Instead, we place our standard libraries in env/lib/python2.7/ this is what you get as a result of

sysconfig.get_python_lib(standard_lib=True)

but our lib-dynload is not under the same directory, instead it's at

env/python/2.7.6/lib/python2.7/lib-dynload/

so every module in env/python/2.7.6/lib/python2.7/lib-dynload/ gets skipped as it's not located in env/lib/python2.7/ (the package_dir.startswith(std_lib) line)

This is what is causing the deletion of the useful modules.

Contributor

MirkoRossini commented Sep 23, 2015

I think the main difference is that our environment doesn't use any of the shared libraries under /usr/lib64/python2.7
Instead, we place our standard libraries in env/lib/python2.7/ this is what you get as a result of

sysconfig.get_python_lib(standard_lib=True)

but our lib-dynload is not under the same directory, instead it's at

env/python/2.7.6/lib/python2.7/lib-dynload/

so every module in env/python/2.7.6/lib/python2.7/lib-dynload/ gets skipped as it's not located in env/lib/python2.7/ (the package_dir.startswith(std_lib) line)

This is what is causing the deletion of the useful modules.

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Sep 23, 2015

Member

@arcivanov I'm okay with that. Forking by default makes it harder to produce wrong coverage numbers and the pybuilder metrics would still be accurate.

Member

mriehl commented Sep 23, 2015

@arcivanov I'm okay with that. Forking by default makes it harder to produce wrong coverage numbers and the pybuilder metrics would still be accurate.

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 23, 2015

Contributor

@MirkoRossini I wonder if I can get the information about your environment somehow. Python definitely knows to look for those directories so we should be able to look where Python does.

@mriehl even if I get more info from Python I think we might need to follow the method proposed above.

Contributor

arcivanov commented Sep 23, 2015

@MirkoRossini I wonder if I can get the information about your environment somehow. Python definitely knows to look for those directories so we should be able to look where Python does.

@mriehl even if I get more info from Python I think we might need to follow the method proposed above.

@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 23, 2015

Contributor

@arcivanov my previous comment contains the explanation on why the code doesn't consider the lib-dynload directory as being system. Do you need other details? Is there something I should attach?

Contributor

MirkoRossini commented Sep 23, 2015

@arcivanov my previous comment contains the explanation on why the code doesn't consider the lib-dynload directory as being system. Do you need other details? Is there something I should attach?

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 23, 2015

Contributor

@MirkoRossini I understood your explanation, I was simply thinking outloud on whether there was some way for me to query Python for those directories that you use. Thanks for your help, no more data necessary.

Contributor

arcivanov commented Sep 23, 2015

@MirkoRossini I understood your explanation, I was simply thinking outloud on whether there was some way for me to query Python for those directories that you use. Thanks for your help, no more data necessary.

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 24, 2015

Contributor

@MirkoRossini Actually I do have a favor to ask you. When I run the following in my environment here's what I get:

>>> from distutils import sysconfig
>>> sysconfig.get_python_lib(standard_lib=True)
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7'
>>> sysconfig.get_python_lib()
'/Users/arcivanov/pyhome/current/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True)
'/Users/arcivanov/pyhome/current/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True, standard_lib=True)
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7'

What do you get in yours running the same?

Contributor

arcivanov commented Sep 24, 2015

@MirkoRossini Actually I do have a favor to ask you. When I run the following in my environment here's what I get:

>>> from distutils import sysconfig
>>> sysconfig.get_python_lib(standard_lib=True)
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7'
>>> sysconfig.get_python_lib()
'/Users/arcivanov/pyhome/current/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True)
'/Users/arcivanov/pyhome/current/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True, standard_lib=True)
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7'

What do you get in yours running the same?

@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 24, 2015

Contributor

@arcivanov thanks for working on this:

>>> from distutils import sysconfig
>>> sysconfig.get_python_lib(standard_lib=True)
'/tmp/test/env-infra/lib/python2.7'
>>> sysconfig.get_python_lib()
'/tmp/test/env-infra/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True)
'/tmp/test/env-infra/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True, standard_lib=True)
'/tmp/test/env-infra/lib/python2.7'

I made progress on my side though. I checked and lib-dynload is actually symlinked in the correct location.
After some digging, I found that the problem is caused by this line:

canon_sys_path = [os.path.realpath(package_dir) for package_dir in sys.path]

realpath removes the symlink, so the libraries don't appear to be in the correct location. sys.path correctly shows

/tmp/test/env/lib/python2.7/lib-dynload

Is there a reason why we need to follow the symlinks and get the real path? Otherwise I can work on a pull request

Contributor

MirkoRossini commented Sep 24, 2015

@arcivanov thanks for working on this:

>>> from distutils import sysconfig
>>> sysconfig.get_python_lib(standard_lib=True)
'/tmp/test/env-infra/lib/python2.7'
>>> sysconfig.get_python_lib()
'/tmp/test/env-infra/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True)
'/tmp/test/env-infra/lib/python2.7/site-packages'
>>> sysconfig.get_python_lib(plat_specific=True, standard_lib=True)
'/tmp/test/env-infra/lib/python2.7'

I made progress on my side though. I checked and lib-dynload is actually symlinked in the correct location.
After some digging, I found that the problem is caused by this line:

canon_sys_path = [os.path.realpath(package_dir) for package_dir in sys.path]

realpath removes the symlink, so the libraries don't appear to be in the correct location. sys.path correctly shows

/tmp/test/env/lib/python2.7/lib-dynload

Is there a reason why we need to follow the symlinks and get the real path? Otherwise I can work on a pull request

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 24, 2015

Contributor

Ah, that's the thing, my /Users/arcivanov/pyhome/current is a symlink as well but I'm not using the plat_specific. I think I added the real path to compare apples to apples and I think the simple way to do this will be to canonize everything to make sure we always deal with real paths. I should have the patch tonight. Thanks for you help!

Contributor

arcivanov commented Sep 24, 2015

Ah, that's the thing, my /Users/arcivanov/pyhome/current is a symlink as well but I'm not using the plat_specific. I think I added the real path to compare apples to apples and I think the simple way to do this will be to canonize everything to make sure we always deal with real paths. I should have the patch tonight. Thanks for you help!

@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 24, 2015

Contributor

Thanks a lot :)

Contributor

MirkoRossini commented Sep 24, 2015

Thanks a lot :)

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 25, 2015

arcivanov added a commit that referenced this issue Sep 25, 2015

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 25, 2015

Contributor

@MirkoRossini could you test with the following?

pip install --upgrade https://github.com/pybuilder/pybuilder/archive/issue_211_test.tar.gz

  1. It'll produce a ton of printout info.
  2. It might resolve the issue already simply by doing the realpath for stdlib.
Contributor

arcivanov commented Sep 25, 2015

@MirkoRossini could you test with the following?

pip install --upgrade https://github.com/pybuilder/pybuilder/archive/issue_211_test.tar.gz

  1. It'll produce a ton of printout info.
  2. It might resolve the issue already simply by doing the realpath for stdlib.
@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 25, 2015

Contributor

Hi, I actually get this:

Collecting https://github.com/pybuilder/pybuilder/archive/issue_211_test.tar.gz
  Downloading https://github.com/pybuilder/pybuilder/archive/issue_211_test.tar.gz (121kB)
    100% |################################| 122kB 460kB/s 
    PyBuilder version ${version}
    Build started at 2015-09-25 10:52:20
    ------------------------------------------------------------
    ------------------------------------------------------------
    BUILD FAILED - Cannot determine git revision: git rev-list HEAD failed:
    fatal: Not a git repository (or any parent up to mount point /tmp)
    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
    ------------------------------------------------------------
    Build finished at 2015-09-25 10:52:20
    Build took 0 seconds (74 ms)
    Complete output from command python setup.py egg_info:
    PyBuilder version ${version}

    Build started at 2015-09-25 10:52:20

    ------------------------------------------------------------

    ------------------------------------------------------------

    BUILD FAILED - Cannot determine git revision: git rev-list HEAD failed:

    fatal: Not a git repository (or any parent up to mount point /tmp)

    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).



    ------------------------------------------------------------

    Build finished at 2015-09-25 10:52:20

    Build took 0 seconds (74 ms)

    ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-W2cega-build
Contributor

MirkoRossini commented Sep 25, 2015

Hi, I actually get this:

Collecting https://github.com/pybuilder/pybuilder/archive/issue_211_test.tar.gz
  Downloading https://github.com/pybuilder/pybuilder/archive/issue_211_test.tar.gz (121kB)
    100% |################################| 122kB 460kB/s 
    PyBuilder version ${version}
    Build started at 2015-09-25 10:52:20
    ------------------------------------------------------------
    ------------------------------------------------------------
    BUILD FAILED - Cannot determine git revision: git rev-list HEAD failed:
    fatal: Not a git repository (or any parent up to mount point /tmp)
    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
    ------------------------------------------------------------
    Build finished at 2015-09-25 10:52:20
    Build took 0 seconds (74 ms)
    Complete output from command python setup.py egg_info:
    PyBuilder version ${version}

    Build started at 2015-09-25 10:52:20

    ------------------------------------------------------------

    ------------------------------------------------------------

    BUILD FAILED - Cannot determine git revision: git rev-list HEAD failed:

    fatal: Not a git repository (or any parent up to mount point /tmp)

    Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).



    ------------------------------------------------------------

    Build finished at 2015-09-25 10:52:20

    Build took 0 seconds (74 ms)

    ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-W2cega-build
@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Sep 25, 2015

Member

@MirkoRossini sorry, that was me and my travis woes. Can you retry? I fixed the branch.

Member

mriehl commented Sep 25, 2015

@MirkoRossini sorry, that was me and my travis woes. Can you retry? I fixed the branch.

@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 25, 2015

Contributor

Thanks, I can install now! Here is what I get:

PyBuilder version 0.11.2.dev0
Build started at 2015-09-25 11:12:08
------------------------------------------------------------
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /tmp/test
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /tmp/test/src/unittest/python
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[INFO]  Building distribution in /tmp/test/target/dist/test-1.0.dev0
[INFO]  Copying scripts to /tmp/test/target/dist/test-1.0.dev0/scripts
[INFO]  Writing setup.py as /tmp/test/target/dist/test-1.0.dev0/setup.py
[INFO]  Collecting coverage information
[WARN]  coverage_branch_threshold_warn is 0 and branch coverage will not be checked
[WARN]  coverage_branch_partial_threshold_warn is 0 and partial branch coverage will not be checked
Original sys path: ['/tmp/test/env-infra/bin', '/apps/infrafs1/mrossini/.local/lib/python2.7/site-packages/rsa-3.1.2-py2.7.egg', '/apps/infrafs1/mrossini/.local/lib/python2.7/site-packages/pyasn1-0.1.7-py2.7.egg', '/tmp/test/env-infra/lib/python2.7/site-packages', '/tmp/test/env-infra/lib/python27.zip', '/tmp/test/env-infra/lib/python2.7', '/tmp/test/env-infra/python/2.7.6/lib/python2.7/plat-linux2', '/tmp/test/env-infra/python/2.7.6/lib/python2.7/lib-tk', '/tmp/test/env-infra/lib/python2.7/lib-old', '/tmp/test/env-infra/python/2.7.6/lib/python2.7/lib-dynload', '/apps/infrafs1/mrossini/.local/lib/python2.7/site-packages']
Stdlib path: /tmp/test/env-infra/lib/python2.7
Stdlib sys path: ['/tmp/test/env-infra/lib/python2.7/site-packages', '/tmp/test/env-infra/lib/python2.7', '/tmp/test/env-infra/lib/python2.7/lib-old']
Essential (sys module) module 'heapq'
Non-essential module 'distutils'
Essential (sys module) module 'functools'
Essential (sys module) module 'random'
Non-essential module 'datetime'
Essential (sys module) module 'sysconfig'
Essential (built-in) module 'gc'
Non-essential module 'distutils.sysconfig'
Non-essential module 'ctypes._endian'
Essential (sys module) module 'fnmatch'
Essential (sys module) module 'struct'
Essential (sys module) module 'tempfile'
Essential (sys module) module 'pprint'
Non-essential module '_json'
Essential (built-in) module 'imp'
Essential (in sys_packages) module 'pybuilder.plugins.python.setuptools_plugin_helper'
Non-essential module 'multiprocessing.forking'
Essential (in sys_packages) module 'pybuilder.pluginhelper'
Essential (built-in) module 'zipimport'
Essential (sys module) module 'string'
Non-essential module 'multiprocessing.synchronize'
Non-essential module 'encodings.utf_8'
Non-essential module 'unittest.util'
Essential (in sys_packages) module 'pybuilder.plugins.python.coverage_plugin'
Non-essential module 'json.encoder'
Essential (sys module) module 'subprocess'
Essential (built-in) module 'signal'
Essential (in sys_packages) module 'pybuilder'
Essential (sys module) module 'threading'
Essential (sys module) module 'token'
Essential (in sys_packages) module 'pybuilder.ci_server_interaction'
Non-essential module 'multiprocessing'
Essential (sys module) module 'dis'
Non-essential module 'cStringIO'
Essential (sys module) module 'locale'
Essential (in sys_packages) module 'pybuilder.execution'
Non-essential module 'multiprocessing.process'
Essential (sys module) module 'atexit'
Non-essential module 'encodings'
Essential (in sys_packages) module 'pybuilder.plugins.python.flake8_plugin'
Essential (sys module) module 'abc'
Essential (in sys_packages) module 'pybuilder.excp_util_2'
Essential (in sys_packages) module 'pybuilder.errors'
Non-essential module '_multiprocessing'
Essential (sys module) module 're'
Essential (in sys_packages) module 'pybuilder.terminal'
Non-essential module 'unittest.suite'
Non-essential module 'math'
Non-essential module 'fcntl'
Non-essential module 'unittest.case'
Essential (sys module) module 'optparse'
Essential (sys module) module 'UserDict'
Essential (in sys_packages) module 'pybuilder.plugins.core_plugin'
Essential (sys module) module 'inspect'
Essential (in sys_packages) module 'pybuilder.plugins.python.unittest_plugin'
Essential (sys module) module 'Queue'
Non-essential module 'json.scanner'
Non-essential module 'ctypes'
Non-essential module '_ctypes'
Essential (sys module) module 'codecs'
Non-essential module 'unittest.runner'
Non-essential module 'unittest.signals'
Non-essential module 'unittest.loader'
Non-essential module '_functools'
Non-essential module '_locale'
Essential (built-in) module 'thread'
Essential (sys module) module 'StringIO'
Essential (sys module) module 'traceback'
Essential (sys module) module 'weakref'
Non-essential module 'itertools'
Essential (sys module) module 'opcode'
Essential (in sys_packages) module 'pybuilder.core'
Non-essential module 'multiprocessing.util'
Essential (sys module) module 'os'
Essential (built-in) module 'marshal'
Essential (sys module) module '__future__'
Essential (in sys_packages) module 'pybuilder.plugins.python.distutils_plugin'
Non-essential module '_collections'
Essential (built-in) module '_sre'
Non-essential module 'unittest'
Essential (in sys_packages) module 'pybuilder.utils'
Essential (built-in) module '__builtin__'
Essential (in sys_packages) module 'pybuilder.pluginhelper.external_command'
Essential (in sys_packages) module 'pybuilder.plugins.python.core_plugin'
Essential (sys module) module 'platform'
Non-essential module 'operator'
Non-essential module 'select'
Non-essential module '_heapq'
Essential (sys module) module 'posixpath'
Essential (in sys_packages) module 'tblib.cpython'
Essential (built-in) module 'errno'
Non-essential module 'multiprocessing.connection'
Non-essential module '_socket'
Non-essential module 'binascii'
Essential (sys module) module 'sre_constants'
Non-essential module 'json'
Essential (sys module) module 'posixpath'
Essential (sys module) module 'tokenize'
Essential (sys module) module 'collections'
Essential (built-in) module '_warnings'
Non-essential module 'cPickle'
Essential (built-in) module '_codecs'
Essential (in sys_packages) module 'pybuilder.scaffolding'
Essential (in sys_packages) module 'tblib.pickling_support'
Essential (built-in) module 'pwd'
Essential (sys module) module 'copy'
Non-essential module 'unittest.main'
Essential (sys module) module '_sysconfigdata'
Non-essential module '_struct'
Essential (in sys_packages) module 'pybuilder.plugins.python'
Essential (sys module) module 'hashlib'
Essential (in sys_packages) module 'tblib'
Essential (sys module) module 'keyword'
Essential (in sys_packages) module 'pybuilder.cli'
Essential (built-in) module 'posix'
Non-essential module 'encodings.aliases'
Essential (in sys_packages) module 'pybuilder.reactor'
Essential (built-in) module 'exceptions'
Essential (sys module) module 'sre_parse'
Essential (sys module) module 'pickle'
Non-essential module 'json.decoder'
Essential (sys module) module 'copy_reg'
Essential (sys module) module 'sre_compile'
Non-essential module '_hashlib'
Non-essential module '_random'
Essential (sys module) module 'site'
Essential (sys module) module 'io'
Essential (built-in) module '__main__'
Essential (sys module) module 'shutil'
Essential (in sys_packages) module 'pybuilder.plugins.python.install_dependencies_plugin'
Essential (in sys_packages) module 'pybuilder.plugins.analysis_plugin'
Non-essential module 'unittest.result'
Non-essential module 'strop'
Non-essential module 'grp'
Essential (sys module) module 'linecache'
Essential (sys module) module 'socket'
Essential (sys module) module 'gettext'
Non-essential module 'encodings.hex_codec'
Non-essential module 'multiprocessing.queues'
Essential (sys module) module '_abcoll'
Non-essential module 'build'
Essential (sys module) module 'genericpath'
Essential (sys module) module 'stat'
Non-essential module '_ssl'
Essential (sys module) module 'warnings'
Essential (in sys_packages) module 'pybuilder.plugins.python.python_plugin_helper'
Essential (sys module) module 'textwrap'
Essential (built-in) module 'sys'
Essential (in sys_packages) module 'pybuilder.pluginloader'
Essential (sys module) module 'types'
Essential (in sys_packages) module 'pybuilder.graph_utils'
Essential (in sys_packages) module 'pybuilder.plugins'
Essential (built-in) module '_weakref'
Essential (sys module) module 'difflib'
Non-essential module 'distutils.errors'
Non-essential module '_io'
Essential (sys module) module '_weakrefset'
Non-essential module 'time'
Traceback (most recent call last):
  File "/tmp/test/env-infra/bin/pyb", line 25, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 366, in main
    end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'
------------------------------------------------------------
BUILD FAILED - 'module' object has no attribute 'time'
------------------------------------------------------------
Build finished at 2015-09-25 11:12:09
Build took 0 seconds (569 ms)
Contributor

MirkoRossini commented Sep 25, 2015

Thanks, I can install now! Here is what I get:

PyBuilder version 0.11.2.dev0
Build started at 2015-09-25 11:12:08
------------------------------------------------------------
[INFO]  Building test version 1.0.dev0
[INFO]  Executing build in /tmp/test
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /tmp/test/src/unittest/python
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[INFO]  Building distribution in /tmp/test/target/dist/test-1.0.dev0
[INFO]  Copying scripts to /tmp/test/target/dist/test-1.0.dev0/scripts
[INFO]  Writing setup.py as /tmp/test/target/dist/test-1.0.dev0/setup.py
[INFO]  Collecting coverage information
[WARN]  coverage_branch_threshold_warn is 0 and branch coverage will not be checked
[WARN]  coverage_branch_partial_threshold_warn is 0 and partial branch coverage will not be checked
Original sys path: ['/tmp/test/env-infra/bin', '/apps/infrafs1/mrossini/.local/lib/python2.7/site-packages/rsa-3.1.2-py2.7.egg', '/apps/infrafs1/mrossini/.local/lib/python2.7/site-packages/pyasn1-0.1.7-py2.7.egg', '/tmp/test/env-infra/lib/python2.7/site-packages', '/tmp/test/env-infra/lib/python27.zip', '/tmp/test/env-infra/lib/python2.7', '/tmp/test/env-infra/python/2.7.6/lib/python2.7/plat-linux2', '/tmp/test/env-infra/python/2.7.6/lib/python2.7/lib-tk', '/tmp/test/env-infra/lib/python2.7/lib-old', '/tmp/test/env-infra/python/2.7.6/lib/python2.7/lib-dynload', '/apps/infrafs1/mrossini/.local/lib/python2.7/site-packages']
Stdlib path: /tmp/test/env-infra/lib/python2.7
Stdlib sys path: ['/tmp/test/env-infra/lib/python2.7/site-packages', '/tmp/test/env-infra/lib/python2.7', '/tmp/test/env-infra/lib/python2.7/lib-old']
Essential (sys module) module 'heapq'
Non-essential module 'distutils'
Essential (sys module) module 'functools'
Essential (sys module) module 'random'
Non-essential module 'datetime'
Essential (sys module) module 'sysconfig'
Essential (built-in) module 'gc'
Non-essential module 'distutils.sysconfig'
Non-essential module 'ctypes._endian'
Essential (sys module) module 'fnmatch'
Essential (sys module) module 'struct'
Essential (sys module) module 'tempfile'
Essential (sys module) module 'pprint'
Non-essential module '_json'
Essential (built-in) module 'imp'
Essential (in sys_packages) module 'pybuilder.plugins.python.setuptools_plugin_helper'
Non-essential module 'multiprocessing.forking'
Essential (in sys_packages) module 'pybuilder.pluginhelper'
Essential (built-in) module 'zipimport'
Essential (sys module) module 'string'
Non-essential module 'multiprocessing.synchronize'
Non-essential module 'encodings.utf_8'
Non-essential module 'unittest.util'
Essential (in sys_packages) module 'pybuilder.plugins.python.coverage_plugin'
Non-essential module 'json.encoder'
Essential (sys module) module 'subprocess'
Essential (built-in) module 'signal'
Essential (in sys_packages) module 'pybuilder'
Essential (sys module) module 'threading'
Essential (sys module) module 'token'
Essential (in sys_packages) module 'pybuilder.ci_server_interaction'
Non-essential module 'multiprocessing'
Essential (sys module) module 'dis'
Non-essential module 'cStringIO'
Essential (sys module) module 'locale'
Essential (in sys_packages) module 'pybuilder.execution'
Non-essential module 'multiprocessing.process'
Essential (sys module) module 'atexit'
Non-essential module 'encodings'
Essential (in sys_packages) module 'pybuilder.plugins.python.flake8_plugin'
Essential (sys module) module 'abc'
Essential (in sys_packages) module 'pybuilder.excp_util_2'
Essential (in sys_packages) module 'pybuilder.errors'
Non-essential module '_multiprocessing'
Essential (sys module) module 're'
Essential (in sys_packages) module 'pybuilder.terminal'
Non-essential module 'unittest.suite'
Non-essential module 'math'
Non-essential module 'fcntl'
Non-essential module 'unittest.case'
Essential (sys module) module 'optparse'
Essential (sys module) module 'UserDict'
Essential (in sys_packages) module 'pybuilder.plugins.core_plugin'
Essential (sys module) module 'inspect'
Essential (in sys_packages) module 'pybuilder.plugins.python.unittest_plugin'
Essential (sys module) module 'Queue'
Non-essential module 'json.scanner'
Non-essential module 'ctypes'
Non-essential module '_ctypes'
Essential (sys module) module 'codecs'
Non-essential module 'unittest.runner'
Non-essential module 'unittest.signals'
Non-essential module 'unittest.loader'
Non-essential module '_functools'
Non-essential module '_locale'
Essential (built-in) module 'thread'
Essential (sys module) module 'StringIO'
Essential (sys module) module 'traceback'
Essential (sys module) module 'weakref'
Non-essential module 'itertools'
Essential (sys module) module 'opcode'
Essential (in sys_packages) module 'pybuilder.core'
Non-essential module 'multiprocessing.util'
Essential (sys module) module 'os'
Essential (built-in) module 'marshal'
Essential (sys module) module '__future__'
Essential (in sys_packages) module 'pybuilder.plugins.python.distutils_plugin'
Non-essential module '_collections'
Essential (built-in) module '_sre'
Non-essential module 'unittest'
Essential (in sys_packages) module 'pybuilder.utils'
Essential (built-in) module '__builtin__'
Essential (in sys_packages) module 'pybuilder.pluginhelper.external_command'
Essential (in sys_packages) module 'pybuilder.plugins.python.core_plugin'
Essential (sys module) module 'platform'
Non-essential module 'operator'
Non-essential module 'select'
Non-essential module '_heapq'
Essential (sys module) module 'posixpath'
Essential (in sys_packages) module 'tblib.cpython'
Essential (built-in) module 'errno'
Non-essential module 'multiprocessing.connection'
Non-essential module '_socket'
Non-essential module 'binascii'
Essential (sys module) module 'sre_constants'
Non-essential module 'json'
Essential (sys module) module 'posixpath'
Essential (sys module) module 'tokenize'
Essential (sys module) module 'collections'
Essential (built-in) module '_warnings'
Non-essential module 'cPickle'
Essential (built-in) module '_codecs'
Essential (in sys_packages) module 'pybuilder.scaffolding'
Essential (in sys_packages) module 'tblib.pickling_support'
Essential (built-in) module 'pwd'
Essential (sys module) module 'copy'
Non-essential module 'unittest.main'
Essential (sys module) module '_sysconfigdata'
Non-essential module '_struct'
Essential (in sys_packages) module 'pybuilder.plugins.python'
Essential (sys module) module 'hashlib'
Essential (in sys_packages) module 'tblib'
Essential (sys module) module 'keyword'
Essential (in sys_packages) module 'pybuilder.cli'
Essential (built-in) module 'posix'
Non-essential module 'encodings.aliases'
Essential (in sys_packages) module 'pybuilder.reactor'
Essential (built-in) module 'exceptions'
Essential (sys module) module 'sre_parse'
Essential (sys module) module 'pickle'
Non-essential module 'json.decoder'
Essential (sys module) module 'copy_reg'
Essential (sys module) module 'sre_compile'
Non-essential module '_hashlib'
Non-essential module '_random'
Essential (sys module) module 'site'
Essential (sys module) module 'io'
Essential (built-in) module '__main__'
Essential (sys module) module 'shutil'
Essential (in sys_packages) module 'pybuilder.plugins.python.install_dependencies_plugin'
Essential (in sys_packages) module 'pybuilder.plugins.analysis_plugin'
Non-essential module 'unittest.result'
Non-essential module 'strop'
Non-essential module 'grp'
Essential (sys module) module 'linecache'
Essential (sys module) module 'socket'
Essential (sys module) module 'gettext'
Non-essential module 'encodings.hex_codec'
Non-essential module 'multiprocessing.queues'
Essential (sys module) module '_abcoll'
Non-essential module 'build'
Essential (sys module) module 'genericpath'
Essential (sys module) module 'stat'
Non-essential module '_ssl'
Essential (sys module) module 'warnings'
Essential (in sys_packages) module 'pybuilder.plugins.python.python_plugin_helper'
Essential (sys module) module 'textwrap'
Essential (built-in) module 'sys'
Essential (in sys_packages) module 'pybuilder.pluginloader'
Essential (sys module) module 'types'
Essential (in sys_packages) module 'pybuilder.graph_utils'
Essential (in sys_packages) module 'pybuilder.plugins'
Essential (built-in) module '_weakref'
Essential (sys module) module 'difflib'
Non-essential module 'distutils.errors'
Non-essential module '_io'
Essential (sys module) module '_weakrefset'
Non-essential module 'time'
Traceback (most recent call last):
  File "/tmp/test/env-infra/bin/pyb", line 25, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/tmp/test/env-infra/lib/python2.7/site-packages/pybuilder/cli.py", line 366, in main
    end = datetime.datetime.now()
AttributeError: 'module' object has no attribute 'datetime'
------------------------------------------------------------
BUILD FAILED - 'module' object has no attribute 'time'
------------------------------------------------------------
Build finished at 2015-09-25 11:12:09
Build took 0 seconds (569 ms)
@MirkoRossini

This comment has been minimized.

Show comment
Hide comment
@MirkoRossini

MirkoRossini Sep 25, 2015

Contributor

maybe an approach could be to create a set of symlinks and realpaths and consider all of them as valid paths? It would definitely fix my problem (sys path is an actual directory, sys-dynload is a symlink), I don't think it would affect other people.

Contributor

MirkoRossini commented Sep 25, 2015

maybe an approach could be to create a set of symlinks and realpaths and consider all of them as valid paths? It would definitely fix my problem (sys path is an actual directory, sys-dynload is a symlink), I don't think it would affect other people.

@alex-dow

This comment has been minimized.

Show comment
Hide comment
@alex-dow

alex-dow Sep 25, 2015

I just wanted to say, I have a similar problem, but it is isn't consistent.

On both machines, I do the following:

virtualenv env
source env/bin/activate
pyb clean install_dependencies analyze publish

Machine 1 is using python 2.7.9 on Ubuntu 15.04 with pyb 0.11.1, and works fine
Machine 2 is using python 2.7.6 on Ubuntu 14.04 with pyb 0.11.1, and does not work. I've run it in console, and from Jenkins and have the same exception in both

[INFO]  Collecting coverage information
[WARN]  coverage_fork is deprecated, coverage always runs in its own fork
[DEBUG] Forking process to do coverage analysis
[DEBUG] Module 'pybuilder_prettySetup' coverage to be verified
Traceback (most recent call last):
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/utils.py", line 298, in fork_process
    raise_exception(result[1], result[2])
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/utils.py", line 280, in instrumented_target
    send_value = (target(*args, **kwargs), None, None)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 86, in do_coverage
    _delete_non_essential_modules()
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 233, in _delete_non_essential_modules
    if not _is_module_essential(module.__name__, sys_packages, sys_modules):
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pip/_vendor/six.py", line 115, in __getattr__
    raise AttributeError
AttributeError

not sure if this helps. If you need any other details from either machine let me know

alex-dow commented Sep 25, 2015

I just wanted to say, I have a similar problem, but it is isn't consistent.

On both machines, I do the following:

virtualenv env
source env/bin/activate
pyb clean install_dependencies analyze publish

Machine 1 is using python 2.7.9 on Ubuntu 15.04 with pyb 0.11.1, and works fine
Machine 2 is using python 2.7.6 on Ubuntu 14.04 with pyb 0.11.1, and does not work. I've run it in console, and from Jenkins and have the same exception in both

[INFO]  Collecting coverage information
[WARN]  coverage_fork is deprecated, coverage always runs in its own fork
[DEBUG] Forking process to do coverage analysis
[DEBUG] Module 'pybuilder_prettySetup' coverage to be verified
Traceback (most recent call last):
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/cli.py", line 354, in main
    environments=options.environments, tasks=arguments)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/reactor.py", line 163, in build
    reactor=self)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 300, in execute_execution_plan
    summaries.append(self.execute_task(task, **keyword_arguments))
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 258, in execute_task
    if self.execute_action(action, keyword_arguments):
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 290, in execute_action
    action.execute(arguments)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/execution.py", line 91, in execute
    self.callable(*arguments)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 50, in verify_coverage
    run_coverage(project, logger, reactor, "coverage", "coverage", "run_unit_tests")
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 68, in run_coverage
    target_task, shortest_plan))
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/utils.py", line 298, in fork_process
    raise_exception(result[1], result[2])
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/utils.py", line 280, in instrumented_target
    send_value = (target(*args, **kwargs), None, None)
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 86, in do_coverage
    _delete_non_essential_modules()
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 233, in _delete_non_essential_modules
    if not _is_module_essential(module.__name__, sys_packages, sys_modules):
  File "/psikon/.jenkins/jobs/pybuilder_prettySetup/workspace/env/local/lib/python2.7/site-packages/pip/_vendor/six.py", line 115, in __getattr__
    raise AttributeError
AttributeError

not sure if this helps. If you need any other details from either machine let me know

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 25, 2015

Contributor

Ok, I'm sticking to the original plan - we can't handle custom environments 😭

Contributor

arcivanov commented Sep 25, 2015

Ok, I'm sticking to the original plan - we can't handle custom environments 😭

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 25, 2015

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 25, 2015

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Sep 25, 2015

@arcivanov arcivanov closed this in #215 Sep 25, 2015

arcivanov added a commit that referenced this issue Sep 25, 2015

Merge pull request #215 from arcivanov/issue_211
coverage plugin causes issues during build deleting sys modules #211

@arcivanov arcivanov removed the in progress label Sep 25, 2015

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Sep 25, 2015

Contributor

No more resetting modules by default and never on Windows.
If you need module reset functionality project.set_property("coverage_reset_modules", True), which is the case for PyB itself.

Contributor

arcivanov commented Sep 25, 2015

No more resetting modules by default and never on Windows.
If you need module reset functionality project.set_property("coverage_reset_modules", True), which is the case for PyB itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment