Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292) #737

Closed
samsonych opened this issue Jul 20, 2020 · 13 comments · Fixed by #739
Closed

KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292) #737

samsonych opened this issue Jul 20, 2020 · 13 comments · Fixed by #739

Comments

@samsonych
Copy link

samsonych commented Jul 20, 2020

I faced the following error after upgrading the pybuilder from 0.11 to 0.12 (environment 'system': CPython version 3.6.8.final.0 on win32).

PyBuilder version 0.12.6
Build started at 2020-07-20 22:30:50
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:my_pybuilder_plugin, module name 'my_pybuilder_plugin'"
[INFO]  Processing plugin packages 'my_pybuilder_plugin' to be installed with {}
------------------------------------------------------------
BUILD FAILED - KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292)
------------------------------------------------------------
Build finished at 2020-07-20 22:31:09
Build took 19 seconds (19333 ms)

I'm not sure that I fully understand the method pybuilder.execution.ExecutionManager.register_late_task_dependencies but it seems to me there is a error parameter in the debug message. Please look at the line 292) or check it in this pull request #738 .

I suppose it is necessary to use self._dependencies_pending_tasks instead of dependencies in this case. At least my project was built successfully after this fix.

@arcivanov
Copy link
Member

Thanks for filing this. Could you please share the relevant source for your plugin?
People including myself are using PyB with external plugins and I'm not sure the cause/fix is actually the true cause.

@arcivanov
Copy link
Member

arcivanov commented Jul 20, 2020

It would also be super helpful if you could run the build with -v -X and post the logs here.

@samsonych
Copy link
Author

I'm sorry, Arcadiy, but I have no possibility to share the code of this plugin.

The build log is here:

(venv_win_3.6) D:\src\EAS\aws\eas-redirector-lambda>pyb -v -X
[DEBUG] Registered Python environment 'system': CPython version 3.6.8.final.0 on win32 in ['d:\\src\\eas\\aws\\eas-redirector-lambda\\venv_win_3.6\\scripts\\python.exe']
PyBuilder version 0.12.6
Build started at 2020-07-21 10:25:50
------------------------------------------------------------
[DEBUG] Loading project module from .\build.py
[DEBUG] Setting up plugins VEnv at 'd:\src\eas\aws\eas-redirector-lambda\.pybuilder\plugins\cpython-3.6.8.final.0'
[DEBUG] Registered Python environment 'pybuilder': CPython version 3.6.8.final.0 on win32 in ['d:\\src\\eas\\aws\\eas-redirector-lambda\\.pybuilder\\plugins\\cpython-3.6.8.final.0\\script
s\\python.exe']
[DEBUG] Patched __import__ system to defer plugin loading
[INFO]  Installing or updating plugin "pypi:pybuilder_defaults_plugin, module name 'pybuilder_defaults_plugin'"
[DEBUG] Inspecting package pybuilder_defaults_plugin
[DEBUG] Package pybuilder_defaults_plugin is not installed and will be installed
[INFO]  Processing plugin packages 'pybuilder_defaults_plugin' to be installed with {}
[DEBUG] Executing command: 'd:\\src\\eas\\aws\\eas-redirector-lambda\\.pybuilder\\plugins\\cpython-3.6.8.final.0\\scripts\\python.exe' '-m' 'pip.__main__' 'install' 'pybuilder_def
aults_plugin'
[DEBUG] Loading plugin 'pypi:pybuilder_defaults_plugin'
[DEBUG] Trying to load third party plugin 'pypi:pybuilder_defaults_plugin', module 'pybuilder_defaults_plugin'
[DEBUG] Found third party plugin 'pypi:pybuilder_defaults_plugin', module 'pybuilder_defaults_plugin'
[DEBUG] Loading plugin 'filter_resources'
[DEBUG] Trying to load direct plugin 'filter_resources', module 'pybuilder.plugins.filter_resources_plugin'
[DEBUG] Found direct plugin 'filter_resources', module 'pybuilder.plugins.filter_resources_plugin'
[DEBUG] Loading plugin 'copy_resources'
[DEBUG] Trying to load direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Found direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Loading plugin 'python.core'
[DEBUG] Trying to load direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Found direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Loading plugin 'python.install_dependencies'
[DEBUG] Trying to load direct plugin 'python.install_dependencies', module 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Found direct plugin 'python.install_dependencies', module 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Loading plugin 'python.distutils'
[DEBUG] Trying to load direct plugin 'python.distutils', module 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Found direct plugin 'python.distutils', module 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Loading plugin 'python.unittest'
[DEBUG] Trying to load direct plugin 'python.unittest', module 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Found direct plugin 'python.unittest', module 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Loading plugin 'python.integrationtest'
[DEBUG] Trying to load direct plugin 'python.integrationtest', module 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Found direct plugin 'python.integrationtest', module 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Loading plugin 'python.coverage'
[DEBUG] Trying to load direct plugin 'python.coverage', module 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Found direct plugin 'python.coverage', module 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Loading plugin 'python.flake8'
[DEBUG] Trying to load direct plugin 'python.flake8', module 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Found direct plugin 'python.flake8', module 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Loading plugin 'core'
[DEBUG] Trying to load direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Found direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Loading plugin 'analysis'
[DEBUG] Trying to load direct plugin 'analysis', module 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found direct plugin 'analysis', module 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering initializer 'init_python_directories'
[DEBUG] Found task 'package' with dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering initializer 'initialize_install_dependencies_plugin'
[DEBUG] Found task 'install_build_dependencies' with dependencies [prepare]
[DEBUG] Registering task 'install_build_dependencies'
[DEBUG] Found task 'install_dependencies' with dependencies [prepare]
[DEBUG] Registering task 'install_dependencies'
[DEBUG] Found task 'install_runtime_dependencies' with dependencies [prepare]
[DEBUG] Registering task 'install_runtime_dependencies'
[DEBUG] Found task 'list_dependencies' with dependencies []
[DEBUG] Registering task 'list_dependencies'
[DEBUG] Found action build_binary_distribution
[DEBUG] Registering action 'build_binary_distribution'
[DEBUG] Registering initializer 'initialize_distutils_plugin'
[DEBUG] Found task 'install' with dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found action set_description
[DEBUG] Registering action 'set_description'
[DEBUG] Found task 'upload' with dependencies []
[DEBUG] Registering task 'upload'
[DEBUG] Found action write_manifest_file
[DEBUG] Registering action 'write_manifest_file'
[DEBUG] Found action write_setup_script
[DEBUG] Registering action 'write_setup_script'
[DEBUG] Found action coverage_init
[DEBUG] Registering action 'coverage_init'
[DEBUG] Registering initializer 'init_test_source_directory'
[DEBUG] Found task 'run_unit_tests' with dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found action coverage_init
[DEBUG] Registering action 'coverage_init'
[DEBUG] Registering initializer 'initialize_integrationtest_plugin'
[DEBUG] Found task 'run_integration_tests' with dependencies []
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'analyze' with dependencies [run_unit_tests]
[DEBUG] Registering task 'analyze'
[DEBUG] Found task 'coverage' with dependencies [verify]
[DEBUG] Registering task 'coverage'
[DEBUG] Registering initializer 'init_coverage_properties'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering late dependency of task 'publish' on [coverage(optional)]
[DEBUG] Found task 'analyze' with dependencies [prepare]
[DEBUG] Registering task 'analyze'
[DEBUG] Found action assert_flake8_is_executable
[DEBUG] Registering action 'assert_flake8_is_executable'
[DEBUG] Registering initializer 'initialize_flake8_plugin'
[DEBUG] Found task 'analyze' with dependencies [prepare]
[DEBUG] Registering task 'analyze'
[DEBUG] Found task 'build_lambda_package' with dependencies [clean, install_dependencies, verify, publish]
[DEBUG] Registering task 'build_lambda_package'
[DEBUG] Found action configure_pylint
[DEBUG] Registering action 'configure_pylint'
[DEBUG] Found action git_release
[DEBUG] Registering action 'git_release'
[DEBUG] Registering initializer 'initialize_default_configuration'
[DEBUG] Found task 'prepare_release' with dependencies []
[DEBUG] Registering task 'prepare_release'
[DEBUG] Found task 'release' with dependencies [prepare_release, clean, install_build_dependencies, install_runtime_dependencies, verify, publish]
[DEBUG] Registering task 'release'
[DEBUG] Found task 'release_lambda' with dependencies [prepare_release, clean, install_build_dependencies, install_runtime_dependencies, verify, publish, build_lambda_package]
[DEBUG] Registering task 'release_lambda'
[DEBUG] Found task 'verify' with dependencies [run_unit_tests, run_integration_tests, analyze]
[DEBUG] Registering task 'verify'
[DEBUG] Found task 'clean' with dependencies []
[DEBUG] Registering task 'clean'
[DEBUG] Found task 'compile_sources' with dependencies [prepare]
[DEBUG] Registering task 'compile_sources'
[DEBUG] Registering initializer 'init'
[DEBUG] Found task 'install' with dependencies [package, publish(optional)]
[DEBUG] Registering task 'install'
[DEBUG] Found task 'package' with dependencies [compile_sources, run_unit_tests(optional)]
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Found task 'print_module_path' with dependencies []
[DEBUG] Registering task 'print_module_path'
[DEBUG] Found task 'print_scripts_path' with dependencies []
[DEBUG] Registering task 'print_scripts_path'
[DEBUG] Found task 'publish' with dependencies [package, verify(optional)]
[DEBUG] Registering task 'publish'
[DEBUG] Found task 'run_integration_tests' with dependencies [package]
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'run_unit_tests' with dependencies [compile_sources]
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found task 'verify' with dependencies [run_integration_tests(optional)]
[DEBUG] Registering task 'verify'
Traceback (most recent call last):
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\cli.py", line 460, in main
    offline=options.offline
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\reactor.py", line 210, in prepare_build
    self._collect_project_annotations()
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\reactor.py", line 560, in _collect_project_annotations
    self.collect_project_annotations(deferred_plugin[1])
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\reactor.py", line 394, in collect_project_annotations
    self.execution_manager.register_late_task_dependencies(injected_task_dependencies)
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\execution.py", line 292, in register_late_task_dependencies
    self.logger.debug("Resolved late dependency of task '%s' on %s", name, dependencies[name])
KeyError: 'publish'
------------------------------------------------------------
BUILD FAILED - KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292)
------------------------------------------------------------
Build finished at 2020-07-21 10:26:09
Build took 19 seconds (19470 ms)

@arcivanov
Copy link
Member

I understand. Can you share plugin source without function/method bodies? Just function/method declarations?

@samsonych
Copy link
Author

@arcivanov Ok. Hope this helps.

build.py

from pybuilder.core import init, use_plugin

use_plugin("python.core")
use_plugin("python.install_dependencies")
use_plugin("python.unittest")
use_plugin("python.integrationtest")
use_plugin("python.distutils")

name = "pybuilder-defaults-plugin"
version = "1.1.1"
summary = "PyBuilder plugin to provide common configuration for python projects"
default_task = ["clean", "install_dependencies", "install"]

@init
def initialize(project):
    project.set_property("dir_dist", "target/dist")
    project.depends_on("twine", "==1.15.0")
    project.depends_on("pylint", "~=2.4.0")
    project.depends_on("importlib_resources", "==1.0.2")
    project.include_file("pybuilder_defaults_plugin.configs", 'default_pylintrc')

__init__.py

from pybuilder.errors import BuildFailedException
from pybuilder.core import init, use_plugin, task, depends, description, after, before

use_plugin("python.core")
use_plugin("python.install_dependencies")
use_plugin("python.distutils")
use_plugin("python.unittest")
use_plugin("python.integrationtest")
use_plugin("python.coverage")
use_plugin('python.flake8')

@init
def initialize_default_configuration(project):
	pass

@before("analyze", only_once=True)
def configure_pylint(project, logger):
    pass

@task
@depends("prepare")
def analyze(project, logger):
    pass

@task
@depends("run_unit_tests", "run_integration_tests", "analyze")
def verify():
    pass

@task
@depends("clean", "install_dependencies", "verify", "publish")
@description("Builds AWS lambda package. Zip package contains project sources along with required dependencies")
def build_lambda_package(project, logger):
    pass

@task
@depends(*RELEASE_TASKS)
@description("Uploads a project to Edmunds PyPi index and pushes release tag to git")
def release(project, logger):
    pass

@task
@depends(*RELEASE_TASKS, build_lambda_package)
@description("Builds lambda package and pushes release tag to git")
def release_lambda():
    pass

@after([release, release_lambda], only_once=True)
def git_release(project, logger):
	pass

@task
@description("Prepares release by removing '.dev' suffix from project version")
def prepare_release(project, logger):
	pass

@arcivanov
Copy link
Member

Thanks! What are RELEASE_TASKS?

@samsonych
Copy link
Author

What are RELEASE_TASKS?

Sorry.

RELEASE_TASKS = ["prepare_release", "clean", "install_build_dependencies", "install_runtime_dependencies", "verify",
 "publish"]

@samsonych
Copy link
Author

Hi @arcivanov
Are there some updates? Does the provided code help to reproduce or define this issue?

@arcivanov
Copy link
Member

@samsonych Yes, thanks, working on it!

@arcivanov
Copy link
Member

@samsonych As suspected, the problem is somewhat different from the naive fix.

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Jul 25, 2020
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Jul 25, 2020
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Jul 25, 2020
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Jul 26, 2020
arcivanov added a commit to arcivanov/pybuilder that referenced this issue Jul 26, 2020
@arcivanov
Copy link
Member

@samsonych so, upon further inspection I have stumbled upon a different bug, and your fix for your problem may be right on point. I'll verify/confirm your fix and will merge it if I have no other issues.

@arcivanov
Copy link
Member

@samsonych I managed to reproduce it independently, however the above example you provided does not reproduce the issue.

@samsonych
Copy link
Author

... and your fix for your problem may be right on point

Glad to see that )

Arcadiy, thanks for your reply and assistance.

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Jul 26, 2020
TaskDependency.task may not be set during late dependency injection
Late dependency injection on a not-yet-existing task would fail to
log properly

Added various test to cover

fixes pybuilder#737
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants