KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292)

[samsonych]

I faced the following error after upgrading the pybuilder from 0.11 to 0.12 (environment 'system': CPython version 3.6.8.final.0 on win32).

PyBuilder version 0.12.6
Build started at 2020-07-20 22:30:50
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:my_pybuilder_plugin, module name 'my_pybuilder_plugin'"
[INFO]  Processing plugin packages 'my_pybuilder_plugin' to be installed with {}
------------------------------------------------------------
BUILD FAILED - KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292)
------------------------------------------------------------
Build finished at 2020-07-20 22:31:09
Build took 19 seconds (19333 ms)

I'm not sure that I fully understand the method pybuilder.execution.ExecutionManager.register_late_task_dependencies but it seems to me there is a error parameter in the debug message. Please look at the line 292) or check it in this pull request #738 .

I suppose it is necessary to use self._dependencies_pending_tasks instead of dependencies in this case. At least my project was built successfully after this fix.

[arcivanov]

Thanks for filing this. Could you please share the relevant source for your plugin?
People including myself are using PyB with external plugins and I'm not sure the cause/fix is actually the true cause.

[arcivanov]

It would also be super helpful if you could run the build with -v -X and post the logs here.

[samsonych]

I'm sorry, Arcadiy, but I have no possibility to share the code of this plugin.

The build log is here:

(venv_win_3.6) D:\src\EAS\aws\eas-redirector-lambda>pyb -v -X
[DEBUG] Registered Python environment 'system': CPython version 3.6.8.final.0 on win32 in ['d:\\src\\eas\\aws\\eas-redirector-lambda\\venv_win_3.6\\scripts\\python.exe']
PyBuilder version 0.12.6
Build started at 2020-07-21 10:25:50
------------------------------------------------------------
[DEBUG] Loading project module from .\build.py
[DEBUG] Setting up plugins VEnv at 'd:\src\eas\aws\eas-redirector-lambda\.pybuilder\plugins\cpython-3.6.8.final.0'
[DEBUG] Registered Python environment 'pybuilder': CPython version 3.6.8.final.0 on win32 in ['d:\\src\\eas\\aws\\eas-redirector-lambda\\.pybuilder\\plugins\\cpython-3.6.8.final.0\\script
s\\python.exe']
[DEBUG] Patched __import__ system to defer plugin loading
[INFO]  Installing or updating plugin "pypi:pybuilder_defaults_plugin, module name 'pybuilder_defaults_plugin'"
[DEBUG] Inspecting package pybuilder_defaults_plugin
[DEBUG] Package pybuilder_defaults_plugin is not installed and will be installed
[INFO]  Processing plugin packages 'pybuilder_defaults_plugin' to be installed with {}
[DEBUG] Executing command: 'd:\\src\\eas\\aws\\eas-redirector-lambda\\.pybuilder\\plugins\\cpython-3.6.8.final.0\\scripts\\python.exe' '-m' 'pip.__main__' 'install' 'pybuilder_def
aults_plugin'
[DEBUG] Loading plugin 'pypi:pybuilder_defaults_plugin'
[DEBUG] Trying to load third party plugin 'pypi:pybuilder_defaults_plugin', module 'pybuilder_defaults_plugin'
[DEBUG] Found third party plugin 'pypi:pybuilder_defaults_plugin', module 'pybuilder_defaults_plugin'
[DEBUG] Loading plugin 'filter_resources'
[DEBUG] Trying to load direct plugin 'filter_resources', module 'pybuilder.plugins.filter_resources_plugin'
[DEBUG] Found direct plugin 'filter_resources', module 'pybuilder.plugins.filter_resources_plugin'
[DEBUG] Loading plugin 'copy_resources'
[DEBUG] Trying to load direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Found direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Loading plugin 'python.core'
[DEBUG] Trying to load direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Found direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Loading plugin 'python.install_dependencies'
[DEBUG] Trying to load direct plugin 'python.install_dependencies', module 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Found direct plugin 'python.install_dependencies', module 'pybuilder.plugins.python.install_dependencies_plugin'
[DEBUG] Loading plugin 'python.distutils'
[DEBUG] Trying to load direct plugin 'python.distutils', module 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Found direct plugin 'python.distutils', module 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Loading plugin 'python.unittest'
[DEBUG] Trying to load direct plugin 'python.unittest', module 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Found direct plugin 'python.unittest', module 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Loading plugin 'python.integrationtest'
[DEBUG] Trying to load direct plugin 'python.integrationtest', module 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Found direct plugin 'python.integrationtest', module 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Loading plugin 'python.coverage'
[DEBUG] Trying to load direct plugin 'python.coverage', module 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Found direct plugin 'python.coverage', module 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Loading plugin 'python.flake8'
[DEBUG] Trying to load direct plugin 'python.flake8', module 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Found direct plugin 'python.flake8', module 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Loading plugin 'core'
[DEBUG] Trying to load direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Found direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Loading plugin 'analysis'
[DEBUG] Trying to load direct plugin 'analysis', module 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found direct plugin 'analysis', module 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering initializer 'init_python_directories'
[DEBUG] Found task 'package' with dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering initializer 'initialize_install_dependencies_plugin'
[DEBUG] Found task 'install_build_dependencies' with dependencies [prepare]
[DEBUG] Registering task 'install_build_dependencies'
[DEBUG] Found task 'install_dependencies' with dependencies [prepare]
[DEBUG] Registering task 'install_dependencies'
[DEBUG] Found task 'install_runtime_dependencies' with dependencies [prepare]
[DEBUG] Registering task 'install_runtime_dependencies'
[DEBUG] Found task 'list_dependencies' with dependencies []
[DEBUG] Registering task 'list_dependencies'
[DEBUG] Found action build_binary_distribution
[DEBUG] Registering action 'build_binary_distribution'
[DEBUG] Registering initializer 'initialize_distutils_plugin'
[DEBUG] Found task 'install' with dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found action set_description
[DEBUG] Registering action 'set_description'
[DEBUG] Found task 'upload' with dependencies []
[DEBUG] Registering task 'upload'
[DEBUG] Found action write_manifest_file
[DEBUG] Registering action 'write_manifest_file'
[DEBUG] Found action write_setup_script
[DEBUG] Registering action 'write_setup_script'
[DEBUG] Found action coverage_init
[DEBUG] Registering action 'coverage_init'
[DEBUG] Registering initializer 'init_test_source_directory'
[DEBUG] Found task 'run_unit_tests' with dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found action coverage_init
[DEBUG] Registering action 'coverage_init'
[DEBUG] Registering initializer 'initialize_integrationtest_plugin'
[DEBUG] Found task 'run_integration_tests' with dependencies []
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'analyze' with dependencies [run_unit_tests]
[DEBUG] Registering task 'analyze'
[DEBUG] Found task 'coverage' with dependencies [verify]
[DEBUG] Registering task 'coverage'
[DEBUG] Registering initializer 'init_coverage_properties'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering late dependency of task 'publish' on [coverage(optional)]
[DEBUG] Found task 'analyze' with dependencies [prepare]
[DEBUG] Registering task 'analyze'
[DEBUG] Found action assert_flake8_is_executable
[DEBUG] Registering action 'assert_flake8_is_executable'
[DEBUG] Registering initializer 'initialize_flake8_plugin'
[DEBUG] Found task 'analyze' with dependencies [prepare]
[DEBUG] Registering task 'analyze'
[DEBUG] Found task 'build_lambda_package' with dependencies [clean, install_dependencies, verify, publish]
[DEBUG] Registering task 'build_lambda_package'
[DEBUG] Found action configure_pylint
[DEBUG] Registering action 'configure_pylint'
[DEBUG] Found action git_release
[DEBUG] Registering action 'git_release'
[DEBUG] Registering initializer 'initialize_default_configuration'
[DEBUG] Found task 'prepare_release' with dependencies []
[DEBUG] Registering task 'prepare_release'
[DEBUG] Found task 'release' with dependencies [prepare_release, clean, install_build_dependencies, install_runtime_dependencies, verify, publish]
[DEBUG] Registering task 'release'
[DEBUG] Found task 'release_lambda' with dependencies [prepare_release, clean, install_build_dependencies, install_runtime_dependencies, verify, publish, build_lambda_package]
[DEBUG] Registering task 'release_lambda'
[DEBUG] Found task 'verify' with dependencies [run_unit_tests, run_integration_tests, analyze]
[DEBUG] Registering task 'verify'
[DEBUG] Found task 'clean' with dependencies []
[DEBUG] Registering task 'clean'
[DEBUG] Found task 'compile_sources' with dependencies [prepare]
[DEBUG] Registering task 'compile_sources'
[DEBUG] Registering initializer 'init'
[DEBUG] Found task 'install' with dependencies [package, publish(optional)]
[DEBUG] Registering task 'install'
[DEBUG] Found task 'package' with dependencies [compile_sources, run_unit_tests(optional)]
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Found task 'print_module_path' with dependencies []
[DEBUG] Registering task 'print_module_path'
[DEBUG] Found task 'print_scripts_path' with dependencies []
[DEBUG] Registering task 'print_scripts_path'
[DEBUG] Found task 'publish' with dependencies [package, verify(optional)]
[DEBUG] Registering task 'publish'
[DEBUG] Found task 'run_integration_tests' with dependencies [package]
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'run_unit_tests' with dependencies [compile_sources]
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found task 'verify' with dependencies [run_integration_tests(optional)]
[DEBUG] Registering task 'verify'
Traceback (most recent call last):
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\cli.py", line 460, in main
    offline=options.offline
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\reactor.py", line 210, in prepare_build
    self._collect_project_annotations()
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\reactor.py", line 560, in _collect_project_annotations
    self.collect_project_annotations(deferred_plugin[1])
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\reactor.py", line 394, in collect_project_annotations
    self.execution_manager.register_late_task_dependencies(injected_task_dependencies)
  File "d:\src\eas\aws\eas-redirector-lambda\venv_win_3.6\lib\site-packages\pybuilder\execution.py", line 292, in register_late_task_dependencies
    self.logger.debug("Resolved late dependency of task '%s' on %s", name, dependencies[name])
KeyError: 'publish'
------------------------------------------------------------
BUILD FAILED - KeyError: 'publish' (lib\site-packages\pybuilder\execution.py:292)
------------------------------------------------------------
Build finished at 2020-07-21 10:26:09
Build took 19 seconds (19470 ms)

[arcivanov]

I understand. Can you share plugin source without function/method bodies? Just function/method declarations?

[samsonych]

@arcivanov Ok. Hope this helps.

build.py

from pybuilder.core import init, use_plugin

use_plugin("python.core")
use_plugin("python.install_dependencies")
use_plugin("python.unittest")
use_plugin("python.integrationtest")
use_plugin("python.distutils")

name = "pybuilder-defaults-plugin"
version = "1.1.1"
summary = "PyBuilder plugin to provide common configuration for python projects"
default_task = ["clean", "install_dependencies", "install"]

@init
def initialize(project):
    project.set_property("dir_dist", "target/dist")
    project.depends_on("twine", "==1.15.0")
    project.depends_on("pylint", "~=2.4.0")
    project.depends_on("importlib_resources", "==1.0.2")
    project.include_file("pybuilder_defaults_plugin.configs", 'default_pylintrc')

__init__.py

from pybuilder.errors import BuildFailedException
from pybuilder.core import init, use_plugin, task, depends, description, after, before

use_plugin("python.core")
use_plugin("python.install_dependencies")
use_plugin("python.distutils")
use_plugin("python.unittest")
use_plugin("python.integrationtest")
use_plugin("python.coverage")
use_plugin('python.flake8')

@init
def initialize_default_configuration(project):
	pass

@before("analyze", only_once=True)
def configure_pylint(project, logger):
    pass

@task
@depends("prepare")
def analyze(project, logger):
    pass

@task
@depends("run_unit_tests", "run_integration_tests", "analyze")
def verify():
    pass

@task
@depends("clean", "install_dependencies", "verify", "publish")
@description("Builds AWS lambda package. Zip package contains project sources along with required dependencies")
def build_lambda_package(project, logger):
    pass

@task
@depends(*RELEASE_TASKS)
@description("Uploads a project to Edmunds PyPi index and pushes release tag to git")
def release(project, logger):
    pass

@task
@depends(*RELEASE_TASKS, build_lambda_package)
@description("Builds lambda package and pushes release tag to git")
def release_lambda():
    pass

@after([release, release_lambda], only_once=True)
def git_release(project, logger):
	pass

@task
@description("Prepares release by removing '.dev' suffix from project version")
def prepare_release(project, logger):
	pass

[arcivanov]

Thanks! What are RELEASE_TASKS?

[samsonych]

What are RELEASE_TASKS?

Sorry.

RELEASE_TASKS = ["prepare_release", "clean", "install_build_dependencies", "install_runtime_dependencies", "verify",
 "publish"]

[samsonych]

Hi @arcivanov
Are there some updates? Does the provided code help to reproduce or define this issue?

[arcivanov]

@samsonych Yes, thanks, working on it!

[arcivanov]

@samsonych As suspected, the problem is somewhat different from the naive fix.