Handle plugin registration failure ContextualVersionConflict with log instead of raising error #1542

noklam · 2022-05-16T15:42:50Z

Signed-off-by: noklam nok.lam.chan@quantumblack.com

Description

Currently, when using kedro with plugins, the conflict version between libraries will stop kedro completely to load. The more ideal way is to ignore the plugins and throw warnings.

These errors can at least happen in 3 different ways.

Conflict library version specified in kedro itself. This could happen if you install kedro, then downgrade pip. In theory, if you use Python API it will run successfully, but anything involving the entry points will fail. The main reason is that pkg_resource always validates the dependencies, and there seems to be no easy way to fix it.
Conflict library version coming from kedro-xxxxx plugins. This is the direct cause of the creation of this PR, the less intrusive way to deal with it is just to log the error instead of terminating the entire program. I choose to use a higher level of LOG level as for certain plugins, it could mean the run is actually no longer valid. (i.e. kedro-mlflow or any plugin that actually has additional behavior for a kedro run.
Load entry points when we want to call the __main__ of a kedro package programmatically.

Development notes

This PR replaces the usage of pkg_resource with importlib-xxx for a few reasons.
1.1 The pkg_resource is an old library and maybe deprecated soon, even the official website of pkg_resource suggest using importlib-xxx alternatives whenever possible. pluggy and pytest replaced pkg_resource with importlib too.
1.2. Improve import speed - the pkg_resource import lots of thing at the top-level, which slow down the CLI. #1476
1.3. Reduce the chance that the ContextualVersionConflict types of error. This is not solved unfortunately, it seems that this validation still happens if one of kedro's dependencies uses pkg_resource. dynaconf is one of them, and importing dynaconf will trigger the same error.
When the plugin registry fails, kedro will log them instead of terminate the program.
Move setuptools to pyproject.toml (PEP-518, not really related to the PR but found out this when studying how the python build system works.)
❌ micropkg is still using pkg_resource for requirement parsing, it is not handled now since it is out of scope.

Compatibility issue:

https://bugs.python.org/issue44246
Unfortunately the entry_points() API is not consistent between Python version, importlib.metadata is only available with python>=3.8. I spend quite a while to figure this out and make all test passed. For now, the backward compatible importlib_metadata seems to be a better option, but it is not with the standard lib importlib.

This is how it looks if the plugin's registration fails. For example I explicitly downgrade plotly so kedro-viz is not registered successfully.

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam · 2022-05-19T12:58:57Z

kedro/framework/cli/utils.py

-        except Exception as exc:
-            raise KedroCliError(f"Loading {name} commands from {entry_point}") from exc
+        except Exception as exc:  # pylint: disable=broad-except
+            logger.warning(
+                KedroCliError(f"Fail to load {name} commands from {entry_point}")
+            )
+            logger.warning(exc)


This is the most important part. It's hard to determine from the framework side whether a kedro's run is still valid if plugins fails. kedro-viz is defintely ok to just ignore it, but for kedro-mlflow type of plugin that actually modify the run behavior, it may be problematic so I am using warning level.

Similar suggestion here to the other occurrence in cli.py.

noklam · 2022-05-19T12:59:46Z

pyproject.toml

+# PEP-518 https://peps.python.org/pep-0518/
+[build-system]
+# Minimum requirements for the build system to execute.
+requires = ["setuptools>=38.0", "wheel"]  # PEP 508 specifications.
+


This is something I just come across when I work on this PR, no related to the PR particularly.

We've got "setuptools>=38.0" and wheel requirements in various other places in the repo (just do a Ctrl+F to see them). Does this mean we can remove them from there?

If yes then that would be nice but a bit of a bigger change, so probably worth doing in a separate PR.

good point, created a new PR and will revert this soon.

noklam · 2022-05-19T13:01:35Z

tests/framework/cli/test_cli.py

@@ -299,37 +299,44 @@ def test_project_groups(self, entry_points, entry_point):
        entry_point.load.return_value = "groups"
        groups = load_entry_points("project")
        assert groups == ["groups"]
-        entry_points.assert_called_once_with(group="kedro.project_commands")
+        entry_points.return_value.select.assert_called_once_with(


For the test, it's mainly just catching the log instead of catching the error. Due to the entry_points API, I have to mock 2 layers which makes the assertion a bit longer.

noklam · 2022-05-19T13:38:06Z

I still have to make it work for py37... but feel free to drop comments.

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

…p-kedro-running-when-dependency-clashes

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

…ct-error-stop-kedro-running-when-dependency-clashes' into fix/1487-contextualversionconflict-error-stop-kedro-running-when-dependency-clashes Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

antonymilne

Looks good! Another ticket which turned out to be much harder than expected, great job tackling it ⭐

I think a more conventional way to deal with a backported package would be

try to import importlib.metadata and only if that fails import importlib_metadata
add python_version < '3.8' to importlib_metadata in requirements so that it's only installed if necessary

See, for example, https://github.com/pallets/click/pull/1890/files.

But I don't know if this is possible here because of the API change you mention 😬

antonymilne · 2022-05-19T21:18:27Z

kedro/framework/cli/cli.py

@@ -65,10 +68,9 @@ def info():
    plugin_versions = {}
    plugin_entry_points = defaultdict(set)
    for plugin_entry_point, group in ENTRY_POINT_GROUPS.items():
-        for entry_point in pkg_resources.iter_entry_points(group=group):
+        for entry_point in importlib_metadata.entry_points().select(group=group):


Could this also raise some kind of exception? Or is it just the entry_point.load that we should wrap in the try?

Though I guess this is just kedro info rather than being triggered on every single kedro command like the other instances so it doesn't matter if there's uncaught exceptions anyway.

This is indeed the solution I go with initially, but later I find out the API was not consistent. So technically only Python 3.10 can use the stdlib.

Originally I have this conditional import block in kedro.utils, otherwise I have to do this conditionally everywhere. Do you think it is the right place to do so? 3c12bcc

It could fail, but like you said it should be just kedro info, unlike other plugin where the program still run without loading it. I think it is ok to leave it.

Urghh, what a pain. I think what you did in 3c12bcc was ok, although probably it's more usual to just repeat the conditional import in multiple files.

BUT if the standard library API is only right in Python 3.10 then let's just forget about it and go for importlib_metadata all the way like you're doing now 👍

I have considered that, but it also means we have to copy the same block in multiple files, and also anywhere that use mocker.patch, which is quite hard to read.

For the context, I started this PR with importlib_metadata, so I wasn't aware of the inconsistent API between the Python version. In theory, if I avoid using the select API, it could be compatible with python3.8-3.10. But for both case, we still need to have the conditional import, so I would rather just stick with importlib_metadata, unless this extra dependencies is causing trouble.

Yeah, that totally makes sense. I'm definitely happy to go with importlib_metadata throughout. It's not worth the extra complexity doing it with standard library only for Python 3.10.

antonymilne · 2022-05-19T21:23:30Z

kedro/framework/cli/cli.py

+            logger.warning(KedroCliError(f"Fail to initialize {entry_point}"))
+            logger.warning(exc)


Suggested change

logger.warning(KedroCliError(f"Fail to initialize {entry_point}"))

logger.warning(exc)

logger.warning(f"Failed to initialise %s. Full exception: %s", entry_point, exc)

Unless there's as good reason to use the KedroCliError still?

antonymilne · 2022-05-19T21:24:53Z

kedro/framework/cli/utils.py

-        except Exception as exc:
-            raise KedroCliError(f"Loading {name} commands from {entry_point}") from exc
+        except Exception as exc:  # pylint: disable=broad-except
+            logger.warning(
+                KedroCliError(f"Fail to load {name} commands from {entry_point}")
+            )
+            logger.warning(exc)


Similar suggestion here to the other occurrence in cli.py.

antonymilne · 2022-05-19T21:25:36Z

kedro/framework/cli/cli.py

@@ -98,12 +100,13 @@ def docs():

 def _init_plugins():
    group = ENTRY_POINT_GROUPS["init"]
-    for entry_point in pkg_resources.iter_entry_points(group=group):
+    for entry_point in importlib_metadata.entry_points().select(group=group):


Would it make sense to refactor this function a bit so it uses load_entry_points in utils.py? At a glance they look very similar.

The original test for _init_plugins is actually incorrectly testing the load exception, it should be testing the error during hook initialisation instead.

init_hook = entry_point.load() # previously the the error was triggered here init_hook() # this is what we want to test instead

antonymilne · 2022-05-19T21:27:30Z

tests/framework/cli/test_cli.py

        entry_point.module_name = "bob.fred"

        result = CliRunner().invoke(cli, ["info"])
+        print(result.output)


Suggested change

print(result.output)

antonymilne · 2022-05-19T21:31:56Z

pyproject.toml

+# PEP-518 https://peps.python.org/pep-0518/
+[build-system]
+# Minimum requirements for the build system to execute.
+requires = ["setuptools>=38.0", "wheel"]  # PEP 508 specifications.
+


We've got "setuptools>=38.0" and wheel requirements in various other places in the repo (just do a Ctrl+F to see them). Does this mean we can remove them from there?

If yes then that would be nice but a bit of a bigger change, so probably worth doing in a separate PR.

noklam · 2022-05-19T22:21:19Z

Couples notes for the reviews:

refactor the entry_points loop for load_entry_points() and _init_plugins()
Revert PEP-518 changes and do it in a separate PR instead.
Consider just log the error instead of using KedroCLIError

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

…p-kedro-running-when-dependency-clashes

merelcht

Great work! ⭐ Thanks for the thorough description on the PR, that made it very easy to understand the changes made.
I'm happy to see the move to importlib_metadata now pkg_resources doesn't seem to be the proper standard anymore.

antonymilne

Really great work! Just one small comment. And also don't forget to add to RELEASE.md.

antonymilne · 2022-05-20T15:56:22Z

kedro/framework/cli/cli.py

            init_hook()
-        except Exception as exc:
-            raise KedroCliError(f"Initializing {entry_point}") from exc
+        except Exception as exc:  # pylint: disable=broad-except


Now you've made it clearer exactly what raises the exception here, I actually think we should remove the try/except altogether and just do a plain

for init_hook in init_hooks: init_hooks()

Reasoning:

exceptions raised by entry_point.load() are now handled by load_entry_points

exceptions raised by init_hook() should still be raised, not hidden and logged, and there's not much point wrapping them in KedroCliError like we used to. This is analogous to what happens e.g. if a plugin with a hook raises an exception - it doesn't get caught and logged or converted into a kedro error message

All this is kind of minor because realistically no one uses init_hooks and it's probably redundant now we have before_command_run anyway... But if it's not too annoying to chance I think this would be a small improvement.

Good catch, I think it is actually wrong to catch the error, error should be raised!

requirements.txt

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

…ct-error-stop-kedro-running-when-dependency-clashes' into fix/1487-contextualversionconflict-error-stop-kedro-running-when-dependency-clashes Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam added 5 commits May 16, 2022 16:32

ignore loading error

72b84ca

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Move setuptools to pyproject.toml to align with PEP-518

176f89f

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

revert changes

a19ee63

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

[WIP] - replace pkg_resource with importlib_metadata

9d77ddd

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

[WIP] - replace pkg_resource

96500ad

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam changed the title ~~ContextualVersionConflict Error stop kedro to load~~ Replacing pkg_resources with importlib_metadata for entrypoints discovery May 18, 2022

noklam added 3 commits May 18, 2022 12:12

Replace entrypoint discovery with importlib

cd45c3d

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Tidy up logging message, linting

9ce96e9

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

small changes to tidy up the variables

a0765e6

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam changed the title ~~Replacing pkg_resources with importlib_metadata for entrypoints discovery~~ Handle plugin registration failure with gentle logging instead of raising error May 18, 2022

noklam added 3 commits May 18, 2022 17:32

Add conditional import to fix importlib not available in python 3.7

3c12bcc

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Fix test cases

139a075

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Update error message

c4d655c

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam changed the title ~~Handle plugin registration failure with gentle logging instead of raising error~~ Handle plugin registration failure ContextualVersionConflict with log instead of raising error May 19, 2022

noklam added 2 commits May 19, 2022 13:11

Switch back to importlib_metadata and fix tests

0ce181a

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

add importlib as requirements

3abfdc9

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam marked this pull request as ready for review May 19, 2022 12:56

noklam requested a review from idanov as a code owner May 19, 2022 12:56

noklam requested review from merelcht and antonymilne May 19, 2022 12:56

noklam commented May 19, 2022

View reviewed changes

noklam and others added 4 commits May 19, 2022 14:57

Fix test with py37

200405b

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Merge branch 'main' into fix/1487-contextualversionconflict-error-sto…

cd7ba3a

…p-kedro-running-when-dependency-clashes

Fix linting

3d4aa55

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Merge remote-tracking branch 'origin/fix/1487-contextualversionconfli…

4de3c11

…ct-error-stop-kedro-running-when-dependency-clashes' into fix/1487-contextualversionconflict-error-stop-kedro-running-when-dependency-clashes Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam requested a review from SajidAlamQB May 19, 2022 15:05

antonymilne reviewed May 19, 2022

View reviewed changes

noklam mentioned this pull request May 19, 2022

Clean up setuptools and wheel requirements to align with PEP-518 #1559

Closed

noklam added 5 commits May 20, 2022 11:10

Apply changes from review

91fb855

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Refactor and fix wrong test

3209dbe

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Fix some tests... should work now

e20183c

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

I forgot to put the setuptools back...

d804e77

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

I have gone too far and removed a necessary dependency

fc28f4a

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam requested a review from antonymilne May 20, 2022 11:31

Merge branch 'main' into fix/1487-contextualversionconflict-error-sto…

b3dbbab

…p-kedro-running-when-dependency-clashes

noklam removed the request for review from idanov May 20, 2022 12:11

merelcht approved these changes May 20, 2022

View reviewed changes

antonymilne approved these changes May 20, 2022

View reviewed changes

noklam added 3 commits May 20, 2022 17:11

apply comments after reviews

c61fd82

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

Merge remote-tracking branch 'origin/fix/1487-contextualversionconfli…

d7dff70

…ct-error-stop-kedro-running-when-dependency-clashes' into fix/1487-contextualversionconflict-error-stop-kedro-running-when-dependency-clashes Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

linting

66a1b7d

Signed-off-by: noklam <nok.lam.chan@quantumblack.com>

noklam merged commit 8b357c9 into main May 20, 2022

noklam deleted the fix/1487-contextualversionconflict-error-stop-kedro-running-when-dependency-clashes branch May 20, 2022 16:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle plugin registration failure ContextualVersionConflict with log instead of raising error #1542

Handle plugin registration failure ContextualVersionConflict with log instead of raising error #1542

noklam commented May 16, 2022 •

edited

noklam May 19, 2022

antonymilne May 19, 2022

noklam May 19, 2022

antonymilne May 19, 2022

noklam May 19, 2022

noklam May 19, 2022

noklam commented May 19, 2022

antonymilne left a comment

antonymilne May 19, 2022

antonymilne May 19, 2022

noklam May 19, 2022 •

edited

noklam May 19, 2022

antonymilne May 19, 2022

noklam May 19, 2022

antonymilne May 19, 2022 •

edited

antonymilne May 19, 2022

antonymilne May 19, 2022

antonymilne May 19, 2022

noklam May 20, 2022

antonymilne May 19, 2022

antonymilne May 19, 2022

noklam commented May 19, 2022 •

edited

merelcht left a comment

antonymilne left a comment

antonymilne May 20, 2022

noklam May 20, 2022

		logger.warning(KedroCliError(f"Fail to initialize {entry_point}"))
		logger.warning(exc)

	logger.warning(KedroCliError(f"Fail to initialize {entry_point}"))
	logger.warning(exc)
	logger.warning(f"Failed to initialise %s. Full exception: %s", entry_point, exc)

Handle plugin registration failure ContextualVersionConflict with log instead of raising error #1542

Handle plugin registration failure ContextualVersionConflict with log instead of raising error #1542

Conversation

noklam commented May 16, 2022 • edited

Description

Development notes

Compatibility issue:

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented May 19, 2022

antonymilne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam May 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antonymilne May 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented May 19, 2022 • edited

merelcht left a comment

Choose a reason for hiding this comment

antonymilne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented May 16, 2022 •

edited

noklam May 19, 2022 •

edited

antonymilne May 19, 2022 •

edited

noklam commented May 19, 2022 •

edited