ContextualVersionConflict Error Stop Kedro running when dependency clashes

[rashidakanchwala]

Description

ContextualVersionConflict error arises on two ocassions when using Kedro :-

• when you install a Kedro plug-in with a dependency that clashes with a kedro, or a kedro-viz dependency
• when you do %reload_kedro on Databricks

Steps to Reproduce on Databricks

1. clone a kedro project on github
2. make sure you have kedro-viz installed on Databricks in the same place as kedro (i.e. Cluster Library or using pip command)
3. load kedro IPython extension
4. try using %reload_kedro. line magic command

Steps to Reproduce outside Databricks

1. for a kedro-plugin, in the setup.py add to install_requires a dependency that clashes with kedro
2. pip install the kedro-plugin
3. do a kedro run when in the project folder

Actual Result

Expected Result

Instead of an error message which breaks Kedro, it should just display a warning maybe?

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): 0.18
• Python version used (python -V): 3.8
• Operating system and version: Linux

[rashidakanchwala]

@AntonyMilneQB - please review.

[noklam]

I think this shows up even when we do kedro or any kedro xxx commands, I am not 100% sure if it raise the same ContextualVersionConflictError, but it is definitely related to the dependencies conflict.

[noklam]

I can reproduce the error with these steps.

conda create -n test python==3.8.5
conda activate test
pip install kedro kedro-viz
pip install pip==21.0.0
kedro new

conda create -n test python==3.8.5
conda activate test
pip install kedro kedro-viz
pip install plotly<4.0 # kedro-viz requires >=4.0
kedro new

pip version shouldn't really matter if we already have the library installed, nonetheless it still crash the program and prevent it runs.

    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pip 21.0 (/Users/Nok_Lam_Chan/miniconda3/envs/test/lib/python3.8/site-packages), Requirement.parse('pip>=21.2'), {'pip-tools'})

[noklam]

@rashidakanchwala Is this a Databricks specific issue or it would happen locally too?

[antonymilne]

This is indeed not a databricks (or kedro-viz) specific issue, but rather something related to plugins in general.

I believe the solution would be to change these lines

kedro/kedro/framework/cli/utils.py

Lines 334 to 337 in 676600c

	try:
	entry_point_commands.append(entry_point.load())
	except Exception as exc:
	raise KedroCliError(f"Loading {name} commands from {entry_point}") from exc

to instead catch the error and emit a warning rather than throw a KedroCliError. There might be other places in the codebase where we should make the same change (e.g. _init_plugins).

[noklam]

Have a quick look this morning, maybe tricker than I thought. Seems like this is not the only way the entrypoint is loaded, the USER/env/bin/kedro also loads the entry point. Will share more details later.

[antonymilne]

@noklam Could you explain that a bit more? 🤔