python microservice refuses to start: setuptools dep conflict

[RafalSkolasinski]

Describe the bug

Integration and notebook tests are failing at scale.

Failure on starting python microservice

starting microservice
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (setuptools 41.4.0 (/opt/conda/lib/python3.7/site-packages), Requirement.parse('setuptools>=46.1'), {'setuptools-rust'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/bin/seldon-core-microservice", line 33, in <module>
    sys.exit(load_entry_point('seldon-core', 'console_scripts', 'seldon-core-microservice')())
  File "/opt/conda/bin/seldon-core-microservice", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/conda/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 100, in load
    module = import_module(match.group('module'))
  File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/microservice/python/seldon_core/microservice.py", line 16, in <module>
    from seldon_core.gunicorn_utils import (
  File "/microservice/python/seldon_core/gunicorn_utils.py", line 7, in <module>
    from gunicorn.app.base import BaseApplication
  File "/opt/conda/lib/python3.7/site-packages/gunicorn/app/base.py", line 11, in <module>
    from gunicorn import util
  File "/opt/conda/lib/python3.7/site-packages/gunicorn/util.py", line 26, in <module>
    import pkg_resources
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3250, in <module>
    @_call_aside
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3263, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/opt/conda/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (setuptools 41.4.0 (/opt/conda/lib/python3.7/site-packages), Requirement.parse('setuptools>=46.1'), {'setuptools-rust'})

To reproduce

1. Run integration / notebook testgs
2. Observe disaster

Expected behaviour

All tests passes.

Environment

master, kind 0.6.1, CI

[RafalSkolasinski]

It turns out this bug is not yet manifesting itself in the CI as it occurs only on newly build wrapper images and CI only builds wrapper images when it detects changes in the python code.

[axsaucedo]

@RafalSkolasinski can we close this now that the fixes have gone in?

[RafalSkolasinski]

Yes, should be closed by related PR.