Inconsistent python dependencies "tlcpack-sphinx-addon, synr" in CI and Docker setup scripts

[leandron]

There are some dependencies that are inconsistently declared within Docker setup scripts and some CI test scripts. The cases I noticed are declared within docker/install/ubuntu_install_python_package.sh, tests/scripts/task_ci_python_setup.sh and tests/scripts/task_ci_setup.sh with different version constraints:

1. tlcpack-sphinx-addon is missing from the Docker setup
2. synr version specification is inconsistent

tvm/docker/install/ubuntu_install_python_package.sh

Lines 23 to 24 in f449070

	# install libraries for python package on ubuntu
	pip3 install six numpy pytest cython decorator scipy tornado pytest pytest-xdist pytest-profiling mypy orderedset attrs requests Pillow packaging cloudpickle synr

tvm/tests/scripts/task_ci_python_setup.sh

Lines 32 to 33 in f449070

	python3 -m pip install --user tlcpack-sphinx-addon==0.1.4 synr==0.3.0

tvm/tests/scripts/task_ci_setup.sh

Lines 32 to 33 in f449070

	python3 -m pip install --user tlcpack-sphinx-addon==0.1.4 synr==0.3.0

I'm opening this as an issue, and not a PR directly, because I wanted to see how we can improve this, in line with https://discuss.tvm.apache.org/t/rfc-python-dependencies-in-tvm-ci-containers/9011.

cc @tqchen @areusch @Mousius @u99127

[tqchen]

Thanks @leandron Because these two deps can be updated more frequently(and ci image update cycle is slower), we choose to use ci_setup script to install them instead of hard baking the CI image. We should also remove synr from ci the docker image for now.

[areusch]

yeah I also hit this problem. I think we need to fix the CI image update process so it is not such a burden. My opinion is we should implement one method to include dependencies in TVM CI and support that method, and avoid making CI-time additions like these two. I hit this problem with updating ci-arm and ci-cpu containers as well.

I think if we make the CI image update process less of a burden, it wouldn't be a big deal to ask us to update all ci-* containers at once, rather than having ci-cpu built 3 weeks ago and ci-gpu built 3 months ago.

What's not clear to me in this particular case is why synr and tlcpack-sphinx-addon are pinned. @tqchen do you have more insight into that--is it just to ensure the CI is predictable (I.e. because these packages are installed every CI run rather than each container build), or was there another reason?

[leandron]

yeah I also hit this problem. I think we need to fix the CI image update process so it is not such a burden. My opinion is we should implement one method to include dependencies in TVM CI and support that method, and avoid making CI-time additions like these two. I hit this problem with updating ci-arm and ci-cpu containers as well.

I think if we make the CI image update process less of a burden, it wouldn't be a big deal to ask us to update all ci-* containers at once, rather than having ci-cpu built 3 weeks ago and ci-gpu built 3 months ago.

I think updating all images at once would be really good, in terms of keeping track of changes that happen in our dependencies.

Specifically in this case, even if we rebuild (not from scratch) the image, we'll still end up with the old synr version, because Docker will use the cached layer, as the hash of script that install those dependencies didn't change, hence the need for the script to perform the ad-hoc dependency installation.

So, I think your proposal comes in very handy. When any python dependency version change, that consolidated file hash will change, and trigger a proper rebuild of the layers.

[tqchen]

@areusch they are pinned because we want to make sure no cache is needed(e.g. if we change version there will be automatic update process, but won't reinstall other wise).

Note that in the best case scenario we still need a two step in order to update the image(binary image change then the code change) so having a setup hook for fast changing deps might be handy so the developers can directly see the effect of their change. I agree though that if the process improves or the deps stablizes(both will happen) we will get to the other land where docker should be the only dep manager