KWIVER Python fails to install PyTorch to venv

[dstoup]

Building the latest master, 74a818b

1 - Fletch was built with ENABLE_ALL_PACKAGES=ON, ENABLE_PYTHON=ON
2 - KWIVER config set from the default with the following packages ON:
KWIVER_ENABLE_ARROWS
KWIVER_ENABLE_C_BINDINGS
KWIVER_ENABLE_PROCESSES
KWIVER_ENABLE_PYTHON
KWIVER_ENABLE_SPROKIT
KWIVER_ENABLE_TESTS
KWIVER_ENABLE_TOOLS

The following options were enabled by default from finding Fletch packages:
KWIVER_ENABLE_CERES
KWIVER_ENABLE_FFMPEG
KWIVER_ENABLE_GDAL
KWIVER_ENABLE_MVG
KWIVER_ENABLE_OPENCV
KWIVER_ENABLE_PROJ
KWIVER_ENABLE_QT
KWIVER_ENABLE_SUPER3D
KWIVER_ENABLE_VXL

Note:
KWIVER_ENABLE_PYTORCH OFF

3 - source setup_KWIVER.sh - My build can't find certain libraries required for tests without this step and build will fail otherwise.
4 - make -j10 -k

My build fails with the following message:

ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from -r /home/david/kitware/code/vision/kwiver/src/python/requirements.txt (line 7)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.4.0 (from -r /home/david/kitware/code/vision/kwiver/src/python/requirements.txt (line 7))

There are two concerns here:
1 - Why can't pytorch install into my venv?
2 - Why are we even trying to install pytorch when I've disabled it?

Not related, but noticed. Here a built again with a single core to find the step involved:
3 - The build hangs for a very long time at
[ 76%] Pip installing Python dependencies...

Also, I noticed this message during the build .. wanted to post in case it's relevant.

[ 14%] Building CXX object python/kwiver/vital/util/CMakeFiles/vital_python_util.dir/python_exceptions.cxx.o
WARNING: The scripts pip, pip3 and pip3.9 are installed in '/home/david/kitware/code/vision/kwiver/test/testing_venv/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

[as6520]

Hi Dave, The pytorch installation is originating from the requirements.txt. We had discussed the problem in #1055 (comment) when pytorch arrow was being reviewed. I don't think that is a good way to specify the optional requirement based on CMAKE option.

As for the error, what's the version of pip3 on your machine?

[johnwparent]

This occured when CMake created the venv during the build?
Found this SO post that has a number of potential reasons that pytorch wasn't able to be installed. It seems like adding the -f flag to the pip install for pytorch has solved the problem in a number of cases, or ensuring that python and system are both 64bit.
To answer the second part, the venv is grabbing the req.txt, and doesn't consider the CMake flags, so the venv installation during build could certainly be more considerate of that, if that is what's happening here.
Additionally, the pip install into the venv shouldn't break a build, on failure it should just report to that end, and finish as it's logically or'd with the CMake echo command, so that's puzzling to me.

[dstoup]

@as6520 thanks for posting the MR comments. That makes sense but I think I need to digest it a bit to get a more clear picture.

@johnwparent I updated the issue report above with a bit more info. It happens here:
[76%] Pip installing Python dependencies...

I also added a message I saw fly by about not having the venv's bin directory in my PATH. Not sure what impact if any that would have on this issue but thought I should include it anyway. See above.

In the end, it's a build failure. I still feel like pytorch shouldn't be installed when it's not enabled. But, given that it's listed in the requirements.txt, that's a little trickier. For now I will disable python but I don't love having to do that. It looks like the folks in that SO post are using pytorch.org directly to install it. Do we need to add that repo in certain cases?

EDIT @johnwparent :

Sorry, tired and missed some of the details you mentioned above.

"It seems like adding the -f flag to the pip install for pytorch has solved the problem in a number of cases, or ensuring that python and system are both 64bit."

My system has full 64-bit support. It's multilib so also supports 32-bit but that shouldn't impact this issue. It's worth a try adding -f and I can do that now as a test. I will look for the pip install step and add it to test.

" the venv is grabbing the req.txt, and doesn't consider the CMake flags, so the venv installation during build could certainly be more considerate of that ..... Additionally, the pip install into the venv shouldn't break a build"

That's probably a good idea. I don't think we have any other python packages specifically listed as a CMake option so it might be a lot of effort to build a custom req.txt for the build step just for pytorch, but do agree that regardless of that, it would be good if it wasn't a hard build error.

Another Edit:

Sorry, I just noticed that it's not actually a hard build error. It looked that way at first but might have been something different. It fails on pytorch, gives and ugly error message but then does seem to finish the build. Not sure if it happens that way because I've run the build a few times now but will test that as well.

[johnwparent]

@dstoup At the very least, I can say that extra message about the venv not being in the path is ok, when the pip install occurs the venv is not active, which as far as I can tell is fine, and the venv/bin is added to the path before any of those packages are used.

The pip install command can be found on line 280 of kwiver-setup-python.cmake.

Pip can be imported like other packages, so we could use it via a python script that enables/disables packages based on CMake variables, I can't see that being very difficult to create, and could allow for more flexibility going forward. If that sounds like something you would like to see, I can start on that.

The non failure/ugly error should be that way, the pip command is or'd into the CMake echo command which should report the failure to the command line, and then continue the build. The build hangup on the pip install is more than likely a result of how many packages we're installing and how long it takes pip to install them.

[dstoup]

Thanks John, see my comment about the the not actually failing. I was deceived by the output and didn't notice that it is just an ugly warning but the build finished.

I don't know if I care too much about the addition of pytorch to venv. As long as it's not breaking the build, I think it's ok to leave as is. That does beg the question though, will the pytorch arrow use the venv pytorch? I don't think I have it installed on my system and the arrow does configure, so I'm assuming we're consistent about using the venv version in this case but not sure.

One more clue I found which is most likely the cause of the issue based on the following comment in SO ....

"This can also happen if your Python version is too new. Pytorch currently does not support past 3.7.9."

Python and testing enabled.
Searching for Python3 Interp...
Found Python3: /usr/bin/python3.9 (found version "3.9.0") found components: Interpreter
Creating virtualenv @/home/david/kitware/code/vision/kwiver/test/testing_venv for testing...'/usr/bin/python3.9' '-m' 'venv' '/home/david/kitware/code/vision/kwiver/test/testing_venv'

Interestingly, the CMake message before the one I posted is this:

-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.7.9", minimum required is "3.4")
-- python_site_packages = lib/python3.7/site-packages
-- kwiver_python_install_path = /usr/local/lib/python3.7/site-packages
-- Found PythonInterp: /usr/bin/python3 (found version "3.7.9")

That means that the venv creation is not using the same python executable that CMake found originally.

I have several python 3 versions installed. My system default is 3.7.9 but we're preferring the most recent 3.9. If the SO comment is correct, that's probably the reason.

[Erotemic]

That means that the venv creation is not using the same python executable that CMake found originally.

Looks like we need to find the locations that aren't using PYTHON_EXECUTABLE defined by kwiver-setup-python.cmake to invoke python commands.

Looks like CMake/activate_venv.sh.in is using conda instead of respecting PYTHON_EXECUTABLE.

[johnwparent]

@Erotemic The location causing the issue where PYTHON_EXECUTABLE is not respected is in kwiver-setup-python.cmake line 221. It was included to guarantee that the venv module could be used to construct a virtual environment. A check should have been added to confirm if PYTHON_EXECUTABLE is v2 or v3, and to then default to PYTHON_EXECUTABLE if v3. The problem is really caused by python3.9 being found by the call at Line 221 in the aforementioned file. If we were to default to PYTHON_EXECUTABLE if v3, is there a guarantee the earlier command would not also potentially find Python3.9?

That conda command is used only to activate or deactivate a virtual environment if Anaconda is the discovered Python environment, and a virtual environment was successfully created. I may be way off base, and please correct me if I am, but I believe that would not result in this error if the earlier line was not causing it.