Conda installation guide

[PaulaMihalcea]

The current installation guide is unclear about the actual usage of the psbody-mesh package after building it, as well as impossible to execute on machines without sudo permissions.

The installation part of the documentation has been thus rewritten in order to make it more clear and complete, as well as sudo-free (by using Anaconda).

The parts with the download in the site-packages folder and the renaming of the mesh folder to psbody is what actually makes using this package possible, and this rectified version of the guide contains the cleanest possible way of doing it - the alternatives being copying the mesh folder after compilation in the folders of the projects that use it (bad) and/or editing the import lines in the relative scripts (even worse), as well as the PYTHONPATH variable.

[raffienficiaud]

The current installation guide is unclear about the actual usage of the psbody-mesh package after building it, as well as impossible to execute on machines without sudo permissions.

Thanks. The only difficult part is about Boost, and the easiest way to install it on Linux/OSX is the way described in the document. I personally never use Conda so I would not make it a requirement for installing the mesh package. Rather I would help the user get Boost installed if sudo rights are lacking (very well explained on Boost website).

Conda is however a widely used system, any instructions to get mesh working with conda are valuable. I would prefer having a dedicated section such as "install with conda" instead of replacing the existing one. However the hacky solution is a no-go for me.

If you look at the make all command, it is only about 2 instructions:

pip install --upgrade -r requirements.txt
pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .

I am not familiar with conda but I guess we can emulate those 2 instructions with vanilla conda commands that do not require the cloning and renaming steps?

[PaulaMihalcea]

The only difficult part is about Boost, and the easiest way to install it on Linux/OSX is the way described in the document. I personally never use Conda so I would not make it a requirement for installing the mesh package. Rather I would help the user get Boost installed if sudo rights are lacking (very well explained on Boost website).

I must have totally missed the part in the Boost documentation where they say how to install it without sudo rights, which is the reason why I used Conda. My bad. At this point though I am leaving this for posterity, as I am in no way knowledgeable about Boost.

Conda is however a widely used system, any instructions to get mesh working with conda are valuable. I would prefer having a dedicated section such as "install with conda" instead of replacing the existing one.

I must admit that I was not sure about replacing the original instructions with the Conda version, but then I assumed that Conda was widespread and did it anyway. Since you are proof that not everybody uses it, it is only fair that we leave both alternatives in the guide.

However the hacky solution is a no-go for me.

Well, it is a no-go for me, too, but is the only way that works. Please believe me when I say that I have tried a lot of solutions. I absolutely do not exclude the possibility that I missed some vanilla conda instruction for doing it in a nicer way, but then again, I did not find anything else.

If you look at the make all command, it is only about 2 instructions:

pip install --upgrade -r requirements.txt
pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .

I am not familiar with conda but I guess we can emulate those 2 instructions with vanilla conda commands that do not require the cloning and renaming steps?

I do not get it, the cloning is necessary to get the files needed for building the psbody-mesh package from source; how could someone compile it without downloading the files, and what better way of doing it other than cloning into the official repository? In the past I stumbled upon other libraries that needed compiling from source, and they basically did the same thing (an example is PyTorch, see here).

The renaming step can be skipped by executing only git clone https://github.com/MPI-IS/mesh ~/anaconda3/envs/my_venv/lib/python3.8/site-packages/psbody at step 4, which basically does the same thing (clone the psbody-mesh source code in a folder called psbody in the site-packages folder), but in one command.

I would like to specify that the renaming part is necessary because the default name of the psbody-mesh repo is mesh, while the library is supposed to be imported in Python as import psbody.mesh (at least this is how I found it in scripts that used it, like here or here); if you do not rename it, then the import statements must be changed to import mesh.mesh (as the parent folder is mesh), which does not seem to be the way that it was intended to be imported in.

As for the cd part, I found that it can be avoided by specifying the psbody-mesh folder directly in the make all command, so step 5 becomes BOOST_INCLUDE_DIRS=~/anaconda3/envs/my_venv/include/ make -C ~/anaconda3/envs/my_venv/lib/python3.8/site-packages/psbody all. Personally, I prefer keeping the cd part for the sake of clarity, but I just tested it and either way works just fine.

There might be a way of building psbody-mesh from source using Conda commands, see Building conda packages from scratch, and then installing it from the generated package. Personally, I did not even think about searching for something like this; everything I know about Conda I actually learnt while trying to get psbody-mesh to work, so many aspects are new to me, too. This solution however looks more like something for the developers than for the final user - it even says how to upload packages to Anaconda for everyone to use. At the moment I have to meet a tight dealine on a project I am working on (indeed, the one where all this started for me), so I am not in a position to further investigate this approach, as it goes well beyond my knowledge about Conda. It does look promising, but considering that the building requires Boost, there might be some complications along the way, and I am not sure if it would be worth examining if the goal would only be creating a more elegant solution.

Took me a while to understand what you meant. If all what make all does consists of running two pip commands, then it should be as easy as running them in the activated conda environment instead of using make all, so after cloning in any folder, step 5 would literally become:

$ pip install --upgrade -r requirements.txt
$ pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir

However, I just tried it and it does not work; Python throws ModuleNotFoundError: No module named 'psbody'.

Since when using Conda it is recommended to actually stick with conda and not use pip (some truly valuable advice), I tried replacing the first pip command with conda, so $ pip install --upgrade -r requirements.txt translates to:

$ conda install --file requirements.txt
$ conda update --file requirements.txt

Too bad that the very first command (conda install) throws an error, too, namely a PackagesNotFoundError for opencv-python (which looks like it should be installed separately), so I could not continue. One would probably need to add some new channels to the default Conda channels list by using conda config --append channels CHANNEL_NAME, but at this point the requirements.txt kinda loses meaning since it would be easier to manually install each required package instead of manually adding the correct channel for each/most of them before running the conda install command.

After these new findings I am still convinced that the simplest solution is the hacky, no-go way, however ugly it might seem; I remain open to further suggestions, though, possibly from someone more experienced with Conda than us.

I updated the committed README.md file and restored the original installation guide by separating it from the Conda version; I also simplified steps 4 and 5 with the mentioned commands (those that still work).

[raffienficiaud]

There are various things here, but I think we start getting on the right path:

• you are cloning directly inside the virtual environment. If you look again at the pytorch installation, in particular this section, after installing various common conda package, a proper python setup.py install step is performed. This installs the package properly into the virtual environment. You should not be needing to clone to site-package at all ever.
• the renaming: this is a functionality of setup.py called (if my memory does not fail me) meta package or something. This means that the layout of the source and the layout of the installed package are different. We are not sticking to the source layout.
• If the requirements.txt as is does not work, I think we can just find the right equivalent conda packages. opencv should be available. I am all fine with not using a requirement file and having that step in the README you are editing (just like the pytorch installation step).
• what pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir . is no more that some python setup.py install with additional options and a clean temporary (and isolated) folder for compiling the stuff. The only specific part is the option that passes the location where Boost is (not needed if Boost installed system-wide) and possibly some setuptools specific stuff that I am pretty sure conda is compatible with. That to say that if we can pass the requirement part, maybe this part will solve by itself.

[PaulaMihalcea]

There are various things here, but I think we start getting on the right path

Yes, we are. Keep reading!

* you are cloning directly inside the virtual environment. If you look again at the pytorch installation, in particular [this section](https://github.com/pytorch/pytorch#install-pytorch), after installing various common conda package, a proper `python setup.py install` step is performed. This installs the package properly into the virtual environment. You should not be needing to clone to `site-package` at all ever.

Oh, you meant cloning into that specific folder! I thought you meant that the user was not supposed to ever clone into the repo at all. Alright, it makes sense now; I agree that it is not a good practice, I only did it this way to avoid moving the downloaded folder.

* the renaming: this is a functionality of `setup.py` called (if my memory does not fail me) `meta package` or something. This means that the layout of the source and the layout of the installed package are different. We are not sticking to the source layout.

I know nothing about this stuff, but earlier I happened to see this line in the psbody-mesh setup.py file, and so I think I understand what you mean.

* If the `requirements.txt` as is does not work, I think we can just find the right equivalent conda packages. `opencv` should be available. I am all fine with not using a requirement file and having that step in the README you are editing (just like the pytorch installation step).

Yeah, opencv should work just fine and is available from more conventional Conda channels like conda-forge. While I do like the concept of requirements files, I can say from experience that tons of problems arise from them (they contain the wrong versions of the packages, or contain no versions at all, stuff like this), so installing them manually can actually allow a more inexperienced user to understand what he/she is doing (e.g. when I first started the project I am working on, I had no idea what these requirements.txt files had inside, and I could not figure out why I kept getting errors during package installation - then I realized they were nothing more than a list of packages, and proceeded with their manual installation).

* what `pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .` is no more that some `python setup.py install` with additional options and a clean temporary (and isolated) folder for compiling the stuff. The only specific part is the option that passes the location where Boost is (not needed if Boost installed system-wide) and possibly some `setuptools` specific stuff that I am pretty sure `conda` is compatible with. That to say that if we can pass the requirement part, maybe this part will solve by itself.

This is where the good news come in play. I did some more researching after writing my previous comment and realized that in this case using pip in a Conda environment is the right way of proceeding. At this point I thought that not having run python setup.py install when trying to translate the make all command could have been the culprit. I then tried adding this command after pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir (I did not know that they did the same thing) and it worked, so I thought I solved the problem.

Just in case, I created another virtual environment and tried to do it again, and surprise! it did not work. It took me another 30 minutes of extensive research to notice a small detail in your pip install command, something which looked like a typo, but was definitely intentional: the final dot .. Yep, I omitted the dot the first time I tried the pip commands, and missed the error statement in the countless output lines that flooded the terminal... and that is why pip did not work.

So in conclusion, by completely skipping steps 3, 4 and 5 and executing the following instead, the import statement works correctly:

$ git clone https://github.com/MPI-IS/mesh
$ pip install --upgrade -r mesh/requirements.txt
$ pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir mesh/.

And an even nicer thing here is that you can safely delete the mesh folder cloned with git: the imports will still work, since the installation has been executed in a correct way. I just edited the README.md file accordingly.

[raffienficiaud]

LGTM, @jcpassy ?

[raffienficiaud]

All looks good to me @PaulaMihalcea !

[PaulaMihalcea]

Coming back with a quick tip:

if for some reason the second pip command at step 4 does not work (pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir mesh/.), try replacing $$BOOST_INCLUDE_DIRS with the path to the Boost installation of the virtual environment you are using, which should be something like ~/anaconda3/envs/my_venv/include/; for example:

pip install --no-deps --install-option="--boost-location=~/anaconda3/envs/my_venv/include/" --verbose --no-cache-dir mesh/.

Remember to also replace my_venv with the name of your virtual environment, if different.

[jcpassy]

Hi @PaulaMihalcea and @raffienficiaud ,
very sorry for my slow response. Thank you @PaulaMihalcea for your PR, and thank you very much @raffienficiaud for your comments and your review.

Yes, that looks good to me too. Merging!