Setup Python environment and fix python features for packages

[cmpute]

Fix #2974.
Fix possibly #2370

Modifications

• Now installing python3 with executable feature will setup python environment in installed directory python/ and debug/python/ with full Libs and site-packages folder. You can actually use the python.exe in it as a kind of python distribution.
• Add pip to installed python as python-setuptools
• Cython and Numpy are now avaliable.
• Python binding building for vtk, ogre and pybind11 are fixed to fit the folder structure. (SWIG are added to acquired modules for ogre, while ogre cannot be built successfully here on my PC because of some SWIG issues). Generated python modules will be installed to (debug/)python/Lib/site-packages.
• Opencv can be built with python bindings now (which depends on numpy).
• vcpkg env --python will add python/Lib/site-packages/ to PYTHONPATH instead of python/
• Built python packages can be uninstalled by vcpkg remove <...>, however, cache modules which are compiled by python will not be removed by vcpkg remove python3.
• Further python packages (for example numba, pyqt, gdal) that need to be compiled can take advantages of scripts installed by python-setuptools

[miurahr]

It may be necessary to increment version number.

[miurahr]

Need version increment.

[miurahr]

When change made, version should be incremented.

[miurahr]

Need version increment such as 2.2.1-1

[miurahr]

Need version increment.

[cmpute]

Thanks for reminding. Fixed.

[cmpute]

Is this branch able to be merged? I am going to add some python packages but I don't want to make this PR so large.

[ras0219-msft]

Thanks for the PR and sorry for the delays!

First, I'd like to re-establish the core purpose of Vcpkg: we want to solve the problem of third party libraries for C and C++ developers so they can build their own applications. We aren't trying to become an application manager for Windows -- that's Chocolatey or the Windows Store.

We also aren't trying to be a package manager for python: there's already pip and https://docs.anaconda.com/. I think they've done an amazing job at this and we can't (and shouldn't!) dream of outdoing them in the realm of packaging python.

Given this, could you help frame the need for building out a parallel package graph of the core python packages? To be clear, I'm not saying this isn't a great PR -- I just need some help understanding :)

[cmpute]

My main purpose for reorganize the structure for python is to help C/C++ libraries build and distribute their python bindings. Major reasons are listed as follows:

• Previously, packages which have python bindings have a disordered installation path structure. This would make users confused. At least I want to make all python bindings installed under a single regular folder (i.e. python/Lib/site-packages).
• Python itself has only core library built which is not quite enough for developers. Python (the executable) built under debug mode is needed for quite a lot developers, and debug mode python should not use python modules under release mode. This is why I separate /python and /debug/python. (Python objects have different structures under the two mode. Mixing the modules built in different modes may lead to fatal problems)
• Python bindings for C++ libraries may also need python modules backward. The most well-known example is numpy which is needed by opencv and caffe to build their python bindings. So in order to support this, I add python-numpy port and its dependencies (i.e. python-setuptools, python-cython)
• [To be discussed] Also, many python modules are majorly made up of C/C++ codes (e.g. numpy, gdal). These modules may (heavily) depends on C/C++ libraries (not only library files, but also header files). I think supporting building for this kind of modules may not be excluded from vcpkg.

To make my motivations clear, there are some things that I didn't intend to do:

• Add python distribution to vcpkg
• Make vcpkg to be able to build massive normal python modules.
• Add python package manager to vcpkg. (python-setuptools is only added to support building numpy)

In addition, PyPI is good for python users but not good enough for python developers. You may take a look at how we have to search for python wheels for some packages here.

[cmpute]

Any concerns preventing this from being merged?

[TheAifam5]

I made pull request about python2 port: #3999
It's based on python3 so will be nice if the python2 port will be merged first, then you can make fixes for python2 port also :) (if there is any)

Also IMHO would be nice to split the pull request into multiple parts.

[Fleischner]

This looks very nice. Any progress here?

I understand the argument that vcpck is not an package manager for python. However as I understand the goal here is very different: all ports that distribute python bindings with their original source (eg opencv) should work out of the box with python-embedded (!) build by the vcpkg python port.

So when launching an python environment from source, installed ports should be available and just work. As far as I understand using binary distributions or anaconda does not work here, because this will use different incompatible dlls from this distribution. (?)

Embedded python usage example:

1. use vcpck to build opencv with python => This will generate opencv dlls and python dll and populate site-packages folder as suggested
2. in your c++ application the following pseudo code 3-liner:
   Py_Initialize();
   PyRun_SimpleString("import sys\n sys.path.append('./site-packages/')");
   PyRun_SimpleFile("myopencvdemo.py");
   should start an python environment that when using opencv calls back into the same (!) dlls generated in step 1
3. Maybe it is necessary to copy some of the dlls and the site-packages folder to your app bin directory by hand as it is not automatically resolved by vcpck integrate.

[cmpute]

@ras0219-msft @Rastaban @vicroms @grdowns Will you accept this pull request if I fix it with current portfiles? It's hung up for a long time.

[TheAifam5]

@cmpute Please fix the merge issues. Otherwise they are not able to merge your changes.

[JackBoosY]

Hi @cmpute, thanks for this PR! Could you resolve these conflicts?

[dan-shaw]

@cmpute Going to close this PR as this PR has been open too long. If you are interested in working on this, feel free to reopen, but it might make more sense to make a new PR (otherwise we might lose track of it).

[dan-shaw]

Also I realize we haven't given much feedback on this PR for a long time. If you are still interested in some feedback, let me know and we can reopen this and prioritize this PR.