Purpose: venv & modules management support

[huan]

When I'm developing my FaceNet project with python-bridge, I use venv to get an isolated Python environment, inside my project directory.

The reason to do this is that I want to do an Electron Package in the future, so I need all the python modules be there out-of-the-box. The venv directory includes the Tensorflow, Numpy, etc, cost about 500MB storage size.

Today, I'm thinking of if I should split the different neural network models to different NPM module, like: facenet for Google FaceNet, chinese-whisper for the Chinese Whisper Althorithm, `Mtcnn for the MTCNN model.

However, it turns out that if I use the current method to create those npm modules, they all will create a Python venv by themselves, then there will be lots of venv directories inside themselves, cause storage waste.

My purpose is: manage the venv by python-bridge, install all Python modules inside a venv directory of python-bridge.

Then, all other modules can share the venv directory via python-bridge, instead of creating & install by themselves.

The Code

API might look like:

python = pythonBridge({ venv: 'dir' })
if (!await python`'tensorflow' in sys.modules`) {
  await python.pip.install('tensorflow')
}

[munro]

All you should have to do to run the bridge in a virtualenv is pass in {python: venv_python_path, env: venv_env_variables}—and it should be good to go!

So this would work really well as a separate module, and I think you could get away with a very light wrapper around virtualenv so that you don't have to write too much glue code.

import { PythonVirtualEnv } from 'python-virtualenv'
import { PythonBridge } from 'python-bridge'
import { promisify } from 'utils';
import { join as path_join } from 'path';

declare interface PythonVirtualOptionsBase {
    python: string // python intepreter to use in env
    args: string // extra arguments to pass to virtualenv
}

declare interface PythonVirtualOptionsPath extends PythonVirtualOptionsBase {
    path: string, // path to virtualenv
}

declare interface PythonVirtualOptionsNamed extends PythonVirtualOptionsBase {
    name: string, // name of virtualenv, creates in ~/.virtualenvs/
    virtualenv_path: string | undefined, // override for ~/.virtualenvs
}

declare class PythonVirtualEnv {
    python: string; // Python path
    env?: { [key: string]: string | undefined; };
    created: boolean; // If it created the virtualenv on initializing

    constructor(PythonVirtualOptionsPath | PythonVirtualOptionsNamed)  // creates new virtualenv if doesn't exist
    unlink(): void; // delete virtualenv

    // child_process API, but everything runs in the virtual env
    exec(command[, options][, callback]);
    execFile(file[, args][, options][, callback]);
    fork(modulePath[, args][, options]);
    spawn(command[, args][, options]);
}

async function example() {
    const virtualenv_path = path_join(__dirname, '.virtualenv')
    const requirements_path = path_join(__dirname, 'requirements.txt');
    const virtualenv = new PythonVirtualEnv({python: 'python3', path: virtualenv_path});
    const exec = promisify(utils.virtualenv.exec);
    await exec('pip install pip==9.0.1');  // skips if installed
    await exec(`pip install -r ${requirements_path}`);  // skips if installed

    // run python bridge inside virtualenv
    const python = new PythonBridge({python: virtualenv.python, env: virtualenv.env});

    // install tensorflow
    if (!await python`'tensorflow' in sys.modules`) {
        await virtualenv.exec('pip', ['install', 'tensorflow'])
    }
}

[huan]

The python-virtualenv seems is what I want.

Will play with the code later. I believe it worth to be put in FAQ.

Thanks!