This repository has been archived by the owner on May 31, 2020. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Node script to run Python code using Batavia (#671)
* Added basic command to run some python code using batavia and node * The compile-and-encode script now outputs the filename and bytecode as JSON Cleaned up the run-in-batavia script and updated its description * Replace Python run-in-batavia script with Node-based run_in_batavia script * Output null if module is not found Directly compile Python file given a path Add current directory to Python path so imports work * Improved error handling. * Added documentation for the run_in_batavia.js command. * Added run_in_batavia.js and compile_module.py in the manifest file * Remove old version of loader. * Removed special-case handling of Python paths ending with .py Correction run_with_batavia -> run_in_batavia * Added npm script to allow running Python code as ``npm run python file.py`` * The module compile is now never passed a file name, just the module name * Updated tutorial and readme file for running Python code on command line using Batavia * Allow running ``npm run python`` from subdirectory of project
- Loading branch information
1 parent
218698b
commit 67fde32
Showing
8 changed files
with
262 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
# coding=utf-8 | ||
"""Compile Python module and returns its file name and base64-encoded bytecode | ||
as JSON. | ||
For use by 'run_in_batavia' script.""" | ||
|
||
import argparse | ||
import json | ||
import py_compile | ||
import sys | ||
|
||
import base64 | ||
import importlib | ||
import os | ||
import tempfile | ||
|
||
sys.path.insert(0, os.getcwd()) | ||
|
||
|
||
def get_module_path(module): | ||
module_spec = importlib.util.find_spec(module) | ||
|
||
# TODO: handle importing namespace packages | ||
if module_spec is None or module_spec.origin == 'namespace': | ||
return | ||
|
||
return module_spec.origin | ||
|
||
|
||
def python_module_to_b64_pyc(module): | ||
module_file = get_module_path(module) | ||
|
||
if module_file is None: | ||
return | ||
|
||
fp = tempfile.NamedTemporaryFile(delete=False) | ||
fp.close() | ||
|
||
try: | ||
py_compile.compile(module_file, cfile=fp.name) | ||
with open(fp.name, 'rb') as fin: | ||
pyc = fin.read() | ||
finally: | ||
os.unlink(fp.name) | ||
|
||
return { | ||
'filename': os.path.basename(module_file), | ||
'bytecode': base64.b64encode(pyc).decode('utf8'), | ||
} | ||
|
||
|
||
def main(): | ||
parser = argparse.ArgumentParser(description=__doc__) | ||
parser.add_argument('module', help='Python module') | ||
|
||
args = parser.parse_args() | ||
|
||
print(json.dumps(python_module_to_b64_pyc(args.module), | ||
indent=4)) | ||
|
||
|
||
if __name__ == '__main__': | ||
main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,4 +13,5 @@ reference objects and classes defined natively in JavaScript. | |
|
||
tutorial-0 | ||
tutorial-1 | ||
tutorial-2 | ||
faq |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
Tutorial: Running Python code using Batavia from the command line | ||
================================================================= | ||
|
||
Batavia includes a simple command-line script that you can use to run your | ||
Python code. To use this tool you need to have followed the instructions from | ||
:doc:`tutorial-0`. | ||
|
||
You can now run Python code from a code from the command line as follows: | ||
|
||
.. code-block:: bash | ||
npm run python /path/to/python_file.py | ||
This runs the ``run_in_batavia.js`` script which in turn runs the Python code. | ||
This command will only work if you call it within the Batavia project directory | ||
and provide it the absolute path to the Python file to run. | ||
|
||
You can alternatively directly run the ``run_in_batavia.js`` in Node. If | ||
you are not in the Batavia project directory you can still use this script as | ||
follows: | ||
|
||
.. code-bloc:: bash | ||
|
||
node /path/to/batavia/run_in_batavia.js /path/to/python_file.py | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,153 @@ | ||
#!/usr/bin/env node | ||
/** | ||
* Run Python file in a new Batavia VM. | ||
*/ | ||
|
||
const fs = require('fs') | ||
const path = require('path') | ||
const {spawnSync} = require('child_process') | ||
const batavia = require('./batavia/batavia.js') | ||
|
||
// The compile_module script is in the same directory as this script. | ||
const compileScriptPath = path.join(__dirname, 'compile_module.py') | ||
|
||
function displayHelpText() { | ||
console.log(` | ||
usage: [-h] file | ||
Runs Python file in node JS using using the Batavia virtual machine in Node. | ||
positional arguments: | ||
file Python file to run | ||
optional arguments: | ||
-h, --help show this help message and exit`.trim()) | ||
} | ||
|
||
function main() { | ||
if (process.argv.length === 3) { | ||
const argument = process.argv[2] | ||
|
||
if (argument === '-h' || argument === '--help') { | ||
displayHelpText() | ||
return | ||
} | ||
|
||
let resolveBase | ||
// The INIT_CWD environment variable is set in npm v5.4 and above. It | ||
// contains the directory in which the npm was run. If we have that | ||
// information we can resolve paths relative to it. | ||
// If the file path not relative and INIT_CWD is defined, we'll resolve | ||
// the provided path relative to it. | ||
if (process.env.INIT_CWD && !path.isAbsolute(argument[0])) { | ||
resolveBase = process.env.INIT_CWD | ||
} else { | ||
resolveBase = '' | ||
} | ||
|
||
const filePath = path.resolve(resolveBase, argument) | ||
const modulePath = path.basename(filePath, '.py') | ||
const basePath = path.dirname(filePath) | ||
|
||
fs.access( | ||
filePath, | ||
fs.constants.F_OK | fs.constants.R_OK, | ||
function(err) { | ||
if (err) { | ||
console.log( | ||
'File "' + argument + '" does not exist ' + | ||
'or cannot be accessed by current user.') | ||
} else { | ||
try { | ||
runInBatavia(basePath, modulePath) | ||
} catch (error) { | ||
if (error instanceof TypeError) { | ||
console.log('Invalid Python file.') | ||
} else throw error | ||
} | ||
} | ||
}) | ||
} else { | ||
displayHelpText() | ||
} | ||
} | ||
|
||
/** | ||
* Runs the specified Python module | ||
* | ||
* @param basePath | ||
* @param module | ||
*/ | ||
function runInBatavia(basePath, module) { | ||
const vm = new batavia.VirtualMachine({ | ||
loader: makeBataviaLoader(basePath), | ||
frame: null | ||
}) | ||
|
||
vm.run(module, []) | ||
} | ||
|
||
/** | ||
* Creates a loader function for the Batavia VM that looks for code around the | ||
* specified base path. | ||
* | ||
* @param basePath | ||
* @returns {bataviaLoader} | ||
*/ | ||
function makeBataviaLoader(basePath) { | ||
/** | ||
* Compiles the specified Python module and returns its bytecode in the | ||
* format that Batavia expects. Returns null if module was not found so an | ||
* ImportError can be raised. | ||
* | ||
* @param modulePath | ||
* @returns {*} | ||
*/ | ||
function bataviaLoader(modulePath) { | ||
const compileProcess = spawnSync( | ||
'python3', | ||
[compileScriptPath, modulePath], | ||
{cwd: basePath} | ||
) | ||
|
||
checkForErrors(compileProcess) | ||
|
||
const module = JSON.parse(compileProcess.stdout) | ||
|
||
// Module compiler will output null if the specified module was not | ||
// found. | ||
if (!module) { return null } | ||
|
||
return { | ||
'__python__': true, | ||
'bytecode': module.bytecode.trim(), | ||
'filename': module.filename.trim() | ||
} | ||
} | ||
|
||
return bataviaLoader | ||
} | ||
|
||
/** | ||
* Checks the Python compile process result for errors. In case of error it | ||
* alerts the user quits the program. | ||
* | ||
* @param processResult | ||
*/ | ||
function checkForErrors(processResult) { | ||
if (processResult.error) { | ||
console.log( | ||
'There was an error running the Python compile process.\n' + | ||
'Ensure that Python 3 is installed and available as "python3".') | ||
process.exit(1) | ||
} | ||
if (processResult.status !== 0) { | ||
console.log('There was an error during import.') | ||
console.log(processResult.stderr.toString()) | ||
process.exit(1) | ||
} | ||
} | ||
|
||
if (require.main === module) { | ||
main() | ||
} |