[RFC] A fake module from source API (mostly for wasm targets)

[youknowone]

Summary

Provides an API VirtualMachine::compile_module that creates a Python module from source code.

Detailed Explanation

We can use vm.compile and builtins.exec to provide dynamic libraries without system IO.

Below is not actual code, but it shows the general flow.

let scope = rustpython_vm::scope::Scope::with_builtins(None, vm.new_dict());
let code = vm.compile(<source text>, "<source>", Mode::Exec); // compiled code
vm.run_code_obj(code_obj, scope);
// now `scope` contains module attributes
let module = PyModule::new();
for (name, value) in scope.locals {
    module.set_attr(name, value);
}

We can package these complex procedures into a single function.

The WASM architecture limits the use of Python modules due to access to system io.
Currently, the only way to use Python modules in WASM is to freeze them at compile time.
This API is not only limited to WASM, but will be very useful for WASM.

Drawbacks, Rationale, and Alternatives

• Better API name or space? I didn't consider putting import in the name, like vm.import_from_source, because in Python, import implies complex filesystem usage via importlib.

Unresolved Questions

[youknowone]

@DimitrisJim @coolreader18 Do you have advices?

[coolreader18]

Not relevant to naming, but I thought the standard procedure was to make the scope's locals be the actual dict of the module itself?

[coolreader18]

Ok, I thought there was some sort of function like this already, found it:

RustPython/wasm/lib/src/vm_class.rs

Lines 274 to 309 in bdb0c8f

	pub fn inject_module(
	&self,
	name: String,
	source: &str,
	imports: Option<Object>,
	) -> Result<(), JsValue> {
	self.with_vm(|vm, _| {
	let code = vm
	.compile(source, Mode::Exec, name.clone())
	.map_err(convert::syntax_err)?;
	let attrs = vm.ctx.new_dict();
	attrs
	.set_item("__name__", vm.new_pyobj(name.as_str()), vm)
	.into_js(vm)?;
	if let Some(imports) = imports {
	for entry in convert::object_entries(&imports) {
	let (key, value) = entry?;
	let key: String = Object::from(key).to_string().into();
	attrs
	.set_item(key.as_str(), convert::js_to_py(vm, value), vm)
	.into_js(vm)?;
	}
	}
	vm.run_code_obj(code, Scope::new(None, attrs.clone()))
	.into_js(vm)?;
	let module = vm.new_module(&name, attrs, None);
	let sys_modules = vm.sys_module.get_attr("modules", vm).into_js(vm)?;
	sys_modules.set_item(&name, module.into(), vm).into_js(vm)?;
	Ok(())
	})?
	}

[coolreader18]

Not totally sure if that's the right naming, but it's prior art at least. Though inject_module does feel good I think, inject isn't overloaded with much in Python so that could be a good choice

[coolreader18]

Oh though maybe your proposal wouldn't add it to sys.modules? That feels like the actual "injection" in the operation inject_module performs.

[youknowone]

Thank you! inject_module looks a really good reference. I thought that approach cannot prevent user to put mismatching source code with the same name. But wasm may really need it.

[lattice0]

Yes. Context: we can have python sources with custom functions and call them with arguments, instead of running them. Currently we have to import the python source file. Some applications wouldn't want or even have permission to write to files.