Plugin Support for MetaCall CLI

[akshgpt7]

The idea is to fire up a Node REPL when the CLI is initiated.
The REPL will parse commands and arguments and match them with a list of available commands (plugins), that will be stored in individual files (possibly with the same name as the function they contain).

[akshgpt7]

A few node packages to consider using for parsing arguments:

• https://github.com/sindresorhus/meow
• https://github.com/yargs/yargs
• https://github.com/tj/commander.js
• https://github.com/oclif/oclif

[viferga]

This is an issue related to: #30

This is a bigger issue which is the following: https://github.com/metacall/gsoc-2021#cli-plugin-support

The REPL and parsing of arguments can be implemented in any language, it does not matter. The only problem is that NPM packages are specially difficult to package in Guix (specially if they are not self contained and they have a long dependency tree), which is what we are using to generate the distributable: https://github.com/metacall/distributable

Ceda (https://github.com/Ceda-EI) proposed to do it in Python with argparse (from stdlib) and https://guix.gnu.org/packages/python-prompt-toolkit-1.0.15/ which is already packaged in Guix. This can be a better option if the idea is to use something different to this https://nodejs.org/api/repl.html and / or this https://nodejs.org/api/readline.html in NodeJS.

[viferga]

Related to the implementation, an option can be to refactor the current implementation by probably using ( https://en.cppreference.com/w/cpp/filesystem ) to list the folder and then load all the scripts. If the REPL loop itself is implemented in another language different from C++, then we will need to call back to C in order to provide the command information.

Another option can be to fully implement it in another language and bootstrap it at the beginning. The load of the files can be also done in the other language, but I think it may be interesting to keep part of the C/C++ project so it is possible to have it as a launcher and we do not need to refactor the distributable or any other package depending on metacallcli.

Also as we do not have support for LLVM or Clang Loaders yet, if anybody wants to implement sandboxing ( https://github.com/metacall/gsoc-2021#cli-security-through-seccomp-sandboxing ) it will need the C/C++ project to implement this. So even if we move everything outside, we will need to keep a C/C++ skeleton. It's up to you how much part of the functionality you want to move outside or to the plugins.

[akshgpt7]

Your concerns towards the npm packages are totally valid. I think if we have the option of using the already packaged python-prompt-toolkit, it's a great idea, and we should go with that instead. argparse and the prompt-toolkit would surely work well together.

As for the implementation, I was thinking more of bootstrapping it from the C++ run() function:

core/source/cli/metacallcli/source/application.cpp

Line 581 in 8579039

void application::run()

(idk, it just seems more intuitive to me than making calls every time to the C++ part to verify a command from the list of files)
This would also mean porting some existing commands to Python (or we can call their C++ implementation through metacall, have to flesh it out more).

I think this will refactor most of the CLI code to Python. As for your concern of implementing sandboxing, I saw that libseccomp has bindings for Python (https://github.com/seccomp/libseccomp/blob/main/src/python/seccomp.pyx), so that shouldn't end up being a blocker.

Does that sound right to you, or you insist on keeping more of the C/C++ part intact?

[viferga]

In that case you can remove more parts of the C++ CLI, but I think it should have still a minimal part written in C/C++ in order to bootstrap the Python client, no? Like 3 lines doing a metacall_load_from_file to the REPL (blocking) and a metacall_destroy, no?

[akshgpt7]

That's correct. It will still need to have a starting and closing point at the C++ part where it will be initiated with the metacall API.
Also, as I mentioned, maybe we can also use some of the existing commands written in C++ directly.

[rxbryan]

created this flowchart for the cli-refactor project

[viferga]

This is half implemented, leaving some additional reference here: #351 (comment)