Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support non Python based extension providers #38

Open
Lawouach opened this issue Mar 9, 2018 · 7 comments
Open

Support non Python based extension providers #38

Lawouach opened this issue Mar 9, 2018 · 7 comments

Comments

@Lawouach
Copy link
Contributor

@Lawouach Lawouach commented Mar 9, 2018

This issue's goal is for the community to discuss interest and solutions to support extension providers implemented in languages other than Python.

As a reminder, currently, the toolkit supports three extension providers:

  • http: whereby you declare a URL to call and the toolkit does it for you
  • process: where you provdie the path to a binary which is executed by the toolkit
  • python: where you define a Python function that is imported from a module extension

While Python is considered a good choice for the core and most extensions, we always cared for larger than a single community. @dastergon asked on that subject topic on the community slack and he suggested I should kick the ball with a high-level view of what would need to be done.

Generally speaking, it seems the simplest/easiest integration for calling native code from Python is to export a native library that exports its symbols (much like a C library). When doing that, Python has facilities to call them for you with ctypes.

This is what people seem to generally do:

Alternatives to ctypes are CFFI and cython. The latter is quite interesting because you provide a C-like wrapper on your native extensions and the generated Python code makes it look fairly native. It is popular but requires more work.

There could be two paths:

  1. The core of the toolkit makes it loud and clear it officially supports ctypes/CFFI and you declare it like this:
{
   "type": "probe",
   "name": "my-go-blah",
   "provider": {
        "type": "go",
        "lib_name": "my-go-lib.so",
        "func": "func_name_in_lib",
        "arguments": { ... }
}

This is what is done for Python as well but here that would expect simply a native library.

  1. An extension author wraps entirely the native code inside a Python extension using cython. In that case, the "python" provider is enough and would work as it already does.

I think both are valuable but I wonder what communities would prefer.

@mlafeldt
Copy link

@mlafeldt mlafeldt commented Mar 9, 2018

I recommend using a network RPC interface over brittle extensions. Both Go and Rust produce statically compiled binaries. See https://www.terraform.io/docs/internals/internal-plugins.html for some inspiration.

@Lawouach
Copy link
Contributor Author

@Lawouach Lawouach commented Mar 9, 2018

Although I think a RPC provider would be neat, I'm not sure the network is more reliable than simply calling out an exported function ;)

With that said, it's an interesting approach. My main concern is that some extensions start becoming stateful and more complex than the current interface invites for.

@mlafeldt
Copy link

@mlafeldt mlafeldt commented Mar 9, 2018

We're talking about local network requests only, of course.

Just take my comment as input, nothing more. I'm pretty indifferent to the outcome, TBH.

@Lawouach
Copy link
Contributor Author

@Lawouach Lawouach commented Mar 9, 2018

I think an RPC provider is actually a good idea in general to be honest. That would make the API much more visible to the outside world.

Lawouach added a commit to chaostoolkit/chaostoolkit-documentation that referenced this issue Oct 16, 2018
See chaostoolkit/chaostoolkit-lib#38

Signed-off-by: Sylvain Hellegouarch <sh@defuze.org>
@kv-pradeep-zz
Copy link

@kv-pradeep-zz kv-pradeep-zz commented May 25, 2020

I have recently started using CTK for a product in my organization and I came across this issue during the research. Having the ability in CTK to build custom extensions using python is a great plus. It would be even better if CTK provides ability to have custom extension of a different provider other than python, for example scala in my case. With growing products using CTK for their chaos engineering, I suppose this will soon be a requirement by most. Enabling the provider to run the scripts of different languages by a command that can specified inside the experiment would be fantastic. If the environment has the dependencies of that particular file (java, or scala, or c, etc) the script would run, else it would fail. Having the environment pre-requisites is the reponsibility of the engineers and not of the framework (CTK). Giving an option to run different files by your custom commands would be just fantastic.

@Lawouach
Copy link
Contributor Author

@Lawouach Lawouach commented May 26, 2020

Hey @kv-pradeep great suggestion and thank you for updating the issue.

I wonder, would you be able to sketched what this could look like to you?

@Lawouach
Copy link
Contributor Author

@Lawouach Lawouach commented Jun 17, 2020

As a reference, I wonder if wasm wouldn't be a nice intermediate way of extending the toolkit outside of Python

https://github.com/wasmerio/python-ext-wasm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.