Plugin Development Kit ( PDK )

[darach]

Describe the problem you are trying to solve

The plugin development kit ( PDK ) is an interface that allows developers to develop plugins to tremor with a minimal set of dependencies rather than having to build the full tremor runtime with its full set of dependencies.

Describe the solution you'd like

Separating plugins from the runtime has many advantages:

• Faster build and test for the tremor runtime project
• Easier extensibility as plugins ( dynamic shared object libraries ) can be built and deployed without a release cycle
• Fewer dependencies in the tremor runtime project
• Each plugin can be built, tested, and packaged separately
• Standardises lifecycle of the pluggable artifacts
• Encourages standard internal interfaces for tremor micro-architecture where there is no standard internal API or interface today ( design by contract )
• Would allow runtime upgrades of pluggable artifacts without changes to the runtime ( where compatible )

There is an existing RFC for tremor that covers the requirements in depth:

https://rfcs.tremor.rs/0006-plugin-development-kit/

Suggested initial target:

• Pipeline operators
• Codecs
• Preprocessors
• Postprocessors
• Modules/Functions

Enhancements to initial target:

• Implement connectors RFC ( pre-requirement for connector plugins )
• Contribute to and finalize Connectors and Streams tremor-rfcs#32
• Add source, sink, and peering connectors to pluggable artefacts
• Add a PDK TCK ( test compatibility kit ) that asserts plugin invariants and provides testing mechanisms for plugin developers
• Consider plugin documentation generation and another tooling for better developer convenience and usability
• Make trickle sub-graphs a first-class modular and pluggable artefact

Notes

RFC https://rfcs.tremor.rs/0006-plugin-development-kit/

[Raman-Kumar]

hi, I have just learned the kubernates and studying the source code.
Can I contribute to this PDK or I need more experience.

[marioortizmanero]

As the maintainers of Tremor already know, I was preparing myself to work on this project during the summer via the Google Summer of Code. My proposal wasn't accepted in the end so I won't be able to continue, but the research I made is probably worth taking a look at for whoever attempts this next. I'm posting it here so that it's not lost in the Discord server:

• My proposal, with a brief introduction and some research for FFI: https://nullderef.com/blog/gsoc-proposal/
• An update with research on different technologies that could be used for the PDK: https://nullderef.com/blog/plugin-tech/

Cheers!

[Licenser]

you are a legend mate!

[marioortizmanero]

Surprise update! Please read my new blog post for more progress on this issue: https://nullderef.com/blog/plugin-start/

[marioortizmanero]

New update on dynamic loading: https://nullderef.com/blog/plugin-dynload/

Next up... abi_stable :)

[darach]

I love update day 🤘

[marioortizmanero]

@darach the RFC link has been updated to this, please change it: https://www.tremor.rs/rfc/accepted/plugin-development-kit

[marioortizmanero]

Tracking comment for Connectors PDK

Pull Requests:

• PDK support #1434

Currently tracking my progress here: https://github.com/marioortizmanero/tremor-runtime/projects/1

[marioortizmanero]

Forgot to link the last update!

"Plugins in Rust: Reducing the Pain with Dependencies" - https://nullderef.com/blog/plugin-abi-stable/

The next one should be out in February :)

[marioortizmanero]

New article after the implementation of the first version!

"Plugins in Rust: Getting our Hands Dirty" - https://nullderef.com/blog/plugin-impl/

[marioortizmanero]

Here's the final article of the series, concluding my work: "Plugins in Rust: Wrapping Up" - https://nullderef.com/blog/plugin-end/.

It's been a pleasure to work with Tremor, really. I will be around for any doubts anyone may have. And I really look forward to seeing how the plugin system continues to evolve :)