Extism 1.0 Roadmap - Community Review

[bhelx]

Issues

• refactor!: replace Python CLI with one written in Go cli#25
• https://github.com/extism/docs/issues/99
• https://github.com/extism/docs/issues/101
• https://github.com/extism/docs/issues/100
• https://github.com/extism/docs/issues/102
• https://github.com/extism/docs/issues/103
• Provide users with a Github Action to use the Extism CLI #489
• Testing: Resources/organization #487
• Extism: Enhanced Testing #488
• Bundling libextism #486
• Add Observe SDK support to the Extism runtime #470
• Benchmarking #456
• Explore runtime linking of JS provider js-pdk#19
• Add Observe SDK support go-sdk#21
• Add Observe SDK support js-sdk#3
• Write a new README and getting started guide js-sdk#4
• https://github.com/extism/cli2/issues/1
• https://github.com/extism/cli2/issues/2
• Fat Gem Releases ruby-sdk#6
• Add rubygems release workflow ruby-sdk#7
• Re-write to use Wasmex elixir-sdk#3
• Host Function Support elixir-sdk#2
• README elixir-sdk#4
• Python Reference Documentation python-sdk#2
• Bundle libextism with jar java-sdk#3
• Render and host Javadoc java-sdk#2
• Automate Release with GH workflow java-sdk#1
• Write a new README and getting started guide java-sdk#5
• Write a new README and getting started guide dotnet-sdk#11
• JS PDK Improvements js-pdk#21
• Support Host Functions js-pdk#20
• Explore runtime linking of JS provider js-pdk#19
• Proof-of-Concept Python PDK: Which runtime / path to go down? python-pdk#2
• .NET PDK dotnet-pdk#5
• clarify namespace / module in host function docs #490

High-level Goals

• Easier to install: the shared library should be bundled with the SDK if needed
• More idiomatic SDKs with improved developer experience for each language
  • Build up ecosystems around particular SDKs
  • New Wazero based Go SDK: https://github.com/extism/go-sdk/
• Easier, more flexible releases
  • One single JavaScript library for Web, Node, & Deno
  • All will use the native, included runtime and not wasmtime
• Improved documentation that stays synced with the current code
• Rebrand Extism as a “WebAssembly Framework”
  • We will still focus on making a good plug-in system, but Extism can be used as a friendly abstraction over wasm and we want to emphasize that
  • Documentation updates

Code Organization

• Move Host SDKs to their own repos
  • We will also take this opportunity to decouple releases and changelogs

SDK and PDK Docs

• Move docs from static/inaccessible places to the actual codebases themselves
• Remove the [“integrating into your application”](https://extism.org/docs/integrate-into-your-codebase/dotnet-host-sdk) docs and the [”write a plugin”](https://extism.org/docs/write-a-plugin/js-pdk) docs. Instead, link to either the README of the SDK or the PDK, or the hosted documentation.
  • e.g., if the ecosystem supports it, we should have this documentation in the actual code itself and have the readme linked there. [Elixir is a good example.](https://hexdocs.pm/extism/readme.html)
  • If the ecosystem doesn’t have a standard way to do this or it’s cumbersome, then the examples and docs should be written in the README and in the code.
• All docs should provide a quickstart with a copy-pasteable example of getting things running.
  • Should be simple, flawless, and work every time.
• Deeper dive sections about variables, configs, http, Host Functions, etc.
• Concise example showing how to get strings in and out of a wasm plugin.

Main Docs

• The main docs should be scaled back to just be about the high-level concerns (e.g. concepts).
• Provide use cases for Extism/WebAssembly w/ demos of each
  • Plug-in System, Serverless / FaaS, Web Applications, Webhook replacement
• Enable the community to make PRs into the Extism Docs
• Runnable (in browser) example on the front page (or early in the process)
• Quickstart guide.
  • Show the simplest way to use a Host SDK
  • Show the simplest way to write a plugin (or download one)
  • Link to the actual repos / docs for the SDKs for deeper learning. The goal is to show the flexibility of choosing a host and guest language as well as get users to the “success” point as quickly as possible.
  • Click a tab to switch languages.
• Errors / FAQ Page
  • Mostly a traditional FAQ page but would contain common errors as well
  • Easily searchable
  • Link to sections of the page with a header from exceptions. For example, the common WASI error message could link to: https://extism.org/faq/#err-42-wasi-functions-not-present

Testing

• Provide users with a Github Action to use the Extism CLI

Multi-Runtime SDKs

• Go will use Wazero
  • Keep the current Go library with graceful degradation until sunset
  • https://github.com/extism/go-sdk/
• JS libraries should use native Wasm
  • We should have a universal JS library that works for browser, node, bun, deno (only Node/Deno currently supported)
  • Bonus: Support async host functions
• Convert Python SDK to use maturin
  • see https://github.com/PyO3/maturin
• See if ruby would also benefit from tighter native integration

Runtime Cleanup

• Remove Context type
  • Make Plugin thread safe
  • Unify extism and extism-runtime crates into a single crate
• extism_manifest::Manifest and extism::Manifest shouldn’t be distinct types
• Rework (or remove) caching for plugins loaded via HTTP
• Redesign logging system:
  • cleanup current setup, make it more intuitive
  • implement logging as a configurable host function in each SDK
  • observe-sdk

CLI Improvements

• Re-write CLI in Go or Rust with the goal of easier installation and use
• Templates for creating a plugin in Rust/Go/JS etc.
• Profiling/Observability
  • CLI host could have a stdout adapter and some cmdline flags could enable tracing etc

JS PDK

• Add polyfills
• Support async and pending tasks
• Support ESM
• Upgrade Quickjs dependencies
• Support Host functions

Python PDK

• Build and release one!

.NET PDK

• Beta for the dotnet PDK

Host Function Support

• Add support to Elixir SDK
• Add support to Ruby

Java SDK

• Look at bundling the native library in the jar.

Elixir SDK

• Have plugin.call support bytes instead of string

Other

• Support an official way to pass multiple parameters to a plugin
  • JSON doesn’t work for binary data, Protobuf is heavy-weight

[zshipko]

Here's a PR I'm working on to remove the Context type: #421 - one side effect of this change is that there no longer needs to be a distinction between the extism-runtime and extism crates.

[ubiquitous-dev]

+1 to the rebranding of Extism to be a webassembly framework. I think focusing on the plug-in aspect is one specific use case, but it's helpful for so much more.

[ubiquitous-dev]

async and the event loop support through the JS PDK would be so awesome! Also supporting Javy's new dynamic-link mode, which lets you build a wasm module that depends on a runtime module also being linked into the runtime that actually has quickjs in it, so your wasm module is only like 5 kb and the rest of the quickjs runtime is static across all instances. It enables a bunch of more sophisticated runtime-specific use-cases. A similar concept would be amazing for the .NET PDK and possibly other languages as well, where it makes sense to bundle the runtime separately and just package up the bytecode.

[bhelx]

Note: Ruby needs host function support

[bhelx]

I believe there is a way to package the java jar with a native binary so i'm gonna explore that on the roadmap too.