[improve][fn] support WASM functions which can support many languages

[loongs-zhang]

Motivation

The wasm bytecode is designed to be encoded in a size- and load-time-efficient binary format. WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms.

This PR support run WASM functions

Verifying this change

see pulsar-functions/java-examples/src/main/rust-function/README.md and org.apache.pulsar.functions.api.examples.WasmFunction

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[RobertIndie]

This is a new feature for the Pulsar Function. Please create a PIP for this or start a discussion on the dev mailing list first. You could refer to here for how to submit a PIP.

[loongs-zhang]

see #21992 for doc

[lburgazzoli]

Just for info, a pure wasm java runtime is now in development https://github.com/dylibso/chicory

I did use it successfully in some POC:

• https://lburgazzoli.github.io/posts/2024-02-01_apache_kafka_connect_meets_wasm_part_1/
• https://lburgazzoli.github.io/posts/2024-01-14_apache_camel_meets_wasm_part_1/

[loongs-zhang]

Just for info, a pure wasm java runtime is now in development https://github.com/dylibso/chicory

I did use it successfully in some POC:

• https://lburgazzoli.github.io/posts/2024-02-01_apache_kafka_connect_meets_wasm_part_1/
• https://lburgazzoli.github.io/posts/2024-01-14_apache_camel_meets_wasm_part_1/

It looks very good, what about the performance ?

[lburgazzoli]

At this stage I haven't run any performance measurement as both the runtime and the integrations are at at very early stage.

[loongs-zhang]

Personally, I prefer wasmtime-java, there's also no need to write native for bridge, and the low-level wasmtime is mature enough.

[lburgazzoli]

I think the biggest advantage of chicory over wasmtime-java + wasmtime is the fact that the first is a pure meant to be a pure java implementation so it does not require to have any jni/ffi binding but yeah, in therm of maturity at this stage wasmtime is quite more mature.

For example purpose, I'll try to write a small POC for chicory + pulsar to see how it would looks like

[loongs-zhang]

Welcome, the cross platform capability of chicory is very good due to java, as for the wasmtime-java, it's only support linux-x86_64, windows-x86_64, macos-x86_64 and macos-aarch64. But now, almost all of servers are x86_64, so I overlooked this. @lburgazzoli

You just need to pay attention to the code of pulsar-functions/wasm/src/main/java/org/apache/pulsar/functions/wasm/WasmLoader.java.

[lburgazzoli]

Here a very simple POC: https://github.com/lburgazzoli/pulsar-function-wasm

NOTES:

1. I'm by no mean suggesting that chicory is the runtime that should be adopted
2. since I'm involved in the chicory project, I'm doing a series of POC mainly to explore how to improve it so take it only as an experiment :)

[loongs-zhang]

I would like to provide an SPI where submodules can be implemented using wasmtime-java and chicory respectively :)

About pulsar-function-wasm-sdk, do we need implement similar SDKs for other languages? Of course, except for the existing SDK in pulsar functions.

Then when a language is not supported by pulsar functions, we can quickly provide a lightweight implementation of pulsar functions sdk. When there is more time, we can continue to support the complete SDK.

I think this is a good idea, do you have any other suggestions? @lburgazzoli

[lburgazzoli]

I would like to provide an SPI where submodules can be implemented using wasmtime-java and chicory respectively :)

I'm not sure hot that would really be feasible given the low level APIs the various runtime expose, but if you come up with something I can definitively try it out and help with the chicory implementation.

About pulsar-function-wasm-sdk, do we need implement similar SDKs for other languages? Of course, except for the existing SDK in pulsar functions.

Then when a language is not supported by pulsar functions, we can quickly provide a lightweight implementation of pulsar functions sdk. When there is more time, we can continue to support the complete SDK.

I think this is a good idea, do you have any other suggestions? @lburgazzoli

Given the current state of Wasm, an SDK is likely very needed otherwise each function would be required to re-implement the ABI.

[loongs-zhang]

I will start abstracting SPI interfaces as soon as possible after the Chinese New Year.