Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#6003][pulsar-functions] Possibility to add builtin Functions #6895

Merged
merged 2 commits into from
Jun 5, 2020

Conversation

oncodeit
Copy link
Contributor

@oncodeit oncodeit commented May 6, 2020

Master Issue: #6003

Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors).

The builtin function must include a pulsar-io.yml file with the following content

name: <function-name>
description: <function-desciption>
functionClass: <function-class>

e.g.

name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala

val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

Function folder to be specified in the conf/functions_worker.yml conf file

e.g.
functionsDirectory: ./functions

Function package must be in *.nar format like for source/sink connectors

Modifications

I modified the pulsar-function-utils, pulsar-functions-worker and pulsar-common modules on the basis of the built in connectors implementation.
Also Function.proto has been modified in order to include the builtin property

What this MR does not include

  • modification of pulsar-admin to fetch the available buildin functions
  • the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? must be documented in the docs

@nicolo-paganin nicolo-paganin force-pushed the master branch 2 times, most recently from b96197a to 71b3819 Compare May 6, 2020 14:40
@sijie sijie requested review from jerrypeng, srkukarni, codelipenghui and jiazhai and removed request for jerrypeng May 7, 2020 05:55
@sijie sijie added this to the 2.6.0 milestone May 7, 2020
@sijie sijie added area/function type/feature The PR added a new feature or issue requested a new feature labels May 7, 2020
Copy link
Member

@sijie sijie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks great!

@sijie
Copy link
Member

sijie commented May 18, 2020

@oncodeit can you rebase this pull request to latest master?

@nicolo-paganin
Copy link

@sijie rebase done

@codelipenghui
Copy link
Contributor

@nicolo-paganin Could you please take a look at the failed CI tests?

@nicolo-paganin
Copy link

@codelipenghui I think I need some help, there are 5 CI tests that I really don't know.. something that I never modified..

@nicolo-paganin
Copy link

@codelipenghui I rebased and some CI tests that were failing now are passing, but there are two that are not passing, it is nothing related with this PR.
It seems something wrong with the CI itself and I cannot restart the CI. Do you have any advice?

@nicolo-paganin
Copy link

one of the error is this one https://github.com/streamnative/pulsar/issues/662 already report by @sijie

@nicolo-paganin
Copy link

@codelipenghui any ideas on how to fix the CI?

@codelipenghui
Copy link
Contributor

@nicolo-paganin You can merge the master branch into your branch, there are some conflicts fixed on the master branch.

@nicolo-paganin
Copy link

@codelipenghui ok thanks, the pipeline finally passed

@codelipenghui codelipenghui merged commit 9b0098a into apache:master Jun 5, 2020
zeo1995 pushed a commit to zeo1995/pulsar that referenced this pull request Jun 5, 2020
…te-update

* 'website-update' of github.com:zeo1995/pulsar: (432 commits)
  Fixed ordering issue in KeyShared dispatcher when adding consumer (apache#7106)
  Fix Duplicated messages are sent to dead letter topic apache#6960 (apache#7021)
  [Issue 2793][Doc]--Update the TLS hostname verification for CPP and Python clients (apache#7162)
  [Doc]--set netty mex frame size (apache#7174)
  [Doc] Update for the maximum message size (apache#7171)
  Fixed KeyShared consumers getting stuck on delivery (apache#7105)
  [apache#6003][pulsar-functions] Possibility to add builtin Functions (apache#6895)
  [Issue 6921][pulsar-broker-common] Replaced "Paths.get(...).getParent()", because it's system dependent and uses '\' as path separator on Windows (apache#6992)
  Improve broker unit test CI (apache#7173)
  Fix typo in exception message (apache#7027)
  Support KeyValue Schema Use Null Key And Null Value (apache#7139)
  [Doc]--Update documents for support consumer priority level in failover mode (apache#7136)
  Add schema config to cpp and cgo docs. (apache#7137)
  [Doc]--Update for the maximum message size (apache#7160)
  [C++] Expose ZSTD and Snappy compression to C API (apache#7014)
  [pulsar-proxy] add proxyLogLevel into config file (apache#6948)
  Add multi-hosts example for bookkeeperMetadataServiceUri (apache#6998)
  support for termination of partitioned topic (apache#6126)
  Use pure-java Air-Compressor instead of JNI based libraries (apache#5390)
  [Issues 5709]remove the namespace checking (apache#5716)
  ...

# Conflicts:
#	site2/website/scripts/split-swagger-by-version.js
merlimat pushed a commit to merlimat/pulsar that referenced this pull request Jun 30, 2020
…pache#6895)

Master Issue: apache#6003

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors).

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
cdbartholomew pushed a commit to kafkaesque-io/pulsar that referenced this pull request Jul 24, 2020
…pache#6895)

Master Issue: apache#6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
…pache#6895)

Master Issue: apache#6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/function type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants