Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possibility to add builtin Pulsar Functions with a shared jar copied inside the function folder #6003

Closed
nicolo-paganin opened this issue Jan 6, 2020 · 6 comments
Closed

Possibility to add builtin Pulsar Functions with a shared jar copied inside the function folder #6003

nicolo-paganin opened this issue Jan 6, 2020 · 6 comments
Labels
lifecycle/stale type/feature The PR added a new feature or issue requested a new feature

Comments

@nicolo-paganin
Copy link

nicolo-paganin commented Jan 6, 2020 •
edited
Loading

Is your feature request related to a problem? Please describe.
In order to create a pulsar function you have to build a fat jar (or nar) and the submit the jar when you create the function. The jar is copied by default inside pulsar/tmp/function_name. Currently it is not possible to load a shared jar containing one or more functions to be used without loading the jar (something similar to what is happening loading the a nar containing pulsar IO Sources/Sinks inside che connectors folder).

This could have the following benefits:

  • Function can be distributed with the installation and users can create functions from builtin jars / nars without having the jar / nar file
  • In case of multiple functions (with different functions names) using the same jar, the latter is not duplicated every time a new function is created

Describe the solution you'd like
The possibility to create a new function loading the relative jar (if the function is not one of the builtin ones) or instantiate a new function using the builtin ones the are copied inside the function folder (like the connectors)
@nicolo-paganin nicolo-paganin added the type/feature The PR added a new feature or issue requested a new feature label Jan 6, 2020
@nicolo-paganin nicolo-paganin changed the title Pulsar Functions with shared jar Possibility to add builtin Pulsar Functions with a shared jar (to be copyed inside the function folder) Jan 6, 2020
@nicolo-paganin nicolo-paganin changed the title Possibility to add builtin Pulsar Functions with a shared jar (to be copyed inside the function folder) Possibility to add builtin Pulsar Functions with a shared jar (to be copyed inside the function folder) Jan 6, 2020
@nicolo-paganin nicolo-paganin changed the title Possibility to add builtin Pulsar Functions with a shared jar (to be copyed inside the function folder) Possibility to add builtin Pulsar Functions with a shared jar copied inside the function folder Jan 6, 2020
@nicolo-paganin
Copy link
Author

I found that this feature is part of PIP 50

@sijie sijie mentioned this issue Jan 7, 2020
Open
@jiazhai
Copy link
Member

jiazhai commented Jan 8, 2020

thanks @nicolo-paganin ,right, this is part of the work in pip-50, which tries to manage the nars in a better way.

@sijie sijie mentioned this issue Jan 11, 2020
Closed
@sijie
Copy link
Member

sijie commented Jan 11, 2020

@itaross I have included this task into the Pulsar release 2.6.0. If you have any features that would like to see in 2.6.0, please comment in #5819. We will prioritize those features.

nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 6, 2020
@nicolopaganin
apache#6003 Possibility to add builtin Pulsar Functions with a shared…
0eec58c 
… jar copied inside the function folder
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 6, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
35fddb8 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 6, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
14be639 
…inside the function folder (apache#6003)
@oncodeit oncodeit mentioned this issue May 6, 2020
Merged
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 6, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
b96197a 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 6, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
71b3819 
…inside the function folder (apache#6003)
@nicolo-paganin
Copy link
Author

@sijie I created a PR for this, could you please look at it?
Thanks

@sijie
Copy link
Member

sijie commented May 7, 2020

@nicolo-paganin that's cool. will review it.

nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 19, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
8879a44 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 20, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
70616e8 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 22, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
7bacc3f 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 22, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
4f0bd52 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 22, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
72ff073 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue May 27, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
63a4008 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue Jun 2, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
589fc20 
…inside the function folder (apache#6003)
nicolo-paganin pushed a commit to oncodeit/pulsar that referenced this issue Jun 4, 2020
@nicolopaganin
Possibility to add builtin Pulsar Functions with a shared jar copied …
2fa6309 
…inside the function folder (apache#6003)
codelipenghui pushed a commit that referenced this issue Jun 5, 2020
@oncodeit
[#6003][pulsar-functions] Possibility to add builtin Functions (#6895)
9b0098a 
Master Issue: #6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
zeo1995 pushed a commit to zeo1995/pulsar that referenced this issue Jun 5, 2020
Merge branch 'website-update' of github.com:zeo1995/pulsar into websi…
128b8d5 
…te-update

* 'website-update' of github.com:zeo1995/pulsar: (432 commits)
  Fixed ordering issue in KeyShared dispatcher when adding consumer (apache#7106)
  Fix Duplicated messages are sent to dead letter topic apache#6960 (apache#7021)
  [Issue 2793][Doc]--Update the TLS hostname verification for CPP and Python clients (apache#7162)
  [Doc]--set netty mex frame size (apache#7174)
  [Doc] Update for the maximum message size (apache#7171)
  Fixed KeyShared consumers getting stuck on delivery (apache#7105)
  [apache#6003][pulsar-functions] Possibility to add builtin Functions (apache#6895)
  [Issue 6921][pulsar-broker-common] Replaced "Paths.get(...).getParent()", because it's system dependent and uses '\' as path separator on Windows (apache#6992)
  Improve broker unit test CI (apache#7173)
  Fix typo in exception message (apache#7027)
  Support KeyValue Schema Use Null Key And Null Value (apache#7139)
  [Doc]--Update documents for support consumer priority level in failover mode (apache#7136)
  Add schema config to cpp and cgo docs. (apache#7137)
  [Doc]--Update for the maximum message size (apache#7160)
  [C++] Expose ZSTD and Snappy compression to C API (apache#7014)
  [pulsar-proxy] add proxyLogLevel into config file (apache#6948)
  Add multi-hosts example for bookkeeperMetadataServiceUri (apache#6998)
  support for termination of partitioned topic (apache#6126)
  Use pure-java Air-Compressor instead of JNI based libraries (apache#5390)
  [Issues 5709]remove the namespace checking (apache#5716)
  ...

# Conflicts:
#	site2/website/scripts/split-swagger-by-version.js
merlimat pushed a commit to merlimat/pulsar that referenced this issue Jun 30, 2020
@oncodeit
[apache#6003][pulsar-functions] Possibility to add builtin Functions (a…
802963e 
…pache#6895)

Master Issue: apache#6003

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors).

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
cdbartholomew pushed a commit to kafkaesque-io/pulsar that referenced this issue Jul 24, 2020
@oncodeit @cdbartholomew
[apache#6003][pulsar-functions] Possibility to add builtin Functions (a…
e978244 
…pache#6895)

Master Issue: apache#6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@oncodeit @huangdx0726
[apache#6003][pulsar-functions] Possibility to add builtin Functions (a…
4ab3e68 
…pache#6895)

Master Issue: apache#6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.
@tisonkun
Copy link
Member

tisonkun commented Dec 9, 2022

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.

@tisonkun tisonkun closed this as not planned Won't fix, can't repro, duplicate, stale Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/stale type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@sijie @jiazhai @codelipenghui @nicolo-paganin @tisonkun