Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function runtime pluggable #5463

Merged
merged 5 commits into from Nov 2, 2019

Conversation

@jerrypeng
Copy link
Contributor

jerrypeng commented Oct 25, 2019

Master Issue: #4176

Motivation

Currently, Pulsar Function runtime does not expose a clean interface to make it pluggable.

Modifications

Make Pulsar Function Runtime Factory pluggable to make specifying Function Runtimes more flexible and allow the development of future runtimes to be more smooth since a complete interface is extracted

Also added additional tests, clean up of code, and BC tests

@jerrypeng jerrypeng added this to the 2.5.0 milestone Oct 25, 2019
@jerrypeng jerrypeng requested review from merlimat, sijie, srkukarni and aahmed-se Oct 25, 2019
@jerrypeng jerrypeng self-assigned this Oct 25, 2019
@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Oct 25, 2019

rerun integration tests

1 similar comment
@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Oct 25, 2019

rerun integration tests

@sijie sijie requested a review from wolfstudy Oct 26, 2019
@sijie

This comment has been minimized.

Copy link
Contributor

sijie commented Oct 26, 2019

@wolfstudy please help review this function runtime interface

…tion_runtime_pluggable
@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Oct 30, 2019

@sijie @wolfstudy @merlimat please review

Copy link
Contributor

merlimat left a comment

👍

Copy link
Contributor

sijie left a comment

@jerrypeng overall looks good to me. I have one concern about the configuration settings. I would like to see a smooth upgrade for the users.

@@ -328,162 +332,20 @@ public boolean getTlsEnabled() {
}


@Data

This comment has been minimized.

Copy link
@sijie

sijie Oct 31, 2019

Contributor

Removing configuration is usually treated as a breaking change. Can we keep the old configuration settings for one release but deprecate them? Then after a few more releases, we can remove the settings. Otherwise, it is painful for people to upgrade a cluster.

This comment has been minimized.

Copy link
@jerrypeng

jerrypeng Oct 31, 2019

Author Contributor

This comment has been minimized.

Copy link
@jerrypeng
…tion_runtime_pluggable
@jerrypeng jerrypeng requested a review from sijie Oct 31, 2019
@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Oct 31, 2019

rerun java8 tests

@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Oct 31, 2019

rerun integration tests

@aahmed-se

This comment has been minimized.

Copy link
Contributor

aahmed-se commented Nov 1, 2019

@jerrypeng

org/apache/pulsar/client/impl/ReaderBuilderImpl
Stacktrace
java.lang.NoClassDefFoundError: org/apache/pulsar/client/impl/ReaderBuilderImpl
	at org.apache.pulsar.client.impl.PulsarClientImpl.newReader(PulsarClientImpl.java:192)
	at org.apache.pulsar.tests.integration.functions.PulsarFunctionsTest.testWindowFunction(PulsarFunctionsTest.java:1019)
	at org.apache.pulsar.tests.integration.functions.PulsarFunctionsTest.testSlidingCountWindowTest(PulsarFunctionsTest.java:1084)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Nov 1, 2019

rerun integration tests

@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Nov 1, 2019

@sijie please take a look again. I have replied to your comments.

@sijie
sijie approved these changes Nov 2, 2019
Copy link
Contributor

sijie left a comment

@jerrypeng thank you for your clarification.

One more thought is moving the plugin to a NAR package in the future. This would make Pulsar more extensible without worrying about dependencies conflicts if people would like to develop a new runtime for Pulsar.

@aahmed-se aahmed-se merged commit 9728d54 into apache:master Nov 2, 2019
3 checks passed
3 checks passed
Jenkins: C++ / Python Tests SUCCESS
Details
Jenkins: Integration Tests SUCCESS
Details
Jenkins: Java 8 - Unit Tests SUCCESS
Details
@jerrypeng

This comment has been minimized.

Copy link
Contributor Author

jerrypeng commented Nov 2, 2019

@sijie that is something we should definitely think about. Though I would suggest we have discussion on how to handle plugins in general in Pulsar. Perhaps we should implement or standardize on some sort of plugin loading framework.

@sijie

This comment has been minimized.

Copy link
Contributor

sijie commented Nov 2, 2019

@jerrypeng agree. we have used NAR for functions, connectors, offloaders and protocol handlers. I think we can continue that route.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.