[Functions] Provide an interface for functions worker service #8560

sijie · 2020-11-13T08:17:04Z

Motivation

Make the pulsar functions worker serve as an interface to allow plugin different functions of worker service implementations.

*Motivation* The script was broken after we upgraded the `netty-tc-native` version to `2.0.33.Final`. This change fixes the script to make sure it is able to run on MacOS.

*Motivation* Use the same gson version as kubernetes client does to fix version conflict issue. ``` ERROR] testAuthProviderWrongInterface(org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactoryTest) Time elapsed: 0.07 s <<< FAILURE! org.testng.TestException: The exception was thrown with the wrong message: expected "Function authentication provider.*.must implement KubernetesFunctionAuthProvider" but got "com.google.gson.JsonParser.parseReader(Ljava/io/Reader;)Lcom/google/gson/JsonElement;" at org.testng.internal.ExpectedExceptionsHolder.wrongException(ExpectedExceptionsHolder.java:70) at org.testng.internal.TestInvoker.considerExceptions(TestInvoker.java:737) at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:635) at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214) at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58) at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822) at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128) at java.util.ArrayList.forEach(ArrayList.java:1257) at org.testng.TestRunner.privateRun(TestRunner.java:764) at org.testng.TestRunner.run(TestRunner.java:585) at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) at org.testng.SuiteRunner.run(SuiteRunner.java:286) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) at org.testng.TestNG.runSuites(TestNG.java:1069) at org.testng.TestNG.run(TestNG.java:1037) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Caused by: java.lang.NoSuchMethodError: com.google.gson.JsonParser.parseReader(Ljava/io/Reader;)Lcom/google/gson/JsonElement; at io.kubernetes.client.util.KubeConfig.runExec(KubeConfig.java:330) at io.kubernetes.client.util.KubeConfig.tokenViaExecCredential(KubeConfig.java:269) at io.kubernetes.client.util.KubeConfig.getAccessToken(KubeConfig.java:231) at io.kubernetes.client.util.credentials.KubeconfigAuthentication.<init>(KubeconfigAuthentication.java:46) at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:276) at io.kubernetes.client.util.ClientBuilder.getClientBuilder(ClientBuilder.java:108) at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:87) at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:79) at io.kubernetes.client.util.Config.defaultClient(Config.java:113) at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.setupClient(KubernetesRuntimeFactory.java:339) at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.initialize(KubernetesRuntimeFactory.java:208) at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactoryTest.createKubernetesRuntimeFactory(KubernetesRuntimeFactoryTest.java:184) at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactoryTest.testAuthProvider(KubernetesRuntimeFactoryTest.java:232) at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactoryTest.testAuthProviderWrongInterface(KubernetesRuntimeFactoryTest.java:243) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ```

…er_api

sijie · 2020-11-18T04:15:42Z

/pulsarbot run-failure-checks

…er_api

…act_worker_api

sijie · 2020-11-20T00:37:19Z

/pulsarbot run-failure-checks

sijie · 2020-12-10T18:54:32Z

/pulsarbot run-failure-checks

wolfstudy

LGTM +1

--- Fixes #10332 *Motivation* Sometimes, the superuser is not only needed the client role but also needs some data in the authentication data to check whether the role is the superuser. The changed interface is introduced from #8560, it's safe to change it since there is no release for that.

) --- Fixes apache#10332 *Motivation* Sometimes, the superuser is not only needed the client role but also needs some data in the authentication data to check whether the role is the superuser. The changed interface is introduced from apache#8560, it's safe to change it since there is no release for that.

--- *Motivation* Port the fix apache#10364 to the branch 2.7. Sometimes, the superuser is not only needed the client role but also needs some data in the authentication data to check whether the role is the superuser. The changed interface is introduced from apache#8560, it's safe to change it since there is no release for that.' *Modifications* Change the isSuperUser method to accept the authenticationData and pass it to the authentication provider. So authenticatoin prodiver can get the data saved in the authenticationDataSource.

…er (#11418) --- *Motivation* Port the fix #10364 to the branch 2.7. Sometimes, the superuser is not only needed the client role but also needs some data in the authentication data to check whether the role is the superuser. The changed interface is introduced from #8560, it's safe to change it since there is no release for that.' *Modifications* Change the isSuperUser method to accept the authenticationData and pass it to the authentication provider. So authenticatoin prodiver can get the data saved in the authenticationDataSource.

) --- Fixes apache#10332 *Motivation* Sometimes, the superuser is not only needed the client role but also needs some data in the authentication data to check whether the role is the superuser. The changed interface is introduced from apache#8560, it's safe to change it since there is no release for that.

…19975) ### Motivation The current function worker interfaces introduced by #8560 are hard to extend. It is better to pass an object that we can add fields to as needed. ### Modifications * Introduce a new `Authentication` class to wrap all of the relevant authentication "data". This class is somewhat unfortunate in that we already have many classes that hold authentication information. However, my goal with this class is to wrap all of the relevant auth info. This class is only currently used in the Function Worker API, but I plan to add it to the rest of the Admin HTTP endpoints. * Deprecate all of the methods that are no longer needed. Add a default implementation to call the new methods that supersede those methods. * Add `proxyRoles` setting to function worker. ### Verifying this change There are tests to cover the changes. ### Does this pull request potentially affect one of the following parts: This PR changes several interfaces in completely backwards compatible ways. It's possible we'll want to improve the abstraction for the `Authentication` class by making it an interface. ### Documentation - [x] `doc` This change has Javadoc updates to document the changes. ### Matching PR in forked repository PR in forked repository: michaeljmarshall#37

…19975) The current function worker interfaces introduced by #8560 are hard to extend. It is better to pass an object that we can add fields to as needed. * Introduce a new `Authentication` class to wrap all of the relevant authentication "data". This class is somewhat unfortunate in that we already have many classes that hold authentication information. However, my goal with this class is to wrap all of the relevant auth info. This class is only currently used in the Function Worker API, but I plan to add it to the rest of the Admin HTTP endpoints. * Deprecate all of the methods that are no longer needed. Add a default implementation to call the new methods that supersede those methods. * Add `proxyRoles` setting to function worker. There are tests to cover the changes. This PR changes several interfaces in completely backwards compatible ways. It's possible we'll want to improve the abstraction for the `Authentication` class by making it an interface. - [x] `doc` This change has Javadoc updates to document the changes. PR in forked repository: michaeljmarshall#37 (cherry picked from commit 55acbe6)

…pache#19975) The current function worker interfaces introduced by apache#8560 are hard to extend. It is better to pass an object that we can add fields to as needed. * Introduce a new `Authentication` class to wrap all of the relevant authentication "data". This class is somewhat unfortunate in that we already have many classes that hold authentication information. However, my goal with this class is to wrap all of the relevant auth info. This class is only currently used in the Function Worker API, but I plan to add it to the rest of the Admin HTTP endpoints. * Deprecate all of the methods that are no longer needed. Add a default implementation to call the new methods that supersede those methods. * Add `proxyRoles` setting to function worker. There are tests to cover the changes. This PR changes several interfaces in completely backwards compatible ways. It's possible we'll want to improve the abstraction for the `Authentication` class by making it an interface. - [x] `doc` This change has Javadoc updates to document the changes. PR in forked repository: #37 (cherry picked from commit 55acbe6) (cherry picked from commit 54c97ad)

…pache#19975) The current function worker interfaces introduced by apache#8560 are hard to extend. It is better to pass an object that we can add fields to as needed. * Introduce a new `Authentication` class to wrap all of the relevant authentication "data". This class is somewhat unfortunate in that we already have many classes that hold authentication information. However, my goal with this class is to wrap all of the relevant auth info. This class is only currently used in the Function Worker API, but I plan to add it to the rest of the Admin HTTP endpoints. * Deprecate all of the methods that are no longer needed. Add a default implementation to call the new methods that supersede those methods. * Add `proxyRoles` setting to function worker. There are tests to cover the changes. This PR changes several interfaces in completely backwards compatible ways. It's possible we'll want to improve the abstraction for the `Authentication` class by making it an interface. - [x] `doc` This change has Javadoc updates to document the changes. PR in forked repository: #37 (cherry picked from commit 55acbe6) (cherry picked from commit 54c97ad) (cherry picked from commit cf40926)

…pache#19975) The current function worker interfaces introduced by apache#8560 are hard to extend. It is better to pass an object that we can add fields to as needed. * Introduce a new `Authentication` class to wrap all of the relevant authentication "data". This class is somewhat unfortunate in that we already have many classes that hold authentication information. However, my goal with this class is to wrap all of the relevant auth info. This class is only currently used in the Function Worker API, but I plan to add it to the rest of the Admin HTTP endpoints. * Deprecate all of the methods that are no longer needed. Add a default implementation to call the new methods that supersede those methods. * Add `proxyRoles` setting to function worker. There are tests to cover the changes. This PR changes several interfaces in completely backwards compatible ways. It's possible we'll want to improve the abstraction for the `Authentication` class by making it an interface. - [x] `doc` This change has Javadoc updates to document the changes. PR in forked repository: michaeljmarshall#37 (cherry picked from commit 55acbe6)

Abstract Function Worker Service

57e5c12

sijie added this to the 2.7.0 milestone Nov 13, 2020

sijie requested review from jerrypeng, jiazhai, codelipenghui and wolfstudy November 13, 2020 08:17

sijie self-assigned this Nov 13, 2020

sijie added 3 commits November 13, 2020 18:52

Provide interfaces for managing functions/sources/sinks

b021071

Adjust the tests

93f0c6a

Merge remote-tracking branch 'apache/master' into abstract_worker_api

9337e77

sijie marked this pull request as ready for review November 17, 2020 20:51

sijie added 4 commits November 17, 2020 16:22

Fix the broken rename-netty-native-libs.sh script

492de68

*Motivation* The script was broken after we upgraded the `netty-tc-native` version to `2.0.33.Final`. This change fixes the script to make sure it is able to run on MacOS.

Merge branch 'update_gson_version' into abstract_worker_api

50c8d86

Merge branch 'fix_rename_netty_native_libs_script' into abstract_work…

04212b9

…er_api

sijie and others added 8 commits November 17, 2020 20:16

Fix the license file

dc4efb5

Fix the license file.

ff24a59

Build multiple os

693a039

Merge branch 'fix_rename_netty_native_libs_script' into abstract_work…

0e2348d

…er_api

Merge branch 'update_gson_version' into abstract_worker_api

5458403

Fix the project style error

e41665a

Merge remote-tracking branch 'apache/master' into abstract_worker_api

be574fe

Merge remote-tracking branch 'zymap/fix-build-style-error' into abstr…

4c8ac93

…act_worker_api

wolfstudy mentioned this pull request Nov 18, 2020

[Feature] Integrated functions worker service streamnative/function-mesh#28

Closed

sijie added 4 commits November 18, 2020 14:02

Fix tests

7d49326

Merge remote-tracking branch 'apache/master' into abstract_worker_api

64ad8b7

Fix integration tests

c5af2db

Merge remote-tracking branch 'apache/master' into abstract_worker_api

1714841

codelipenghui modified the milestones: 2.7.0, 2.8.0 Nov 21, 2020

sijie added 4 commits December 9, 2020 16:18

Merge remote-tracking branch 'apache/master' into abstract_worker_api

3ab9975

Merge remote-tracking branch 'apache/master' into abstract_worker_api

7c8e3fd

Fix NPEs on tests

d9f29ff

Fix NPE in PrometheusMetricsGenerator

8bd7502

Fix NPE in standalone

c78ee78

wolfstudy approved these changes Dec 14, 2020

View reviewed changes

sijie added the area/function label Dec 14, 2020

sijie merged commit 26749a8 into apache:master Dec 14, 2020

This was referenced Jan 14, 2021

[issue 8337][Worker] Move initialize dlog namespace metadata to bin/pulsar #8781

Merged

Fix typo in wokerserviceloader #9169

Merged

zymap mentioned this pull request Apr 25, 2021

Fixes function api can not use authdata to check superuser #10364

Merged

This was referenced May 7, 2021

[functions worker] allow custom configs to plugin worker service #10503

Closed

[#10503] allow user to set custom configs to plugin worker service #10504

Merged

sijie mentioned this pull request May 7, 2021

ISSUE-10503: [functions worker] allow custom configs to plugin worker service streamnative/pulsar-archived#2491

Closed

zymap mentioned this pull request Jul 22, 2021

[Branch-2.7] Fixes function api can not use authdata to check superuser #11418

Merged

1 task

BewareMyPower mentioned this pull request Dec 11, 2021

[Branch-2.7] Make the unit test and integration test work #13243

Merged

michaeljmarshall mentioned this pull request Mar 31, 2023

[refactor][fn] Use AuthorizationServer more in Function Worker API #19975

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Functions] Provide an interface for functions worker service #8560

[Functions] Provide an interface for functions worker service #8560

sijie commented Nov 13, 2020

sijie commented Nov 18, 2020

sijie commented Nov 20, 2020

sijie commented Dec 10, 2020

wolfstudy left a comment

[Functions] Provide an interface for functions worker service #8560

[Functions] Provide an interface for functions worker service #8560

Conversation

sijie commented Nov 13, 2020

sijie commented Nov 18, 2020

sijie commented Nov 20, 2020

sijie commented Dec 10, 2020

wolfstudy left a comment

Choose a reason for hiding this comment