Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #8268][Pulsar Function] k8s runtime with go functions support #8352

Merged

Conversation

freeznet
Copy link
Contributor

Fixes #8268

Motivation

currently, go function cannot work with k8s runtime other than java and python, this PR is intended to add go function support with k8s runtime.

Modifications

removed UnsupportedOperationException with GO function, fixed go function executable permissions, and fix arguments passed to go function with correct format.

this PR is making k8s runtime with go function workable, but may not cover all k8s scenarios, so comments are welcome, also need some help with tests, havnt add any tests yet.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

tests WIP

@freeznet freeznet changed the title [Iuuse #8268][Pulsar Function] k8s runtime with go functions support [Issue #8268][Pulsar Function] k8s runtime with go functions support Oct 23, 2020
@freeznet freeznet force-pushed the k8s-runtime-with-go-functions-support branch from 15b1220 to cad7e66 Compare October 27, 2020 01:11
@wolfstudy wolfstudy added this to the 2.7.0 milestone Oct 27, 2020
@wolfstudy
Copy link
Member

@freeznet Can you add an integration test for this?

@freeznet
Copy link
Contributor Author

@freeznet Can you add an integration test for this?

sure, wip

@freeznet freeznet force-pushed the k8s-runtime-with-go-functions-support branch from f873c8e to e6599bf Compare October 28, 2020 09:36
Copy link
Member

@wolfstudy wolfstudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@wolfstudy
Copy link
Member

/pulsarbot run-failure-checks

@freeznet freeznet force-pushed the k8s-runtime-with-go-functions-support branch from e6599bf to e87e125 Compare October 29, 2020 01:28
@freeznet
Copy link
Contributor Author

/pulsarbot run-failure-checks

Copy link
Member

@sijie sijie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change overall looks good. But I think we need to add an integration test to ensure this work.

@@ -52,7 +52,7 @@ public String getSecretsProviderClassName(Function.FunctionDetails functionDetai
case PYTHON:
return "secretsprovider.EnvironmentBasedSecretsProvider";
case GO:
throw new UnsupportedOperationException();
return "";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to create an issue and add a comment here to link to the issue. Otherwise, we will eventually forget to add secrets provider.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i have created a new issue #8425

@freeznet
Copy link
Contributor Author

freeznet commented Nov 2, 2020

The change overall looks good. But I think we need to add an integration test to ensure this work.

@sijie thanks for your review. currently the pulsar repo do not have any integration test implemented with k8s runtime, maybe we should create a new issue to address the lack of integration test for k8s runtime. once the integration tests with k8s runtime implemented, then we can manage the integration tests with k8s runtime & golang functions.

@freeznet
Copy link
Contributor Author

freeznet commented Nov 3, 2020

/pulsarbot run-failure-checks

1 similar comment
@codelipenghui
Copy link
Contributor

/pulsarbot run-failure-checks

@freeznet freeznet force-pushed the k8s-runtime-with-go-functions-support branch from 34ded0e to bbbf01e Compare November 5, 2020 03:08
@codelipenghui
Copy link
Contributor

/pulsarbot run-failure-checks

@freeznet
Copy link
Contributor Author

freeznet commented Nov 8, 2020

/pulsarbot run-failure-checks

3 similar comments
@wolfstudy
Copy link
Member

/pulsarbot run-failure-checks

@freeznet
Copy link
Contributor Author

/pulsarbot run-failure-checks

@wolfstudy
Copy link
Member

/pulsarbot run-failure-checks

@wolfstudy
Copy link
Member

Error:  Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 153.034 s <<< FAILURE! - in org.apache.pulsar.client.api.ClientDeduplicationFailureTest
Error:  testClientDeduplicationCorrectnessWithFailure(org.apache.pulsar.client.api.ClientDeduplicationFailureTest)  Time elapsed: 20.957 s  <<< FAILURE!
java.io.IOException: Error open RocksDB database
	at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:184)
	at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:85)
	at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:60)
	at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:69)
	at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>

@codelipenghui codelipenghui merged commit bd475c2 into apache:master Nov 17, 2020
flowchartsman pushed a commit to flowchartsman/pulsar that referenced this pull request Nov 17, 2020
…pport (apache#8352)

Fixes apache#8268

### Motivation

currently, go function cannot work with k8s runtime other than java and python, this PR is intended to add go function support with k8s runtime.

### Modifications

removed `UnsupportedOperationException` with GO function, fixed go function executable permissions, and fix arguments passed to go function with correct format.

this PR is making k8s runtime with go function workable, but may not cover all k8s scenarios, so comments are welcome, also need some help with tests, havnt add any tests yet.
@codelipenghui
Copy link
Contributor

depends by #8780

codelipenghui pushed a commit that referenced this pull request Dec 7, 2020
…8352)

Fixes #8268

### Motivation

currently, go function cannot work with k8s runtime other than java and python, this PR is intended to add go function support with k8s runtime.

### Modifications

removed `UnsupportedOperationException` with GO function, fixed go function executable permissions, and fix arguments passed to go function with correct format.

this PR is making k8s runtime with go function workable, but may not cover all k8s scenarios, so comments are welcome, also need some help with tests, havnt add any tests yet.

(cherry picked from commit bd475c2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Pulsar Function] Support k8s runtime for Go Functions
4 participants