Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix error OutOfMemoryError while using KeyValue<GenericRecord, GenericRecord> #9981

Merged
merged 1 commit into from
Mar 22, 2021

Conversation

eolivelli
Copy link
Contributor

Motivation

Currently you cannot consume a topic with schema KeyValue<GenericRecord, GenericRecord> due to a problem in HttpLookupService.
The HttpLookupService download the Schema in JSON format but the KeyValue schema is expected to be encoded in binary form.

The error you see in current pulsar master is:

org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1027)
	at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:102)
	at org.apache.pulsar.broker.service.KeyValueTest.keyValueAutoConsumeTest(KeyValueTest.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:100)
	... 28 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.pulsar.common.schema.KeyValue.decode(KeyValue.java:135)
	at org.apache.pulsar.client.impl.schema.KeyValueSchemaInfo.decodeKeyValueSchemaInfo(KeyValueSchemaInfo.java:201)
	at org.apache.pulsar.client.impl.schema.KeyValueSchema.configureSchemaInfo(KeyValueSchema.java:196)
	at org.apache.pulsar.client.impl.PulsarClientImpl.lambda$preProcessSchemaBeforeSubscribe$26(PulsarClientImpl.java:875)
	at org.apache.pulsar.client.impl.PulsarClientImpl$$Lambda$766/571497590.apply(Unknown Source)

Modifications

Use existing utility functions to convert the JSON representation of the KeyValue schema to the expected format.

Verifying this change

This change added tests

Fix a problem in HttpLookupService#getSchema
@eolivelli eolivelli changed the title Allow to use KeyValue<GenericRecord, GenericRecord> Fix error OutOfMemoryError while using KeyValue<GenericRecord, GenericRecord> Mar 21, 2021
@eolivelli
Copy link
Contributor Author

@sijie thanks for your review

@eolivelli eolivelli merged commit 5a4b441 into apache:master Mar 22, 2021
@eolivelli eolivelli deleted the fix/keyvalue-record-client branch March 22, 2021 10:06
@codelipenghui codelipenghui added the cherry-picked/branch-2.7 Archived: 2.7 is end of life label Mar 24, 2021
codelipenghui pushed a commit that referenced this pull request Mar 24, 2021
Fix a problem in HttpLookupService#getSchema

(cherry picked from commit 5a4b441)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-picked/branch-2.7 Archived: 2.7 is end of life release/2.7.2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants