Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Issue 7402] Fix GCS offload operation error #7400

Merged
merged 1 commit into from
Jul 1, 2020
Merged

[Issue 7402] Fix GCS offload operation error #7400

merged 1 commit into from
Jul 1, 2020

Conversation

gaoran10
Copy link
Contributor

@gaoran10 gaoran10 commented Jun 30, 2020 •
edited
Loading

Fix #7402

Motivation

The GCS offload operation is not working well.

Error Log

SEVERE: Error parsing input: null
java.lang.NullPointerException
	at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:319)
	at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:287)
	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$ParameterReader.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:275)
	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:187)
	at com.google.gson.Gson.fromJson(Gson.java:888)
	at com.google.gson.Gson.fromJson(Gson.java:853)
	at org.jclouds.json.internal.GsonWrapper.fromJson(GsonWrapper.java:55)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:82)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:76)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:61)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:41)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
	at com.sun.proxy.$Proxy82.composeObjects(Unknown Source)
	at org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.completeMultipartUpload(GoogleCloudStorageBlobStore.java:405)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy80.completeMultipartUpload(Unknown Source)
	at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$offload$4(BlobStoreManagedLedgerOffloader.java:463)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)




java.util.concurrent.ExecutionException: org.jclouds.http.HttpResponseException: Error parsing input: null
{statusCode=200, message=OK, headers={Alt-Svc=[h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"], Server=[UploadServer], ETag=[CJqtx5TtqeoCEAE=], X-GUploader-UploadID=[AAANsUmvovukkc5rtoafAhiNOnn2wTXoK6qub8xX1a_S2I8wN_3RfqhyswUdhEKTkMMo67j0HfxVf6e70Nzs-YPjrrQ], Vary=[X-Origin, Origin], Pragma=[no-cache], Date=[Tue, 30 Jun 2020 15:28:08 GMT]}, payload=[content=true, contentMetadata=[cacheControl=no-cache, no-store, max-age=0, must-revalidate, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=951, contentMD5=null, contentType=application/json; charset=UTF-8, expires=Mon Jan 01 08:00:00 CST 1990], written=false, isSensitive=false]}

	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest.testHappyCase(BlobStoreManagedLedgerOffloaderTest.java:141)
	at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest_$$_jvstb34_0._d16testHappyCase(BlobStoreManagedLedgerOffloaderTest_$$_jvstb34_0.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.powermock.modules.testng.internal.PowerMockTestNGMethodHandler.invoke(PowerMockTestNGMethodHandler.java:50)
	at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest_$$_jvstb34_0.testHappyCase(BlobStoreManagedLedgerOffloaderTest_$$_jvstb34_0.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110)
Caused by: org.jclouds.http.HttpResponseException: Error parsing input: null
{statusCode=200, message=OK, headers={Alt-Svc=[h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"], Server=[UploadServer], ETag=[CJqtx5TtqeoCEAE=], X-GUploader-UploadID=[AAANsUmvovukkc5rtoafAhiNOnn2wTXoK6qub8xX1a_S2I8wN_3RfqhyswUdhEKTkMMo67j0HfxVf6e70Nzs-YPjrrQ], Vary=[X-Origin, Origin], Pragma=[no-cache], Date=[Tue, 30 Jun 2020 15:28:08 GMT]}, payload=[content=true, contentMetadata=[cacheControl=no-cache, no-store, max-age=0, must-revalidate, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=951, contentMD5=null, contentType=application/json; charset=UTF-8, expires=Mon Jan 01 08:00:00 CST 1990], written=false, isSensitive=false]}
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:67)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:41)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
	at com.sun.proxy.$Proxy82.composeObjects(Unknown Source)
	at org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.completeMultipartUpload(GoogleCloudStorageBlobStore.java:405)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy80.completeMultipartUpload(Unknown Source)
	at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$offload$4(BlobStoreManagedLedgerOffloader.java:463)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:319)
	at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:287)
	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$ParameterReader.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:275)
	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:187)
	at com.google.gson.Gson.fromJson(Gson.java:888)
	at com.google.gson.Gson.fromJson(Gson.java:853)
	at org.jclouds.json.internal.GsonWrapper.fromJson(GsonWrapper.java:55)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:82)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:76)
	at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:61)
	... 28 more

Modifications

  1. Exclude the com.google.code.gson:gson from org.apache.pulsar:managed-ledger in org.apache.pulsar:tiered-storage-jcloud module. This will cause gcs offload operation failed.
  2. Update Jclouds version from 2.2.0 to 2.2.1, the new version fixes some bugs in Jclouds.

Verifying this change

It's not convenient to add real environment tests for GCS and AWS, I make some tests use my personal accounts.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

@gaoran10
1. Update jclouds version from 2.2.0 to 2.2.1.
a4c1fb7 
2. Exclude the `com.google.code.gson:gson` from `org.apache.pulsar:managed-ledger` in `org.apache.pulsar:tiered-storage-jcloud` module. This will cause gcs offload operation failed.
@codelipenghui codelipenghui added this to the 2.7.0 milestone Jul 1, 2020
@codelipenghui codelipenghui requested review from sijie, yjshen, jiazhai and merlimat and removed request for sijie and yjshen July 1, 2020 01:14
@gaoran10
Copy link
Contributor Author

gaoran10 commented Jul 1, 2020

/pulsarbot run-failure-checks

@gaoran10 gaoran10 changed the title [Issue] Fix GCS offload operation error [Issue 7402] Fix GCS offload operation error Jul 1, 2020
@merlimat merlimat merged commit d06a52c into apache:master Jul 1, 2020
@gaoran10 gaoran10 mentioned this pull request Jul 2, 2020
Merged
@gaoran10
Copy link
Contributor Author

This issue of this PR wants to fix is fixed by #7435.

deltasquare4 pushed a commit to pandio-com/pulsar that referenced this pull request Jul 30, 2020
@gaoran10 @deltasquare4
1. Update jclouds version from 2.2.0 to 2.2.1. (apache#7400)
f660077 
2. Exclude the `com.google.code.gson:gson` from `org.apache.pulsar:managed-ledger` in `org.apache.pulsar:tiered-storage-jcloud` module. This will cause gcs offload operation failed.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
@gaoran10 @huangdx0726
1. Update jclouds version from 2.2.0 to 2.2.1. (apache#7400)
103546e 
2. Exclude the `com.google.code.gson:gson` from `org.apache.pulsar:managed-ledger` in `org.apache.pulsar:tiered-storage-jcloud` module. This will cause gcs offload operation failed.
@gaoran10 gaoran10 deleted the gcs-offload-fix branch October 11, 2020 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merlimat merlimat merlimat approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@sijie sijie Awaiting requested review from sijie

@jiazhai jiazhai Awaiting requested review from jiazhai

Assignees

@gaoran10 gaoran10

Labels
None yet
Projects
None yet
Milestone
2.7.0
Development

Successfully merging this pull request may close these issues.

GCS offload operation error
3 participants
@gaoran10 @merlimat @codelipenghui