Ledger offload to tiered storage fails with NoClassDefFoundError: com/jamesmurty/utils/XMLBuilder

[chairmank]

We see this stacktrace on brokers when we trigger offload to tiered storage:

13:35:26.044 [offloader-OrderedScheduler-1-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed delete Blob
java.lang.NoClassDefFoundError: com/jamesmurty/utils/XMLBuilder
        at org.jclouds.s3.binders.BindIterableAsPayloadToDeleteRequest.bindToRequest(BindIterableAsPayloadToDeleteRequest.java:54) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT
]
        at org.jclouds.rest.internal.RestAnnotationProcessor.decorateRequest(RestAnnotationProcessor.java:660) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:354) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:137) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.InvokeHttpMethod.toCommand(InvokeHttpMethod.java:189) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:85) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at com.sun.proxy.$Proxy72.deleteObjects(Unknown Source) ~[?:?]
        at org.jclouds.s3.blobstore.S3BlobStore.removeBlobs(S3BlobStore.java:343) ~[jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$deleteOffloaded$4(BlobStoreManagedLedgerOffloader.java:561) [tiered-storage-jclou
d-2.4.0-SNAPSHOT.nar-unpacked/:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [com.google.guava-guav
a-21.0.jar:?]
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [com.google.guava-guava-21.0.jar:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [com.google.guava-guava-21.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.ClassNotFoundException: com.jamesmurty.utils.XMLBuilder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:588) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?]
        ... 24 more

It seems that com.jamesmurty.utils.XMLBuilder is missing from the tiered-storage-jcloud NAR file.