Offloading fails

[macthestack]

The following happens when I try to offload manually using: pulsar-admin topics offload --size-threshold 0 public/default/my-topic && pulsar-admin topics offload-status -w public/default/my-topic

Offload triggered for persistent://public/default/my-topic for messages before 6:0:-1
Error in offload
null

Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: java.lang.UnsupportedOperationException
command terminated with exit code 1

The last broker log regarding offloading:

INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/my-topic] Metadata prepared for offload of ledger 4 with uuid da861b42-3d30-4158-babd-f8a73e5850b7

Everything else about the cluster seems to be running fine by the looks of logs and grafana.

Steps to reproduce

Spin up a new kubernetes cluster on gcp and try offloading to tiered storage following the cookbook

I have tried both the helm deployment and the gcp specific one and I have also tried both gcp and aws-s3 storage.

System configuration

Pulsar version: 2.1.1

[ivankelly]

Have you configured an offload driver? What does your broker configuration look like?

[macthestack]

### --- Ledger Offloading --- ###

# Driver to use to offload old data to long term storage (Possible values: S3)
managedLedgerOffloadDriver=aws-s3

# Maximum number of thread pool threads for ledger offloading
managedLedgerOffloadMaxThreads=2

# For Amazon S3 ledger offload, AWS region
s3ManagedLedgerOffloadRegion=eu-west-1

# For Amazon S3 ledger offload, Bucket to place offloadedledger into
s3ManagedLedgerOffloadBucket=pulsar-topic-offload

# For Amazon S3 ledger offload, Alternative endpoint to connect to (useful for testing)
s3ManagedLedgerOffloadServiceEndpoint=

# For Amazon S3 ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864

# For Amazon S3 ledger offload, Read buffer size in bytes(1MB by default)
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576

I'm also setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

[ivankelly]

@macthestack did you restart the broker after updating the configuration? UnsupportedOperationException looks like its coming from the Null offloader.
https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/NullLedgerOffloader.java

[macthestack]

I actually set it before the broker start in this sequence:

bin/apply-config-from-env.py conf/broker.conf &&
bin/apply-config-from-env.py conf/pulsar_env.sh &&
bin/pulsar broker

[ivankelly]

Could you post the full broker log?

[macthestack]

sure, I'll spin up a fresh cluster, thanks for the help

[macthestack]

Here we go:

stderr
stdout

Sorry for the format. jq --raw-output '.textPayload' stdout.json -j is helpful

[ivankelly]

@macthestack

# Driver to use to offload old data to long term storage (Possible values: S3)
managedLedgerOffloadDriver=aws-s3

For 2.1.1-incubating, only 'S3' is supported as a driver. It looks like the 2.1.1-incubating version of the docs on the site is out of sync with what is in the codebase :/

Try with S3 as the driver. I'll look into the documentation issue.

[macthestack]

Thanks @ivankelly, that did the trick

[ivankelly]

I'm glad to hear it. Let us know if you have more trouble.