[fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema

[heesung-sn]

Potentially related to
#15267
#14592

Motivation

Broker returns this error, No such ledger exists on Metadata Server to client when deleting a partitioned topic with many partitions and schema.

The root cause is that

• the current code tries to delete the same schema for each topic partition, when that topic partition's reference is not ready.
• the delete schema operation is not idempotent enough, passing the 'No such ledger exists on Metadata Server' bk error to the client, when the schema is not found.

reproduce steps

bin/pulsar-admin namespaces create test/test-traffic --bundles 1
bin/pulsar-admin topics create-partitioned-topic persistent://test/test-traffic/test-topic -p 500
bin/pulsar-admin schemas upload --filename schema.json persistent://test/test-traffic/test-topic
bin/pulsar-admin topics delete-partitioned-topic persistent://test/test-traffic/test-topic



Message: No such ledger exists on Metadata Server -  ledger=xxxxx - operation=Failed to open ledger

Stacktrace:

org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=xxxx - operation=Failed to open ledger
        at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.bkException(BookkeeperSchemaStorage.java:711)
        at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$openLedger$41(BookkeeperSchemaStorage.java:605)
        at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:260)
        at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:116)
        at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
        at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerManager.lambda$readLedgerMetadata$2(PulsarLedgerManager.java:208)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleGetResult(ZKMetadataStore.java:269)
        at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:219)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)


Reason:
 --- An unexpected error occurred in the server ---

Modifications

• Return completed future in BrokerService.deleteSchema when deleting a topic partition since the schema deletion should only happen at the upper level, when the base (partitioned) topic is finally deleted.
• Remove the redundant getSchema check in BrokerService.deleteSchema as the BookkeeperSchemaStorage.delete(String key) already checks the getSchema inside by default.
• Add BK error NoSuchLedgerExistsOnMetadataServerException (-25) in the unrecoverable conditions when throwing SchemaException for better idempotency.

Verifying this change

• Make sure that the change passes the CI checks.

This change added test.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[heesung-sn]

related to : #19882

[codelipenghui]

LGTM!

It better also update the comments at here

[heesung-sn]

It better also update the comments at here

Updated the comment.

[Technoboy-]

Related fix #21574

[KannarFr]

Do __change_events topics involved? I have partitions enforced on the cluster, so __change_events are partitioned too.

Jan 31 14:24:55 yo-pulsar-broker-c3-n6 pulsar[86483]: 2024-01-31T14:24:55,666+0000 [pulsar-io-3-11] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://orga_15611b30-6e4e-44a5-a80f-309d277e27fb/logs/__change_events-partition-0] [multiTopicsReader-6a2bfce189] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"No such ledger exists on Metadata Server -  ledger=20202 - operation=Failed to open ledger","reqId":3644475206336716231, "remote":"yo-pulsar-broker-c3-n2/192.168.2.2:6650", "local":"/192.168.2.6:44728"}
Jan 31 14:24:55 yo-pulsar-broker-c3-n6 pulsar[86483]: 2024-01-31T14:24:55,666+0000 [pulsar-io-3-11] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://orga_15611b30-6e4e-44a5-a80f-309d277e27fb/logs/__change_events-partition-0] [multiTopicsReader-6a2bfce189] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"No such ledger exists on Metadata Server -  ledger=20202 - operation=Failed to open ledger","reqId":3644475206336716231, "remote":"yo-pulsar-broker-c3-n2/192.168.2.2:6650", "local":"/192.168.2.6:44728"} -

Running broker with this patch.

I'm also encountering

Jan 31 14:47:56 yo-pulsar-broker-c3-n1 pulsar[3975173]: 2024-01-31T14:47:56,905+0000 [broker-client-shared-internal-executor-5-1] ERROR org.apache.pulsar.broker.service.BrokerService - Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://orga_19647583-2185-4e67-96e5-202846a9522b/logs/app_e208acde-73a0-4248-8b55-5ae453b13581-partition-0 error_message={"errorMsg":"No such ledger exists on Metadata Server -  ledger=15336 - operation=Failed to open ledger","reqId":836394464329855562, "remote":"yo-pulsar-broker-c3-n1/192.168.2.1:6650", "local":"/192.168.2.1:58448"}    
Jan 31 14:47:56 yo-pulsar-broker-c3-n1 pulsar[3975173]: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"No such ledger exists on Metadata Server -  ledger=15336 - operation=Failed to open ledger","reqId":836394464329855562, "remote":"yo-pulsar-broker-c3-n1/192.168.2.1:6650", "local":"/192.168.2.1:58448"}

[heesung-sn]

@KannarFr
can you share any full stacktrace when you hit "errorMsg":"No such ledger exists on Metadata Server - ledger=15336 ?, or can you share any reproduce steps?

This PR is specifically trying to fix the ledger error, No such ledger exists on Metadata Server, when deleting schema as part of deleting topic. Also, this ledger error is specifically from the schema deletion(schema registry), SchemaException: No such ledger exists on Metadata Server - ledger=xxxx - operation=Failed to open ledger.

Also, do your errors start showing with this fix as regression?

[KannarFr]

Ok, it looks not related to this issue my bad. I'll open another issue about my problem.

[heesung-sn]

Ok, it looks not related to this issue my bad. I'll open another issue about my problem.

I see. Thanks.