Set splitNamespaceBundle with readonly=false.

[Technoboy-]

Master Issue: #14668

Fixes: #14668

Motivation

When we split a not loaded namespace bundle, we will meet the below error:

Failed to find ownership for ServiceUnit:tenant/namespace/0x00000000_0x10000000

Because when validating namespace bundle ownership with readonly=true :

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

Lines 1145 to 1151 in fe7e55d

	try {
	nsBundle = validateNamespaceBundleOwnership(namespaceName, policies.bundles, bundleRange,
	authoritative, true);
	} catch (Exception e) {
	asyncResponse.resume(e);
	return;
	}

and if the bundle is not owned by any broker, it will return empty(line-392):

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java

Lines 388 to 400 in fe7e55d

	ownershipCache.getOwnerAsync(bundle).thenAccept(nsData -> {
	if (!nsData.isPresent()) {
	// No one owns this bundle
	if (options.isReadOnly()) {
	// Do not attempt to acquire ownership
	future.complete(Optional.empty());
	} else {
	// Now, no one owns the namespace yet. Hence, we will try to dynamically assign it
	pulsar.getExecutor().execute(() -> {
	searchForCandidateBroker(bundle, future, options);
	});
	}

so throw the below exception :

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java

Lines 576 to 582 in fe7e55d

	Optional<URL> webUrl = nsService.getWebServiceUrl(bundle, options);
	// Ensure we get a url
	if (webUrl == null || !webUrl.isPresent()) {
	log.warn("Unable to get web service url");
	throw new RestException(Status.PRECONDITION_FAILED,
	"Failed to find ownership for ServiceUnit:" + bundle.toString());
	}

Modification

• Change readonly from true to false when validating namespace bundle ownership.

Documentation

• no-need-doc

[Technoboy-]

As the difference I said in #14668, I'm still curious why standalone works while cluster doesn't work. See the following screenshot, even if readOnly is true, the webUrl is still retrieved successfully.

This is not for cluster only, standalone is also not work. My test could reproduce.

[BewareMyPower]

I deleted my reply at this moment. I'll dig deeper into this issue.

[BewareMyPower]

I think the difference is that when I ran a standalone, each bundle was owned by the broker because I've sent some messages to some partitions. See the original code snippet:

            try (PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build()) {
                for (int i = 0; i < numPartitions; i++) {
                    client.newProducer().topic(topic + TopicName.PARTITIONED_TOPIC_SUFFIX + i)
                            .create().send("1".getBytes());
                }
            }

When I debugged in a standalone, i.e. the nsData returned by ownershipCache.getOwnerAsync(bundle) is not empty.

There is another weird thing I've described in #14668. I'm not sure if it's related.

The standard output is also weird:

[0x00000000, 0x10000000, 0x20000000, 0x30000000, 0x40000000, 0x50000000, 0x60000000, 0x70000000, 0x80000000, 0x90000000, 0xa0000000, 0xb0000000, 0xc0000000, 0xd0000000, 0xe0000000, 0xf0000000, 0xffffffff]
The defaultNumberOfNamespaceBundles is 4, but the initial number of bundles is 16.

Maybe some bundles were not owned by the broker. I'll try to split all bundle ranges to see if there is any bundle range that can be split.

[BewareMyPower]

It works now.

When I ran a standalone, there were 4 bundle ranges. However, when I ran a cluster, there were 16 bundle ranges and some bundle ranges were not owned.

I think we need another issue to track the issue that defaultNumberOfNamespaceBundles doesn't work for cluster mode.