Allow to configure the default number of bundles for new namespaces

[merlimat]

Motivation

By default namespaces are currently created with a single bundle.

In many examples, we need then to explain how to specify a number of bundles (and explaining what bundles are and what they mean).

Bundles are meant to be a very internal concept that in the overweeningly number of cases user shouldn't need to be aware of.
Apart from auto-splitting, which is currently disabled, we need to ensure to start with a reasonable default of bundles that will allow to spread traffic across multiple broker immediately.

Modifications

• Added a configuration setting in broker to set the default number of bundles
• When a namespace is created, it will use that value unless something different is specified by the user

Result

A small step towards hiding the bundles from user sight.

[massakam]

👍

[rdhabalia]

creating more bundles will increase shutdowntime of broker because broker unloads topic in parallel but unloading bundle happens sequentially. Using load-balancer we can always split eligible bundle but we can't merge bundle once we split. So, should we keep this number lower than 16.??

[merlimat]

creating more bundles will increase shutdowntime of broker because broker unloads topic in parallel but unloading bundle happens sequentially. Using load-balancer we can always split eligible bundle but we can't merge bundle once we split. So, should we keep this number lower than 16.??

That's a good point, though I suspect it won't make any difference outside Yahoo 😄

We could lower the default to something like 4, if 16 poses concerns. The goal here is to have the traffic on partitioned topics to be immediately distributed to at least few machines.

In any case, for the shutdown time, probably a better option could be to throttle based on number of topics per bundle.

Instead of shutting down bundles 1 by 1, we could keep shutting down bundles in parallel until a certain threshold of topics is reached. (eg: a semaphore with max 1K topics)

That would remove any disadvantage in having more bundles with fewer topics.

[rdhabalia]

We could lower the default to something like 4, if 16 poses concerns.

Sure, initially we can start with 4 bundles so, topics can be distributed in multiple brokers and then let load-balancer take care further splitting.

[rdhabalia]

👍