PIP 110: Topic metadata

[Technoboy-]

Motivation

The original discussion mail :
https://lists.apache.org/thread/m9dkhq1fs6stsdwh78h84fsl5hs5v67f

Introduce the ability to store metadata about topics.

This would be very useful as with metadata you could add labels and other
pieces of information that would allow defining the purpose of a topic,
custom application-level properties.
This feature will allow application-level diagnostic tools and maintenance
tools to not need external databases to store such metadata.

Imagine that we could add a simple key value map (String keys and String
values) to the topic.
These metadata could be set during topic creation and also updated.

API Changes

We need to add a map type field called metadata to the below methods signature:

• PersistentTopics#createPartitionedTopic

  PersistentTopics#createPartitionedTopic(AsyncResponse asyncResponse, String tenant, String namespace,String encodedTopic,int numPartitions,boolean createLocalTopicOnly, Map<String, String> metadata);  
  

• PersistentTopics#createNonPartitionedTopic

  PersistentTopics#createNonPartitionedTopic(String tenant, String namespace, String encodedTopic, boolean authoritative, Map<String, String> metadata); 
  

Implementation

• For PartitionedTopic
  We will put these metadata to the PartitionedTopicMetadata, serialized to store in configurationMetadataStore, like partitioned-topic partitions.

  public class PartitionedTopicMetadata {
  
   /* Number of partitions for the topic */
   public int partitions;
   
   /* Topic metadata */
   public Map<String, String> metadata;
  

• For NonPartitionedTopic
  We will store the metadata to ML(ManagedLedger) which already supports storing custom properties.

  ManagedLedger#setProperties(Map<String, String> properties) 
  

Compatibility

The proposal will not introduce any compatibility issues.

Tests Plan

Unit tests & integration tests

[codelipenghui]

LGTM +1

[eolivelli]

We should prevent the user to set properties on individual partitions in a partitioned topic (because partitions are like topic for many aspects)

Please also explain the permissions needed to update such properties (and add authz coverage in the test cases )

[eolivelli]

@Technoboy- did you see my comment?

[Technoboy-]

@Technoboy- did you see my comment?

Ok, I will update this in the pr.