-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Closed
Labels
type/bugThe PR fixed a bug or issue reported a bugThe PR fixed a bug or issue reported a bug
Description
Describe the bug
When s3 offloading is turned on but credentials is incorrect the zookeeper /managed-ledger znode entries grow to be fairly large (1MB), causing zk txn log to be in the range of multiple GBs, which could easily run out of the space on the zk log device under default log purging configuration.
An example of the content of such a znode:
k: localhost:2181(CONNECTED) 0] get /managed-ledgers/<topic-path>
????? ?????-*???????????????????2?*
s34
bucket*<bucket>
region us-west-2.
endpoint"https://s3.us-west-2.amazonaws.com4
bucket*<bucket>
region us-west-2.
endpoint"https://s3.us-west-2.amazonaws.com4
bucket*<bucket>
region us-west-2.
where the bucket, endpoint, and region info appears to be duplicated many times. The entry size also grows under normal offloading operation conditions but slower.
To Reproduce
See above. We reproduced the problem with fake topics
Expected behavior
znode entry size should be bounded. There appears no reason to duplicate offloading metadata.
Metadata
Metadata
Assignees
Labels
type/bugThe PR fixed a bug or issue reported a bugThe PR fixed a bug or issue reported a bug