Skip to content

s3 offloading generates huge zk txn logs in some conditions #5798

@vicaya

Description

@vicaya

Describe the bug
When s3 offloading is turned on but credentials is incorrect the zookeeper /managed-ledger znode entries grow to be fairly large (1MB), causing zk txn log to be in the range of multiple GBs, which could easily run out of the space on the zk log device under default log purging configuration.

An example of the content of such a znode:

k: localhost:2181(CONNECTED) 0] get /managed-ledgers/<topic-path>
?????  ?????-*???????????????????2?*
s34
bucket*<bucket>
region  us-west-2.
endpoint"https://s3.us-west-2.amazonaws.com4
bucket*<bucket>
region  us-west-2.
endpoint"https://s3.us-west-2.amazonaws.com4
bucket*<bucket>
region  us-west-2.

where the bucket, endpoint, and region info appears to be duplicated many times. The entry size also grows under normal offloading operation conditions but slower.

To Reproduce
See above. We reproduced the problem with fake topics

Expected behavior
znode entry size should be bounded. There appears no reason to duplicate offloading metadata.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugThe PR fixed a bug or issue reported a bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions