Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add options to include topic metadata and offsets in list #19

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

retzkek
Copy link

@retzkek retzkek commented Jul 24, 2020

Today I needed to review the retentions configured for all our topics. Rather than iterating over all the topics with the shell I thought it might make sense to have the list command grab the metadata too. I added -c and -o options, same as describe, that causes it to grab the config and offsets and stick it into a Metadata field.

Right now it only shows up in the json output, which probably makes sense, since the table would be unwieldy with all the metadata (but it would be really nice to be able to choose columns of interest).

You may have some better idea or preference on how this is done, I'm open to ideas, but this met my need so I wanted to offer it back.

Example usage:

$ trubka list topics  -c -t ingest\.dcache\.billing -f json | \
jq -c '.[] | {"name":.name, "retention_days":(.metadata.configurations | from_entries | ."retention.ms" | tonumber /1000/3600/24)}'
{"name":"ingest.dcache.billing","retention_days":7}
{"name":"ingest.dcache.billing.cms-disk","retention_days":7}
{"name":"ingest.dcache.billing.cms-tape","retention_days":30}

Copy link
Owner

@xitonix xitonix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution @retzkek . Really appreciate it.

My only concern with this PR is the amount of data it will be pulling from the server when the user has many topics (We have got 400+ active topics at the moment for example). But considering the fact that the metadata can be loaded optionally, it's probably fine. That said, I think it would be great if we could have the metadata in, plain, tree and table formats as well, just to keep the output consistent with the rest of the commands. To be honest I'm not sure, about the readability of the tabular output either, but as far as trubka is concerned, I believe it should give it to the user regardless. I'm more than happy to approve your PR and spend a bit of time on the tabular output for the next release. It's also fine by me, if you would like to work on it yourself and pack it in the same PR. Please let me know which way you would prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants