-o when consuming from multiple partitions?

[ottomata]

I think I asked this before, but I don't remember the answer.

When consuming from multiple partitions, what does the -o <offset> flag do? Offsets are for particular partitions. Which partition's offset is -o referring to?

[edenhill]

It applies to all of the partitions, which is highly impractical if it werent for the symbolic offsets:

• beginning
• end
• -NNNN (tail)
• stored

[ottomata]

So, it doesn’t work with manually entered offsets? Or, how does stored work with kafkacat? Is there a way to tell it to store and read offsets in a directory (like kafkatee?)

On Oct 16, 2014, at 12:18 PM, Magnus Edenhill notifications@github.com wrote:

It applies to all of the partitions, which is highly impractical if it werent for the symbolic offsets:

beginning
end
-NNNN (tail)
stored
—
Reply to this email directly or view it on GitHub.

[edenhill]

It doesnt work with absolute offsets, no, or.. it works in the way that it'll use that offset for all partitions ,which is probably wrong.

Use -X topic.offset.store.path=dir to specify a directory.

I'm guessing you want something like:
kafkacat .... -t muh_topic -o 192 -p 0 -o 935 -p 1 -o 512 -p 2

• partition 0: offset 192
• partition 1: offset 935
• partition 2: offset 512

right?

[ottomata]

Ah! offset.store.path will work just fine, I think. Just tested this and I think it works.

Question, if I use -e, or -c , will the offset files be written to when kafkacat exits? Or will they only be written to every auto.commit.interval.ms?

On Oct 16, 2014, at 1:07 PM, Magnus Edenhill notifications@github.com wrote:

It doesnt work with absolute offsets, no, or.. it works in the way that it'll use that offset for all partitions ,which is probably wrong.

Use offset.store.path to specify a directory.

I'm guessing you want something like:
kafkacat .... -t muh_topic -o 192 -p 0 -o 935 -p 1 -o 512 -p 2

partition 0: offset 192
partition 1: offset 935
partition 2: offset 512
right?

—
Reply to this email directly or view it on GitHub.

[edenhill]

Both, every commit.interval.ms and at shutdown.

[ottomata]

What happens if SIGINT (ctrl-C?)

[edenhill]

It will exit cleanly (and commit).

[edenhill]

Huhm, maybe it wont, it will destroy rdkafka but doesnt wait for completion. I should fix that.

[edenhill]

Do you desire the per-partition -o offset switch?

[ottomata]

No, I don’t think so… That might be useful for debugging now and then, but the stored offset files should do just fine.

On Oct 17, 2014, at 3:27 AM, Magnus Edenhill notifications@github.com wrote:

Do you desire the per-partition -o offset switch?

—
Reply to this email directly or view it on GitHub.