Skip to content

[Bug] unaware bucket mode does not support multiple writers writing to the same kafka partition when using kafka as logSystem. #4515

@liming30

Description

@liming30

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

paimon-1.0-snapshot

Compute Engine

flink-1.17

Minimal reproduce step

  1. create an append-only table and use kafka as the logsystem.
  2. use flink to write to the table and set the write parallelism to greater than 1.

What doesn't meet your expectations?

Caused by: java.lang.RuntimeException: bucket-0 appears multiple times, which is not possible.
	at org.apache.paimon.manifest.ManifestCommittable.addLogOffset(ManifestCommittable.java:68)
	at org.apache.paimon.flink.sink.StoreCommitter.combine(StoreCommitter.java:97)
	at org.apache.paimon.flink.sink.StoreCommitter.combine(StoreCommitter.java:79)
	at org.apache.paimon.flink.sink.StoreCommitter.combine(StoreCommitter.java:42)

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions