Fluss fails to drop underlying paimon tables

[TakodaS]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.8.0 (latest release)

Please describe the bug 🐞

Running the following in the Flink engine:

CREATE CATALOG fluss WITH (
    'type' = 'fluss',
    'bootstrap.servers' = 'localhost:9123'
);
USE CATALOG fluss;
CREATE DATABASE foo;
USE foo;

CREATE TABLE bar (
    id STRING
) WITH (
    'table.datalake.enabled' = 'true'
);
DROP TABLE bar;

CREATE TABLE bar (
    id STRING
) WITH (
    'table.datalake.enabled' = 'true'
);

Results in a failure on the last create statement with exception

CatalogException: The table foo.bar already exists in paimon catalog, please first drop the table in paimon catalog or use a new table name.

This suggests fluss fails to correctly delete the underlying paimon table, which I don't think is intended behavior.

EDIT: Fluss server.yaml:

datalake.format: paimon
datalake.paimon.warehouse: s3://paimon-warehouse/fluss-remote
datalake.paimon.s3.endpoint: http://minio:9000
datalake.paimon.s3.access-key: minioadmin
datalake.paimon.s3.secret-key: minioadmin
datalake.paimon.s3.path.style.access: true

s3.endpoint: http://minio:9000
s3.access-key: minioadmin
s3.secret-key: minioadmin
s3.path.style.access: true

data.dir: /tmp/fluss/data
remote.data.dir: s3://paimon-warehouse/fluss-remote

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[TakodaS]

I realized minio is not currently supported by Fluss #1246. Will retry with local storage.

[LiebingYu]

Hi @TakodaS. When a Fluss table is dropped, the corresponding lake table is not automatically dropped. This is the current design. Let me share my understanding of the rationale behind this (though it might not be exhaustive):

In production environments, even though we expect all reads, writes, and DDL operations on the lake table to be managed through Fluss, they are ultimately two distinct tables residing in two separate systems. There will inevitably be scenarios where users no longer need the Fluss table but still wish to retain the underlying lake table.

Therefore, at this stage, when dropping a unified lake-stream table, you need to manually drop both the Fluss table and the lake table.

[TakodaS]

I see. Thanks for your answer, I will close the issue. I have a couple of follow-up questions:

1. would I therefore have to manually use a paimon catalog definition to access the underlying tables?
2. Is there a tool you could recommend to help with schema migration (like sqlmesh or whatnot) that would handle modifying table definitions? Or is this too new to Fluss for any integration yet

Thank you!