Can not query Iceberg over the swarm when the Iceberg table uses a mounted file system.

[aadant]

Type of problem

Choose one of the following items, then delete the others:

Bug report - something's broken

Describe the situation

Create an Iceberg table that is pointing to a shared file system available to all swarm nodes.
Query the table in initiator node with SETTINGS object_storage_cluster = 'swarm'

<Error> ForcedCriticalErrorsLogger: Code: 49. DB::Exception: Can't find table function for engine IcebergLocal. (LOGICAL_ERROR), Stack trace (when copying this message, always include the lines below):

How to reproduce the behavior

Antalya 25.6, see description

Expected behavior

It should work similar to the initiator node.

Logs, error messages, stacktraces, screenshots...

Add any details that might explain the issue.

Additional context

Add any other context about the issue here.

[aadant]

How to reproduce : see Altinity/ice#43 (comment)

# .ice-rest-catalog.yaml
uri: jdbc:sqlite:file:data/ice-rest-catalog/catalog.sqlite?journal_mode=WAL&synchronous=OFF&journal_size_limit=500
warehouse: file://tmp/warehouse
localFileIOBaseDir: /
bearerTokens:
  - value: foo

# .ice.yaml
uri: http://localhost:5000
warehouse: file://tmp/warehouse
localFileIOBaseDir: /
localFileIOAllowAccess:
  - /tmp

ice insert flowers.iris -p file://iris.parquet
ice insert flowers.iris -p --force-no-copy file:///tmp/iris.parquet
ice describe flowers.iris
ice scan flowers.iris --limit=1

in https://github.com/Altinity/antalya-examples/blob/main/docker/docker-compose.yml

with 25.6 and mounting the warehouse location for vector, swarm-1 and swarm-2 (example swarm-1 below ) :

 swarm-1:
    image: altinity/clickhouse-server:25.6.5.20363.altinityantalya
    container_name: swarm-1
    runtime: runc
    hostname: swarm-1
    volumes:
      - ./config.d/swarm:/etc/clickhouse-server/config.d
      - ./users.d:/etc/clickhouse-server/users.d
      - ./logs/swarm-1:/var/log/clickhouse-server
      - /tmp/warehouse:/tmp/warehouse
    environment:
      - AWS_ACCESS_KEY_ID=minio
      - AWS_SECRET_ACCESS_KEY=minio123
      - AWS_REGION=us-east-1

SET allow_experimental_database_iceberg = 1;

-- (re)create iceberg db
DROP DATABASE IF EXISTS ice;

CREATE DATABASE ice
ENGINE = DataLakeCatalog('http://localhost:5000')
SETTINGS catalog_type = 'rest',
auth_header = 'Authorization: Bearer foo',
vended_credentials = false,
warehouse = 'warehouse';

select count(*) from ice.\`flowers.iris\`;

150

create the database in vector, swarm-1, swarm-2 and mount /tmp/warehouse to the containers.

you should get

vector :) select hostName(), count(*) from cluster(all,ice,`flowers.iris`) group by hostName();

SELECT
    hostName(),
    count(*)
FROM cluster(`all`, ice, `flowers.iris`)
GROUP BY hostName()

Query id: 38125ab0-623b-4ee6-8fa8-1738bf354d12

   ┌─hostName()─┬─count()─┐
1. │ vector     │     150 │
2. │ swarm-1    │     150 │
3. │ swarm-2    │     150 │
   └────────────┴─────────┘

3 rows in set. Elapsed: 0.013 sec.

then

vector :) select count(*) from ice.`flowers.iris` SETTINGS object_storage_cluster = 'all';

SELECT count(*)
FROM ice.`flowers.iris`
SETTINGS object_storage_cluster = 'all'

Query id: f08062e6-47be-4060-9070-ad39640e2f64

   ┌─count()─┐
1. │     150 │
   └─────────┘

1 row in set. Elapsed: 0.005 sec.

vector :) select * from ice.`flowers.iris` SETTINGS object_storage_cluster = 'all';

SELECT *
FROM ice.`flowers.iris`
SETTINGS object_storage_cluster = 'all'

Query id: b016ad3b-a0cb-4131-94e9-8ba384d2191f


Elapsed: 0.007 sec.

Received exception from server (version 25.6.5):
Code: 49. DB::Exception: Received from localhost:9000. DB::Exception: Can't find table function for engine IcebergLocal. (LOGICAL_ERROR)

[ianton-ru]

May be I don't fully understand the issue, but query to local iceberg table cannot be executed with swarm - swarm nodes don’t have access to initiator local disk. It is possible only with remote S3/Azure storage.

I see two options:

• Change error message on something like “Distributed query cannot be executed with local table“.
• Ignore object_storage_cluster setting and execute query locally on initiator.

[aadant]

You should support this use case without an error message because there are distributed file systems or network attached file systems.
Some of them are even used to cache object storage. In my example, the same local file system is attached. It should work if the data is available on all nodes. Cc @hodgesrm

[aadant]

I think this would be solved with an icebergLocalCluster table function.
does that make sense ?

[ianton-ru]

I don't think icebergLocalCluster is a right way.
For S3/Azure storage ClickHouse have knowledge that data is shared between nodes, uses ZooKeeper to write coordination, and can split files between nodes during reading.
For local case each ClickHouse node thinks that it uses storage exclusively, without worrying about someone else changing the data. And different nodes can have different data when storage is really local. So distributed reading can completely break real local case.

Correct way is to make new storage type for case with shared disk storage. It is a large work.

Easy way is to make setting like object_storage_cluster_for_local (I'm not good in naming) for this specific case to avoid accidental error with real local tables.

[aadant]

We just need that exact feature similar to fileCluster (check how it was implemented).
icebergLocalCluster is the same thing for Iceberg. How is that an issue ?

[ianton-ru]

#1120