StarRocks version (Required)
StarRocks 4.0.0-dccb665
Deployment type
StarRocks cluster on Kubernetes (using StarRocks Kubernetes Operator) in shared-data mode.
Bug description
After deploying a new cluster with StarRocks v4.0.0-dccb665, it runs properly for a few days. However, at some point, we fail to query one of our production Iceberg tables. It seems StarRocks is trying to access a stale metadata file that no longer exists.
This issue repeats itself every 2-3 days.
Steps to reproduce (Required)
We are not sure how to reproduce it reliably, as it happens intermittently every 2-3 days on a production table.
Infrastructure information:
-
The Iceberg table is updated and maintained by Apache Spark jobs.
-
We use Hive Metastore v3.1.3 as our Iceberg catalog.
-
We use HDFS as the storage layer.
Expected behavior (Required)
Queries on the Iceberg table should execute successfully.
Real behavior (Required)
Queries fail with "File does not exist" or "Failed to open the off-heap table scanner" errors, pointing to metadata files that are no longer present.
Example 1: simple select query
select * from iceberg.schema_name.prod_table limit 5;
Error:
[42000][1064] File does not exist: hdfs://hdfs-cluster-name/path/to/iceberg/table/metadata/snap-2583631546770153293-1-ea437b69-855a-4fb1-a84a-0581ddfa784d.avro
Example 2: Querying snapshots
select * from iceberg.schema_name.prod_table$snapshots;
Error:
[42000][1064] Failed to open the off-heap table scanner. java exception details: java.io.IOException: Failed to open the IcebergSnapshotsTableScanner
at com.starrocks.connector.iceberg.AbstractIcebergMetadataScanner.open(AbstractIcebergMetadataScanner.java:76)
Caused by: org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: hdfs://hdfs-cluster-name/path/to/iceberg/table/metadata/22827-34098b81-8440-4746-8919-80291e8bfc7b.metadata.json
Additional important information
-
REFRESH EXTERNAL TABLE does not work: We tried to run REFRESH EXTERNAL TABLE iceberg.schema_name.prod_table; to explicitly update the metadata cached in StarRocks, but the issue persisted.
-
Workaround: The only workaround that resolved the issue temporarily was to manually DROP and CREATE the iceberg external catalog.
-
We do not experience this issue in other query engines (e.g., Trino, Apache Spark) querying the same table.
-
Version-specific: We do not experience this issue in older StarRocks versions. We successfully queried the exact same table using an existing StarRocks v3.4 cluster with no issues. This leads us to believe it is a new bug in v4.0.
-
Isolated table: So far, we've experienced this issue only with one of our production Iceberg tables. The rest of the tables seem to work well, even though they are also maintained by Apache Spark jobs.
StarRocks version (Required)
StarRocks 4.0.0-dccb665
Deployment type
StarRocks cluster on Kubernetes (using StarRocks Kubernetes Operator) in shared-data mode.
Bug description
After deploying a new cluster with StarRocks v4.0.0-dccb665, it runs properly for a few days. However, at some point, we fail to query one of our production Iceberg tables. It seems StarRocks is trying to access a stale metadata file that no longer exists.
This issue repeats itself every 2-3 days.
Steps to reproduce (Required)
We are not sure how to reproduce it reliably, as it happens intermittently every 2-3 days on a production table.
Infrastructure information:
The Iceberg table is updated and maintained by Apache Spark jobs.
We use Hive Metastore v3.1.3 as our Iceberg catalog.
We use HDFS as the storage layer.
Expected behavior (Required)
Queries on the Iceberg table should execute successfully.
Real behavior (Required)
Queries fail with "File does not exist" or "Failed to open the off-heap table scanner" errors, pointing to metadata files that are no longer present.
Example 1: simple
selectqueryselect * from iceberg.schema_name.prod_table limit 5;Error:
[42000][1064] File does not exist: hdfs://hdfs-cluster-name/path/to/iceberg/table/metadata/snap-2583631546770153293-1-ea437b69-855a-4fb1-a84a-0581ddfa784d.avroExample 2: Querying snapshots
select * from iceberg.schema_name.prod_table$snapshots;Error:
Additional important information
REFRESH EXTERNAL TABLEdoes not work: We tried to runREFRESH EXTERNAL TABLE iceberg.schema_name.prod_table;to explicitly update the metadata cached in StarRocks, but the issue persisted.Workaround: The only workaround that resolved the issue temporarily was to manually DROP and CREATE the iceberg external catalog.
We do not experience this issue in other query engines (e.g., Trino, Apache Spark) querying the same table.
Version-specific: We do not experience this issue in older StarRocks versions. We successfully queried the exact same table using an existing StarRocks v3.4 cluster with no issues. This leads us to believe it is a new bug in v4.0.
Isolated table: So far, we've experienced this issue only with one of our production Iceberg tables. The rest of the tables seem to work well, even though they are also maintained by Apache Spark jobs.