How to query data and write data in Apache Iceberg using StarRocks #23427

alberttwong · 2023-05-15T17:11:58Z

alberttwong
May 15, 2023

This tutorial describes how you can query data in Apache Iceberg using the Apache Iceberg Quickstart.

Note

The last time we ran this tutorial was using StarRocks 3.1.5 and Apache Iceberg 1.4.2

Prerequisites

For this tutorial you need to:

Have Docker Desktop or podman container runtime installed.
Have a MySQL client.
A StarRocks or CelerData database cluster.
An Apache Iceberg Environment.
Configure Apache Iceberg
Create a external catalog and query the data.
Write data into the iceberg tables backed by Minio

Have a MySQL client

This is out of scope for the tutorial.

A StarRocks or CelerData database cluster

This is out of scope for the tutorial. Must use SR 3.0.3 or higher.

An Apache Iceberg environment

This is out of scope for the tutorial. However, we are using the Apache Iceberg docker container for this tutorial.

Configure Apache Iceberg

Follow the Apache Iceberg Docker Quickstart at https://iceberg.apache.org/spark-quickstart/ .

Modify the docker-compose.yml to include the lines:

  starrocks:
    image: starrocks/allin1-ubuntu
    hostname: starrocks-fe
    container_name: allin1-ubuntu-iceberg
    ports:
      - 8030:8030
      - 8040:8040
      - 9030:9030
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1

You will also have to add a network alias to the rest image.

  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
        aliases:
          - iceberg-rest.minio
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000

Here's a cat of my modified docker compose yaml.

version: "3"

services:
  starrocks:
    image: registry.starrocks.io/starrocks/allin1-ubuntu:latest
    hostname: starrocks-fe
    container_name: allin1-ubuntu-iceberg
    ports:
      - 8030:8030
      - 8040:8040
      - 9030:9030
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1

  spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
        aliases:
          - iceberg-rest.minio
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
  minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"]
  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      "
networks:
  iceberg_net:

After this, you can then run docker-compose up and follow the rest of the Apache Iceberg quickstart. Specifically creating the table, inserting data and querying if the data exists.

atwong@Albert-CelerData ~ % docker exec -it spark-iceberg spark-sql
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/12/02 19:00:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/12/02 19:00:44 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark Web UI available at http://3dd8d081c43f:4041
Spark master: local[*], Application Id: local-1701543644357
spark-sql ()> CREATE TABLE demo.nyc.taxis
            > (
            >   vendor_id bigint,
            >   trip_id bigint,
            >   trip_distance float,
            >   fare_amount double,
            >   store_and_fwd_flag string
            > )
            > PARTITIONED BY (vendor_id);
Time taken: 0.988 seconds
spark-sql ()> INSERT INTO demo.nyc.taxis
            > VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');
Time taken: 1.764 seconds
spark-sql ()> SELECT * FROM demo.nyc.taxis;
1	1000371	1.8	15.32	N
1	1000374	8.4	42.13	Y
2	1000372	2.5	22.15	N
2	1000373	0.9	9.01	N
Time taken: 0.516 seconds, Fetched 4 row(s)

Create an external catalog and query the data

Now we're going to create an external catalog in StarRocks.

Login to the SR container within the iceberg docker compose

mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > "

drop catalog iceberg;
create external catalog 'iceberg'
PROPERTIES
(
  "type"="iceberg",
  "iceberg.catalog.type"="rest",
  "iceberg.catalog.uri"="http://iceberg-rest:8181",
  "iceberg.catalog.warehouse"="starrocks",
  "aws.s3.access_key"="admin",
  "aws.s3.secret_key"="password",
  "aws.s3.endpoint"="http://minio:9000",
  "aws.s3.enable_path_style_access"="true",
  "client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
);
show databases from iceberg;
set catalog iceberg;
use nyc;
select * from taxis;

Note

Usually, you don't need to put in the "client.factory" setting. It is there to support the special Apache Iceberg Quickstart environment.

Output should look like this:

atwong@Albert-CelerData ~ % mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > "
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.1.0 3.1.5-5d8438a

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

StarRocks > drop catalog iceberg;
ERROR 1064 (HY000): Unexpected exception: Catalog 'iceberg' doesn't exist
StarRocks > create external catalog 'iceberg'
    -> PROPERTIES
    -> (
    ->   "type"="iceberg",
    ->   "iceberg.catalog.type"="rest",
    ->   "iceberg.catalog.uri"="http://iceberg-rest:8181",
    ->   "iceberg.catalog.warehouse"="starrocks",
    ->   "aws.s3.access_key"="admin",
    ->   "aws.s3.secret_key"="password",
    ->   "aws.s3.endpoint"="http://minio:9000",
    ->   "aws.s3.enable_path_style_access"="true",
    ->   "client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
    -> );
Query OK, 0 rows affected (0.05 sec)

StarRocks > show databases from iceberg;
+----------+
| Database |
+----------+
| nyc      |
+----------+
1 row in set (0.34 sec)

StarRocks > set catalog iceberg;
Query OK, 0 rows affected (0.00 sec)

StarRocks > use nyc;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
StarRocks > select * from taxis;
+-----------+---------+---------------+-------------+--------------------+
| vendor_id | trip_id | trip_distance | fare_amount | store_and_fwd_flag |
+-----------+---------+---------------+-------------+--------------------+
|         1 | 1000371 |           1.8 |       15.32 | N                  |
|         1 | 1000374 |           8.4 |       42.13 | Y                  |
|         2 | 1000372 |           2.5 |       22.15 | N                  |
|         2 | 1000373 |           0.9 |        9.01 | N                  |
+-----------+---------+---------------+-------------+--------------------+
4 rows in set (0.83 sec)

Write data into the iceberg tables backed by Minio

StarRocks > create external catalog 'iceberg'
    -> PROPERTIES
    -> (
    ->   "type"="iceberg",
    ->   "iceberg.catalog.type"="rest",
    ->   "iceberg.catalog.uri"="http://iceberg-rest:8181",
    ->   "iceberg.catalog.warehouse"="starrocks",
    ->   "aws.s3.access_key"="admin",
    ->   "aws.s3.secret_key"="password",
    ->   "aws.s3.endpoint"="http://minio:9000",
    ->   "aws.s3.enable_path_style_access"="true"
    -> );
Query OK, 0 rows affected (0.09 sec)

StarRocks > show databases from iceberg;
Empty set (0.56 sec)

StarRocks > set catalog iceberg;
Query OK, 0 rows affected (0.00 sec)

StarRocks > create database demo
    -> ;
Query OK, 0 rows affected (0.11 sec)

StarRocks > use demo;
Database changed
StarRocks > create table bar(id int, name string) ;
Query OK, 0 rows affected (0.59 sec)

StarRocks >  insert into bar values (100, 'aaa');
Query OK, 1 row affected (7.04 sec)
{'label':'FAKE_ICEBERG_SINK_LABEL', 'status':'VISIBLE', 'txnId':'-1'}

StarRocks > select * from bar;
+------+------+
| id   | name |
+------+------+
|  100 | aaa  |
+------+------+
1 row in set (0.17 sec)

StarRocks >

You'll also see this in the logs.

iceberg-rest           | 2023-07-26T18:28:02.549 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table properties set at catalog level through catalog properties: {}
iceberg-rest           | 2023-07-26T18:28:02.553 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table properties enforced at catalog level through catalog properties: {}
iceberg-rest           | 2023-07-26T18:28:02.975 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Successfully committed to table demo.bar in 411 ms
iceberg-rest           | 2023-07-26T18:28:02.978 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/demo/bar/metadata/00000-bf4f3884-876b-42a1-ab57-f9d7da891927.metadata.json
allin1-ubuntu-iceberg  | [Wed Jul 26 18:28:05 UTC 2023] Alive: true
allin1-ubuntu-iceberg  | [Wed Jul 26 18:28:15 UTC 2023] Alive: true
iceberg-rest           | 2023-07-26T18:28:15.546 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/demo/bar/metadata/00000-bf4f3884-876b-42a1-ab57-f9d7da891927.metadata.json
iceberg-rest           | 2023-07-26T18:28:15.560 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.demo.bar
iceberg-rest           | 2023-07-26T18:28:20.707 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/demo/bar/metadata/00000-bf4f3884-876b-42a1-ab57-f9d7da891927.metadata.json
iceberg-rest           | 2023-07-26T18:28:20.716 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.demo.bar
iceberg-rest           | 2023-07-26T18:28:22.514 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/demo/bar/metadata/00000-bf4f3884-876b-42a1-ab57-f9d7da891927.metadata.json
iceberg-rest           | 2023-07-26T18:28:22.518 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.demo.bar
iceberg-rest           | 2023-07-26T18:28:22.534 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/demo/bar/metadata/00000-bf4f3884-876b-42a1-ab57-f9d7da891927.metadata.json
iceberg-rest           | 2023-07-26T18:28:22.538 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.demo.bar
iceberg-rest           | 2023-07-26T18:28:22.547 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Successfully committed to table demo.bar in 7 ms
iceberg-rest           | 2023-07-26T18:28:22.548 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/demo/bar/metadata/00001-f48972c2-c71d-438a-a027-29004acfe140.metadata.json

alanpaulkwan · 2023-08-09T06:50:50Z

alanpaulkwan
Aug 9, 2023

Could you provide the full docker yaml file?

1 reply

alberttwong Dec 1, 2023
Author

version: "3"

services:
  starrocks:
    image: registry.starrocks.io/starrocks/allin1-ubuntu:latest
    hostname: starrocks-fe
    container_name: allin1-ubuntu-iceberg
    ports:
      - 8030:8030
      - 8040:8040
      - 9030:9030
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1

  spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
        aliases:
          - iceberg-rest.minio
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
  minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"]
  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      "
networks:
  iceberg_net:

zeddit · 2023-11-13T04:23:29Z

zeddit
Nov 13, 2023

hi, does SR supports REST catalog? there is no documentation on the official site. great thanks.

1 reply

alberttwong Nov 13, 2023
Author

Yes. Iceberg REST is the default catalog integration. Look at the tabular section of https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog/

piobogiwo · 2023-11-28T13:13:58Z

piobogiwo
Nov 28, 2023

I have followed the recipe as above -combining tabulario set of dockers with additional starrrocks/allin1ubuntu docker.

I used notebook from spark-iceberg docker.
And nicely created tables in SQL and pySpark in various flavors : with pySpark writeAsTable() and SQL query with various options : USING iceberg/ STORED AS PARQUET/ STORED AS ICEBERG.
I am able to query all these tables. I see them in the file system too. Nice!

Subsequently I tried StarRocks :

mysql reported missing AWS_REGION so I have also replicated to starrocks section in docker-compose all environment variables that were defined for iceberg-rest docker (as defined by tabular-io). That was al from AWS_REGION..all the way through CATALOG_S3_ENDPOINT definitions (I added enviroment variables as adding aws region setting to "CREATE EXTERNAL CATALOG...." didn't work)
with these additional environment settings I can a) create an external catalog as proposed in the instruction above b) SET this iceberg catalog c) USE nyc database d) SHOW tables works too and lists all tables created in the notebook

...but when making SELECT on any of the tables created in and working in the notebook I get :

StarRocks > select * from taxis limit 100;

ERROR 1064 (HY000): BE access S3 file failed, SdkResponseCode=301, SdkErrorType=100, SdkErrorMessage=Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.: file = s3://warehouse/nyc/taxis/data/tpep_pickup_datetime_day=2022-02-25/00000-106-d2cd1139-8fee-4124-bbd2-b92f1fa23b6a-00005.parquet

I will greatly appreciate any suggestions.

4 replies

alberttwong Nov 28, 2023
Author

tabular.io set of dockers? you mean apache iceberg quick start. what version of the allin1 container are you using? execute current_version();

alberttwong Nov 28, 2023
Author

We have separate set of instructions for using tabular.io

alberttwong Nov 30, 2023
Author

Can you reach out to me via community starrocks slack.

alberttwong Dec 1, 2023
Author

#36287

alberttwong · 2023-12-02T19:09:33Z

alberttwong
Dec 2, 2023
Author

logs from a working docker-compose up with SR 3.1.5 and Iceberg 1.4.2

atwong@Albert-CelerData apacheiceberg % docker-compose up
[+] Building 0.0s (0/0)                                    docker:desktop-linux
[+] Running 6/0
 ✔ Network apacheiceberg_iceberg_net  Created                              0.0s
 ✔ Container minio                    Created                              0.0s
 ✔ Container allin1-ubuntu-iceberg    Created                              0.0s
 ✔ Container iceberg-rest             Created                              0.0s
 ✔ Container spark-iceberg            Created                              0.0s
 ✔ Container mc                       Created                              0.0s
Attaching to allin1-ubuntu-iceberg, iceberg-rest, mc, minio, spark-iceberg
allin1-ubuntu-iceberg  | ****************************************************************************************************
allin1-ubuntu-iceberg  | *   ________  _________  ________  ________  ________  ________  ________  ___  __    ________
allin1-ubuntu-iceberg  | *  |\   ____\|\___   ___\\   __  \|\   __  \|\   __  \|\   __  \|\   ____\|\  \|\  \ |\   ____\
allin1-ubuntu-iceberg  | *  \ \  \___|\|___ \  \_\ \  \|\  \ \  \|\  \ \  \|\  \ \  \|\  \ \  \___|\ \  \/  /|\ \  \___|_
allin1-ubuntu-iceberg  | *   \ \_____  \   \ \  \ \ \   __  \ \   _  _\ \   _  _\ \  \\\  \ \  \    \ \   ___  \ \_____  \
allin1-ubuntu-iceberg  | *    \|____|\  \   \ \  \ \ \  \ \  \ \  \\  \\ \  \\  \\ \  \\\  \ \  \____\ \  \\ \  \|____|\  \
allin1-ubuntu-iceberg  | *      ____\_\  \   \ \__\ \ \__\ \__\ \__\\ _\\ \__\\ _\\ \_______\ \_______\ \__\\ \__\____\_\  \
allin1-ubuntu-iceberg  | *     |\_________\   \|__|  \|__|\|__|\|__|\|__|\|__|\|__|\|_______|\|_______|\|__| \|__|\_________\
allin1-ubuntu-iceberg  | *     \|_________|                                                                      \|_________|
allin1-ubuntu-iceberg  | *
allin1-ubuntu-iceberg  | *                      ________  ___       ___       ___  ________     _____
allin1-ubuntu-iceberg  | *                     |\   __  \|\  \     |\  \     |\  \|\   ___  \  / __  \
allin1-ubuntu-iceberg  | *                     \ \  \|\  \ \  \    \ \  \    \ \  \ \  \\ \  \|\/_|\  \
allin1-ubuntu-iceberg  | *                      \ \   __  \ \  \    \ \  \    \ \  \ \  \\ \  \|/ \ \  \
allin1-ubuntu-iceberg  | *                       \ \  \ \  \ \  \____\ \  \____\ \  \ \  \\ \  \   \ \  \
allin1-ubuntu-iceberg  | *                        \ \__\ \__\ \_______\ \_______\ \__\ \__\\ \__\   \ \__\
allin1-ubuntu-iceberg  | *                         \|__|\|__|\|_______|\|_______|\|__|\|__| \|__|    \|__|
allin1-ubuntu-iceberg  | *
allin1-ubuntu-iceberg  | ****************************************************************************************************
allin1-ubuntu-iceberg  |
allin1-ubuntu-iceberg  | StarRocks [(Blazing Fast)]> _
allin1-ubuntu-iceberg  |
allin1-ubuntu-iceberg  |
iceberg-rest           | 2023-12-02T19:00:10.849 INFO  [org.apache.iceberg.rest.RESTCatalogServer] - Creating catalog with properties: {jdbc.password=password, s3.endpoint=http://minio:9000, jdbc.user=user, io-impl=org.apache.iceberg.aws.s3.S3FileIO, catalog-impl=org.apache.iceberg.jdbc.JdbcCatalog, warehouse=s3://warehouse/, uri=jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory}
iceberg-rest           | 2023-12-02T19:00:10.866 INFO  [org.apache.iceberg.CatalogUtil] - Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
minio                  | Formatting 1st pool, 1 set(s), 1 drives per set.
minio                  | WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
minio                  | MinIO Object Storage Server
minio                  | Copyright: 2015-2023 MinIO, Inc.
minio                  | License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
minio                  | Version: RELEASE.2023-11-20T22-40-07Z (go1.21.4 linux/arm64)
minio                  |
minio                  | Status:         1 Online, 0 Offline.
minio                  | S3-API: http://172.21.0.4:9000  http://127.0.0.1:9000
minio                  | Console: http://172.21.0.4:9001 http://127.0.0.1:9001
minio                  |
minio                  | Documentation: https://min.io/docs/minio/linux/index.html
minio                  | Warning: The standard parity is set to 0. This can lead to data loss.
mc                     | Added `minio` successfully.
iceberg-rest           | 2023-12-02T19:00:11.033 INFO  [org.eclipse.jetty.util.log] - Logging initialized @343ms to org.eclipse.jetty.util.log.Slf4jLog
mc                     | mc: <ERROR> Failed to remove `minio/warehouse` recursively. The specified bucket does not exist
spark-iceberg          | starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-1-3dd8d081c43f.out
iceberg-rest           | 2023-12-02T19:00:11.066 INFO  [org.eclipse.jetty.server.Server] - jetty-9.4.51.v20230217; built: 2023-02-17T08:19:37.309Z; git: b45c405e4544384de066f814ed42ae3dceacdd49; jvm 17.0.9+8-LTS
mc                     | Bucket created successfully `minio/warehouse`.
iceberg-rest           | 2023-12-02T19:00:11.077 INFO  [org.eclipse.jetty.server.handler.ContextHandler] - Started o.e.j.s.ServletContextHandler@5552768b{/,null,AVAILABLE}
iceberg-rest           | 2023-12-02T19:00:11.084 INFO  [org.eclipse.jetty.server.AbstractConnector] - Started ServerConnector@55b0dcab{HTTP/1.1, (http/1.1)}{0.0.0.0:8181}
iceberg-rest           | 2023-12-02T19:00:11.084 INFO  [org.eclipse.jetty.server.Server] - Started @394ms
mc                     | mc: Please use 'mc anonymous'
allin1-ubuntu-iceberg  | 2023-12-02 19:00:11,108 INFO Set uid to user 0 succeeded
allin1-ubuntu-iceberg  | 2023-12-02 19:00:11,110 INFO RPC interface 'supervisor' initialized
allin1-ubuntu-iceberg  | 2023-12-02 19:00:11,110 CRIT Server 'unix_http_server' running without any HTTP authentication checking
allin1-ubuntu-iceberg  | 2023-12-02 19:00:11,110 INFO supervisord started with pid 6
minio                  |
minio                  |  You are running an older version of MinIO released 1 week before the latest release
minio                  |  Update: Run `mc admin update`
minio                  |
minio                  |
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12,112 INFO spawned: 'beservice' with pid 524
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12,113 INFO spawned: 'broker' with pid 525
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12,115 INFO spawned: 'director' with pid 526
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12,116 INFO spawned: 'feproxy' with pid 528
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12,118 INFO spawned: 'feservice' with pid 532
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12+00:00 INFO checking if need to perform auto registring Backend and Broker ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12+00:00 INFO checking if FE service query port:9030 alive or not ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:12+00:00 WARN FE service query port:9030 is NOT alive yet!
spark-iceberg          | starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark--org.apache.spark.deploy.worker.Worker-1-3dd8d081c43f.out
allin1-ubuntu-iceberg  | 2023-12-02 19:00:14+00:00 WARN FE service query port:9030 is NOT alive yet!
spark-iceberg          | starting org.apache.spark.deploy.history.HistoryServer, logging to /opt/spark/logs/spark--org.apache.spark.deploy.history.HistoryServer-1-3dd8d081c43f.out
allin1-ubuntu-iceberg  | 2023-12-02 19:00:16+00:00 WARN FE service query port:9030 is NOT alive yet!
allin1-ubuntu-iceberg  | 2023-12-02 19:00:17,412 INFO success: beservice entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
allin1-ubuntu-iceberg  | 2023-12-02 19:00:17,412 INFO success: broker entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
allin1-ubuntu-iceberg  | 2023-12-02 19:00:17,412 INFO success: director entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
allin1-ubuntu-iceberg  | 2023-12-02 19:00:17,413 INFO success: feproxy entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
allin1-ubuntu-iceberg  | 2023-12-02 19:00:17,413 INFO success: feservice entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
allin1-ubuntu-iceberg  | 2023-12-02 19:00:18+00:00 WARN FE service query port:9030 is NOT alive yet!
spark-iceberg          | starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /opt/spark/logs/spark--org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-3dd8d081c43f.out
allin1-ubuntu-iceberg  | 2023-12-02 19:00:20+00:00 WARN FE service query port:9030 is NOT alive yet!
spark-iceberg          | [I 2023-12-02 19:00:21.605 ServerApp] Package notebook took 0.0000s to import
spark-iceberg          | [I 2023-12-02 19:00:21.613 ServerApp] Package jupysql_plugin took 0.0077s to import
spark-iceberg          | [I 2023-12-02 19:00:21.619 ServerApp] Package jupyter_lsp took 0.0050s to import
spark-iceberg          | [W 2023-12-02 19:00:21.619 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
spark-iceberg          | [I 2023-12-02 19:00:21.621 ServerApp] Package jupyter_server_terminals took 0.0021s to import
spark-iceberg          | [I 2023-12-02 19:00:21.621 ServerApp] Package jupyterlab took 0.0000s to import
spark-iceberg          | [I 2023-12-02 19:00:21.635 ServerApp] Package notebook_shim took 0.0000s to import
spark-iceberg          | [W 2023-12-02 19:00:21.635 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
spark-iceberg          | [I 2023-12-02 19:00:21.636 ServerApp] jupysql_plugin | extension was successfully linked.
spark-iceberg          | [I 2023-12-02 19:00:21.636 ServerApp] jupyter_lsp | extension was successfully linked.
spark-iceberg          | [I 2023-12-02 19:00:21.638 ServerApp] jupyter_server_terminals | extension was successfully linked.
spark-iceberg          | [W 2023-12-02 19:00:21.638 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
spark-iceberg          | [W 2023-12-02 19:00:21.638 LabApp] 'password' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
spark-iceberg          | [W 2023-12-02 19:00:21.640 ServerApp] ServerApp.token config is deprecated in 2.0. Use IdentityProvider.token.
spark-iceberg          | [I 2023-12-02 19:00:21.640 ServerApp] jupyterlab | extension was successfully linked.
spark-iceberg          | [I 2023-12-02 19:00:21.642 ServerApp] notebook | extension was successfully linked.
spark-iceberg          | [I 2023-12-02 19:00:21.642 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
spark-iceberg          | [I 2023-12-02 19:00:21.753 ServerApp] notebook_shim | extension was successfully linked.
spark-iceberg          | [W 2023-12-02 19:00:21.762 ServerApp] WARNING: The Jupyter server is listening on all IP addresses and not using encryption. This is not recommended.
spark-iceberg          | [W 2023-12-02 19:00:21.762 ServerApp] WARNING: The Jupyter server is listening on all IP addresses and not using authentication. This is highly insecure and not recommended.
spark-iceberg          | [I 2023-12-02 19:00:21.763 ServerApp] notebook_shim | extension was successfully loaded.
spark-iceberg          | [I 2023-12-02 19:00:21.763 ServerApp] Registered jupysql-plugin server extension
spark-iceberg          | [I 2023-12-02 19:00:21.763 ServerApp] jupysql_plugin | extension was successfully loaded.
spark-iceberg          | [I 2023-12-02 19:00:21.764 ServerApp] jupyter_lsp | extension was successfully loaded.
spark-iceberg          | [I 2023-12-02 19:00:21.765 ServerApp] jupyter_server_terminals | extension was successfully loaded.
spark-iceberg          | [I 2023-12-02 19:00:21.766 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.9/site-packages/jupyterlab
spark-iceberg          | [I 2023-12-02 19:00:21.766 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
spark-iceberg          | [I 2023-12-02 19:00:21.766 LabApp] Extension Manager is 'pypi'.
spark-iceberg          | [I 2023-12-02 19:00:21.767 ServerApp] jupyterlab | extension was successfully loaded.
spark-iceberg          | [I 2023-12-02 19:00:21.768 ServerApp] notebook | extension was successfully loaded.
spark-iceberg          | [I 2023-12-02 19:00:21.769 ServerApp] Serving notebooks from local directory: /home/iceberg/notebooks
spark-iceberg          | [I 2023-12-02 19:00:21.769 ServerApp] Jupyter Server 2.10.0 is running at:
spark-iceberg          | [I 2023-12-02 19:00:21.769 ServerApp] http://localhost:8888/tree
spark-iceberg          | [I 2023-12-02 19:00:21.769 ServerApp]     http://127.0.0.1:8888/tree
spark-iceberg          | [I 2023-12-02 19:00:21.769 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
spark-iceberg          | [I 2023-12-02 19:00:21.779 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
allin1-ubuntu-iceberg  | 2023-12-02 19:00:22+00:00 INFO FE service query port:9030 is alive!
allin1-ubuntu-iceberg  | 2023-12-02 19:00:22+00:00 INFO generate my.cnf file ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:22+00:00 INFO check if need to add BE into FE service ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO Add BE(127.0.0.1:9050) into FE service ...
allin1-ubuntu-iceberg  |
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO BE service already added into FE service ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO check if need to add BROKER into FE service ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO Add BROKER(127.0.0.1:8000) into FE service ...
allin1-ubuntu-iceberg  |
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO broker service already added into FE service ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO cluster initialization DONE!
allin1-ubuntu-iceberg  | 2023-12-02 19:00:23+00:00 INFO wait a few seconds for BE and Broker's heartbeat ...
allin1-ubuntu-iceberg  | 2023-12-02 19:00:33+00:00 INFO StarRocks Cluster information details:
allin1-ubuntu-iceberg  | *************************** 1. row ***************************
allin1-ubuntu-iceberg  |              Name: 127.0.0.1_9010_1701543613413
allin1-ubuntu-iceberg  |                IP: 127.0.0.1
allin1-ubuntu-iceberg  |       EditLogPort: 9010
allin1-ubuntu-iceberg  |          HttpPort: 8030
allin1-ubuntu-iceberg  |         QueryPort: 9030
allin1-ubuntu-iceberg  |           RpcPort: 9020
allin1-ubuntu-iceberg  |              Role: LEADER
allin1-ubuntu-iceberg  |         ClusterId: 930465537
allin1-ubuntu-iceberg  |              Join: true
allin1-ubuntu-iceberg  |             Alive: true
allin1-ubuntu-iceberg  | ReplayedJournalId: 16
allin1-ubuntu-iceberg  |     LastHeartbeat: 2023-12-02 19:00:29
allin1-ubuntu-iceberg  |          IsHelper: true
allin1-ubuntu-iceberg  |            ErrMsg:
allin1-ubuntu-iceberg  |         StartTime: 2023-12-02 19:00:21
allin1-ubuntu-iceberg  |           Version: 3.1.5-5d8438a
allin1-ubuntu-iceberg  | *************************** 1. row ***************************
allin1-ubuntu-iceberg  |             BackendId: 10004
allin1-ubuntu-iceberg  |                    IP: 127.0.0.1
allin1-ubuntu-iceberg  |         HeartbeatPort: 9050
allin1-ubuntu-iceberg  |                BePort: 9060
allin1-ubuntu-iceberg  |              HttpPort: 8040
allin1-ubuntu-iceberg  |              BrpcPort: 8060
allin1-ubuntu-iceberg  |         LastStartTime: 2023-12-02 19:00:24
allin1-ubuntu-iceberg  |         LastHeartbeat: 2023-12-02 19:00:29
allin1-ubuntu-iceberg  |                 Alive: true
allin1-ubuntu-iceberg  |  SystemDecommissioned: false
allin1-ubuntu-iceberg  | ClusterDecommissioned: false
allin1-ubuntu-iceberg  |             TabletNum: 0
allin1-ubuntu-iceberg  |      DataUsedCapacity: 0.000 B
allin1-ubuntu-iceberg  |         AvailCapacity: 47.007 GB
allin1-ubuntu-iceberg  |         TotalCapacity: 117.028 GB
allin1-ubuntu-iceberg  |               UsedPct: 59.83 %
allin1-ubuntu-iceberg  |        MaxDiskUsedPct: 59.83 %
allin1-ubuntu-iceberg  |                ErrMsg:
allin1-ubuntu-iceberg  |               Version: 3.1.5-5d8438a
allin1-ubuntu-iceberg  |                Status: {"lastSuccessReportTabletsTime":"2023-12-02 19:00:25"}
allin1-ubuntu-iceberg  |     DataTotalCapacity: 47.007 GB
allin1-ubuntu-iceberg  |           DataUsedPct: 0.00 %
allin1-ubuntu-iceberg  |              CpuCores: 10
allin1-ubuntu-iceberg  |     NumRunningQueries: 0
allin1-ubuntu-iceberg  |            MemUsedPct: 0.10 %
allin1-ubuntu-iceberg  |            CpuUsedPct: 0.0 %
allin1-ubuntu-iceberg  | *************************** 1. row ***************************
allin1-ubuntu-iceberg  |           Name: allin1broker
allin1-ubuntu-iceberg  |             IP: 127.0.0.1
allin1-ubuntu-iceberg  |           Port: 8000
allin1-ubuntu-iceberg  |          Alive: true
allin1-ubuntu-iceberg  |  LastStartTime: 2023-12-02 19:00:24
allin1-ubuntu-iceberg  | LastUpdateTime: 2023-12-02 19:00:29
allin1-ubuntu-iceberg  |         ErrMsg:
allin1-ubuntu-iceberg  | 2023-12-02 19:00:34+00:00 INFO
allin1-ubuntu-iceberg  | 2023-12-02 19:00:34+00:00 INFO
allin1-ubuntu-iceberg  | 2023-12-02 19:00:34+00:00 INFO  * IMPORTANT NOTICE!
allin1-ubuntu-iceberg  |
allin1-ubuntu-iceberg  | If FE/BE state needs to be persisted, please be sure the following directories are mounted:
allin1-ubuntu-iceberg  | * FE service meta: /data/deploy/starrocks/fe/meta
allin1-ubuntu-iceberg  | * BE service storage: /data/deploy/starrocks/be/storage
allin1-ubuntu-iceberg  |
allin1-ubuntu-iceberg  | 2023-12-02 19:00:34+00:00 INFO
allin1-ubuntu-iceberg  | 2023-12-02 19:00:35+00:00 INFO FE mysql query port: 9030
allin1-ubuntu-iceberg  | 2023-12-02 19:00:35+00:00 INFO FE http service port: 8080
allin1-ubuntu-iceberg  | 2023-12-02 19:00:35+00:00 INFO
allin1-ubuntu-iceberg  | 2023-12-02 19:00:35+00:00 INFO Enjoy the journal to StarRocks blazing-fast lake-house engine!
iceberg-rest           | 2023-12-02T19:00:49.697 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] - Error processing REST request
iceberg-rest           | org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=404, type=NoSuchTableException, message=Table does not exist: nyc.taxis)
iceberg-rest           | org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist: nyc.taxis
iceberg-rest           | 	at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:53)
iceberg-rest           | 	at org.apache.iceberg.rest.CatalogHandlers.loadTable(CatalogHandlers.java:257)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:351)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:444)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogServlet.doGet(RESTCatalogServlet.java:66)
iceberg-rest           | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
iceberg-rest           | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
iceberg-rest           | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
iceberg-rest           | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
iceberg-rest           | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
iceberg-rest           | 	at org.eclipse.jetty.server.Server.handle(Server.java:516)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
iceberg-rest           | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
iceberg-rest           | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
iceberg-rest           | 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
iceberg-rest           | 	at java.base/java.lang.Thread.run(Thread.java:840)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:461)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
iceberg-rest           | 	at org.apache.iceberg.rest.RESTCatalogServlet.doGet(RESTCatalogServlet.java:66)
iceberg-rest           | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
iceberg-rest           | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
iceberg-rest           | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
iceberg-rest           | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
iceberg-rest           | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
iceberg-rest           | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
iceberg-rest           | 	at org.eclipse.jetty.server.Server.handle(Server.java:516)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
iceberg-rest           | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
iceberg-rest           | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
iceberg-rest           | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
iceberg-rest           | 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
iceberg-rest           | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
iceberg-rest           | 	at java.base/java.lang.Thread.run(Thread.java:840)
iceberg-rest           | 2023-12-02T19:00:49.841 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table properties set at catalog level through catalog properties: {}
iceberg-rest           | 2023-12-02T19:00:49.842 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table properties enforced at catalog level through catalog properties: {}
iceberg-rest           | 2023-12-02T19:00:50.172 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Successfully committed to table nyc.taxis in 324 ms
iceberg-rest           | 2023-12-02T19:00:50.175 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00000-3e6e26e9-273d-4cac-a5ab-80a80f3888df.metadata.json
iceberg-rest           | 2023-12-02T19:00:59.105 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00000-3e6e26e9-273d-4cac-a5ab-80a80f3888df.metadata.json
iceberg-rest           | 2023-12-02T19:00:59.110 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis
iceberg-rest           | 2023-12-02T19:00:59.220 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00000-3e6e26e9-273d-4cac-a5ab-80a80f3888df.metadata.json
iceberg-rest           | 2023-12-02T19:00:59.224 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis
iceberg-rest           | 2023-12-02T19:00:59.236 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Successfully committed to table nyc.taxis in 11 ms
iceberg-rest           | 2023-12-02T19:00:59.236 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00001-731cb938-c844-4be3-8d2b-f4e33d1e5a28.metadata.json
iceberg-rest           | 2023-12-02T19:00:59.242 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00001-731cb938-c844-4be3-8d2b-f4e33d1e5a28.metadata.json
iceberg-rest           | 2023-12-02T19:00:59.244 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis
iceberg-rest           | 2023-12-02T19:00:59.328 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00001-731cb938-c844-4be3-8d2b-f4e33d1e5a28.metadata.json
iceberg-rest           | 2023-12-02T19:00:59.332 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis
iceberg-rest           | 2023-12-02T19:03:24.199 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00001-731cb938-c844-4be3-8d2b-f4e33d1e5a28.metadata.json
iceberg-rest           | 2023-12-02T19:03:24.203 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis
iceberg-rest           | 2023-12-02T19:03:24.242 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00001-731cb938-c844-4be3-8d2b-f4e33d1e5a28.metadata.json
iceberg-rest           | 2023-12-02T19:03:24.246 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis
iceberg-rest           | 2023-12-02T19:03:25.023 INFO  [org.apache.iceberg.BaseMetastoreTableOperations] - Refreshing table metadata from new version: s3://warehouse/nyc/taxis/metadata/00001-731cb938-c844-4be3-8d2b-f4e33d1e5a28.metadata.json
iceberg-rest           | 2023-12-02T19:03:25.035 INFO  [org.apache.iceberg.BaseMetastoreCatalog] - Table loaded by catalog: rest_backend.nyc.taxis

0 replies

andreale28 · 2023-12-18T19:02:11Z

andreale28
Dec 18, 2023

Hi @alberttwong, thanks for this tutorial. Since Starrocks supports shared-data architecture which has FE and CN nodes, do we have any tutorials on deploying a shared-data Starrocks and querying Iceberg/Delta/Hudi tables?

3 replies

alberttwong Dec 19, 2023
Author

They are 2 different items. Shared nothing and shared data architecture does not have dependency on iceberg/hudi external catalog. You always have to have a internal catalog (so you pick shared nothing or shared data) and then optionally, can add additional external catalogs like iceberg/hudi.

Using this tutorial, you can use this external catalog with shared data or shared nothing architecture.

andreale28 Dec 19, 2023

That's great. Quick question, I am not too familiar with Docker , how do I add new configuration (for example, aws_s3_endpoint) to fe.conf file in starrocks/allin1-ubuntu image?
I try to add command option in docker-compose file but it does not work.

  starrocks:
    image: starrocks/allin1-ubuntu
    hostname: starrocks-fe
    container_name: allin1-ubuntu
    depends_on:
      - createbuckets
      - metadata
      - hive-metastore
    ports:
      - "8030:8030"
      - "8040:8040"
      - "9030:9030"
    command:
      - /bin/bash
      - -c
      - |
        echo "run_mode=shared_data" >> /data/deploy/starrocks/fe/conf/fe.conf
        echo "aws_s3_endpoint=http://minio:9000" >> /data/deploy/starrocks/fe/conf/fe.conf 
        echo "aws3_s3_region=us-east-1" >> /data/deploy/starrocks/fe/conf/fe.conf 
        echo "aws_s3_access_key=minio" >> /data/deploy/starrocks/fe/conf/fe.conf 
        echo "aws_s3_secret_key=minio123" >> /data/deploy/starrocks/fe/conf/fe.conf 
        /data/deploy/starrocks/fe/bin/start_fe.sh
    networks:
      - spark-network
    environment:
      - STARROCKS_ADMIN_USER=sdeuser
      - STARROCKS_ADMIN_PASSWORD=sdepassword
      - AWS_ACCESS_KEY_ID=minio
      - AWS_SECRET_ACCESS_KEY=minio123
      - AWS_REGION=us-east-1

alberttwong Dec 20, 2023
Author

so go through the quickstart to use shared data architecture and then run starrocks and then type of the sql statements to add an external iceberg catalog.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to query data and write data in Apache Iceberg using StarRocks #23427

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 9 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

How to query data and write data in Apache Iceberg using StarRocks #23427

alberttwong May 15, 2023

Prerequisites

Have a MySQL client

A StarRocks or CelerData database cluster

An Apache Iceberg environment

Configure Apache Iceberg

Create an external catalog and query the data

Write data into the iceberg tables backed by Minio

Replies: 5 comments · 9 replies

alanpaulkwan Aug 9, 2023

alberttwong Dec 1, 2023 Author

zeddit Nov 13, 2023

alberttwong Nov 13, 2023 Author

piobogiwo Nov 28, 2023

alberttwong Nov 28, 2023 Author

alberttwong Nov 28, 2023 Author

alberttwong Nov 30, 2023 Author

alberttwong Dec 1, 2023 Author

alberttwong Dec 2, 2023 Author

andreale28 Dec 18, 2023

alberttwong Dec 19, 2023 Author

andreale28 Dec 19, 2023

alberttwong Dec 20, 2023 Author

alberttwong
May 15, 2023

Replies: 5 comments 9 replies

alanpaulkwan
Aug 9, 2023

alberttwong Dec 1, 2023
Author

zeddit
Nov 13, 2023

alberttwong Nov 13, 2023
Author

piobogiwo
Nov 28, 2023

alberttwong Nov 28, 2023
Author

alberttwong Nov 28, 2023
Author

alberttwong Nov 30, 2023
Author

alberttwong Dec 1, 2023
Author

alberttwong
Dec 2, 2023
Author

andreale28
Dec 18, 2023

alberttwong Dec 19, 2023
Author

alberttwong Dec 20, 2023
Author