[Bug] In disaggregated mode, insert is failing

[Nj-kol]

Search before asking

• I had searched in the issues and found no similar issues.

Version

3.0.5

What's Wrong?

When running doris on disaggregated-cluster mode, insert operation fails. In logs, the starting of the error seems to be :

RuntimeLogger F20250601 12:38:10.633809   877 storage_engine.cpp:124] Check failed: _type == Type::CLOUD ( vs. )

Here is the extended stacktrace:

RuntimeLogger I20250601 12:38:04.119288   740 storage_engine.cpp:431] get root path info cost: 0 ms. tablet counter: 0
RuntimeLogger I20250601 12:38:10.627696   875 fragment_mgr.cpp:703] query_id: 4785b3a2d2354fb3-b79d0bbf562f29b9, coord_addr: TNetworkAddress(hostname=doris-disaggregated-cluster-fe-0.doris-disaggregated-cluster-fe-internal.doris.svc.cluster.local, port=9020), total fragment num on current host: 2, fe process uuid: 1748781044038, query type: LOAD, report audit fe:TNetworkAddress(hostname=doris-disaggregated-cluster-fe-0.doris-disaggregated-cluster-fe-internal.doris.svc.cluster.local, port=9020), use wg:TG[id = 1, name = normal, cpu_share = 1024, memory_limit = 1.08 GB, enable_memory_overcommit = true, version = 0, cpu_hard_limit = -1, scan_thread_num = 64, max_remote_scan_thread_num = 512, min_remote_scan_thread_num = 8, spill_low_watermark=50, spill_high_watermark=80, is_shutdown=false, query_num=0, read_bytes_per_second=-1, remote_read_bytes_per_second=-1]
RuntimeLogger I20250601 12:38:10.632476   876 fragment_mgr.cpp:634] Query 4785b3a2d2354fb3-b79d0bbf562f29b9 start execution
RuntimeLogger I20250601 12:38:10.632651   359 vtablet_writer.cpp:129] init new node for instance 0, incremantal:0
RuntimeLogger I20250601 12:38:10.632721   359 dns_cache.cpp:61] update hostname doris-disaggregated-cluster-cg1-0.doris-disaggregated-cluster-cg1.doris.svc.cluster.local's ip to 10.244.0.242
RuntimeLogger F20250601 12:38:10.633809   877 storage_engine.cpp:124] Check failed: _type == Type::CLOUD ( vs. )
*** Check failure stack trace: ***
    @     0x568b7684d036  google::LogMessage::SendToLog()
    @     0x568b76849a80  google::LogMessage::Flush()
RuntimeLogger I20250601 12:38:10.646747  1963 workload_group_manager.cpp:201]
Process Memory Summary: process memory used 796.32 MB(= 886.80 MB[vm/rss] - 90.48 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 2.84 GB(= 2.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 320.00 B, weighted_memory_limit_ratio: 0.7839843171114707
    @     0x568b7684d879  google::LogMessageFatal::~LogMessageFatal()
    @     0x568b6bdea89a  doris::BaseStorageEngine::to_cloud()
    @     0x568b6c084433  doris::LoadChannel::open()
    @     0x568b6c07ee9a  doris::LoadChannelMgr::open()
    @     0x568b6c1e931d  std::_Function_handler<>::_M_invoke()
    @     0x568b6c20427b  doris::WorkThreadPool<>::work_thread()
    @     0x568b79827370  execute_native_thread_routine
    @     0x7cfdcab6aac3  (unknown)
    @     0x7cfdcabfc850  (unknown)
    @              (nil)  (unknown)
*** Query id: 4785b3a2d2354fb3-b79d0bbf562f29b9 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748781490 (unix time) try "date -d @1748781490" if you are using GNU date ***
*** Current BE git commitID: e277cfb83f ***
*** SIGABRT unknown detail explain (@0xa1) received by PID 161 (TID 877 OR 0x7cfc353ab640) from PID 161; stack trace: ***
RuntimeLogger I20250601 12:38:11.282454  1963 daemon.cpp:227] os physical memory 4.00 GB. process memory used 1.09 GB(= 1.18 GB[vm/rss] - 90.69 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 3.60 GB, soft limit 3.24 GB. sys available memory 2.75 GB(= 2.75 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 204.80 MB, warning water mark 409.60 MB.
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421]
 

What You Expected?

Successful able to insert data

insert into mytable values
(1,0.14,'a1',20),
(2,1.04,'b2',21),
(3,3.14,'c3',22),
(4,4.35,'d4',23);

How to Reproduce?

The deployment spec is exactly same as mentioned on https://doris.apache.org/docs/3.0/install/deploy-on-kubernetes/separating-storage-compute/install-doris-cluster

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Nj-kol]

Would really appreciate if there are updates on this

[XLPE]

Both fe.conf and be.conf need to be configured deploy_mode = cloud

[Nj-kol]

Both fe.conf and be.conf need to be configured deploy_mode = cloud

Hey @XLPE thank you so much for your reply, I figured it out eventually. But I feel this should have been documented in the page https://doris.apache.org/docs/3.0/install/deploy-on-kubernetes/separating-storage-compute/config-fe

The way I resolved it is via the configmap, it would be great if you could include this in the page

apiVersion: v1
kind: ConfigMap
metadata:
  name: fe-configmap
  labels:
    app.kubernetes.io/component: fe
data:
  fe.conf: |
    CUR_DATE=`date +%Y%m%d-%H%M%S`
    # Log dir
    LOG_DIR = ${DORIS_HOME}/log
    # For jdk 17, this JAVA_OPTS will be used as default JVM options
    JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR -Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens java.base/jdk.internal.ref=ALL-UNNAMED"
    # INFO, WARN, ERROR, FATAL
    sys_log_level = INFO
    # NORMAL, BRIEF, ASYNC
    sys_log_mode = NORMAL
    # Default dirs to put jdbc drivers,default value is ${DORIS_HOME}/jdbc_drivers
    # jdbc_drivers_dir = ${DORIS_HOME}/jdbc_drivers
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    enable_fqdn_mode=true
    deploy_mode = cloud