can't create the container from docker-compose-with-s3.yml

[zhifeng110]

when i create the container, i find that some error logs in container.
the meta-node-0 was the first dead container, could you tell me how to solve the probm？

launching `meta`
  2023-07-17T07:30:25.196169525Z  INFO risingwave_meta: Starting meta node
    at src/meta/src/lib.rs:210

  2023-07-17T07:30:25.196319155Z  INFO risingwave_meta: > options: MetaNodeOpts { vpc_id: None, security_group_id: None, listen_addr: "0.0.0.0:5690", advertise_addr: "meta-node-0:5690", dashboard_host: Some("0.0.0.0:5691"), prometheus_host: Some("0.0.0.0:1250"), etcd_endpoints: "etcd-0:2388", etcd_auth: false, etcd_username: "", etcd_password: "", dashboard_ui_path: Some("/risingwave/ui"), prometheus_endpoint: None, connector_rpc_endpoint: Some("connector-node:50051"), privatelink_endpoint_default_tags: None, config_path: "/risingwave.toml", override_opts: OverrideConfigOpts { backend: Some(Etcd), barrier_interval_ms: None, sstable_size_mb: None, block_size_kb: None, bloom_false_positive: None, state_store: Some("hummock+s3://aicl2"), data_directory: Some("hummock_001"), do_not_config_object_storage_lifecycle: None, backup_storage_url: None, backup_storage_directory: None, object_store_streaming_read_timeout_ms: None, object_store_streaming_upload_timeout_ms: None, object_store_upload_timeout_ms: None, object_store_read_timeout_ms: None } }
    at src/meta/src/lib.rs:211

  2023-07-17T07:30:25.196465195Z  INFO risingwave_meta: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: 0, telemetry_enabled: true, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 604800, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, hummock_version_checkpoint_interval_sec: 30, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 300, disable_recovery: false, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Etcd, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_split_compact_group_interval_sec: 180, max_compactor_task_multiplier: 2, move_table_size_limit: 4294967296, split_group_size_limit: 68719476736, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 64, table_write_throughput_threshold: 134217728, min_table_split_write_throughput: 33554432, compaction_task_max_heartbeat_interval_secs: 60 }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024 }, distributed_query_limit: None, enable_barrier_read: true, unrecognized: {} }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: None, async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, dml_channel_initial_permits: 32768 }, unique_user_stream_errors: 10, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, imm_merge_threshold: 4, write_conflict_detection_enabled: false, block_cache_capacity_mb: None, high_priority_ratio_in_percent: None, meta_cache_capacity_mb: None, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, total_buffer_capacity_mb: None, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 10, object_store_streaming_read_timeout_ms: 600000, object_store_streaming_upload_timeout_ms: 600000, object_store_upload_timeout_ms: 3600000, object_store_read_timeout_ms: 3600000, unrecognized: {} }, unrecognized: {} }
    at src/meta/src/lib.rs:213

  2023-07-17T07:30:25.196533921Z  INFO risingwave_meta: > version: 1.0.0-alpha (87637e861e85bc57543b72af2ba4c2f0ea96dd3e)
    at src/meta/src/lib.rs:214

  2023-07-17T07:30:25.196550428Z  INFO risingwave_meta: Meta server listening at 0.0.0.0:5690
    at src/meta/src/lib.rs:249

  2023-07-17T07:30:25.196994875Z  INFO risingwave_meta::rpc::election_client: client meta-node-0:5690 start election
    at src/meta/src/rpc/election_client.rs:79

  2023-07-17T07:30:26.876522686Z  INFO risingwave_meta::rpc::election_client: use lease id 6036944882767452678
    at src/meta/src/rpc/election_client.rs:109

  2023-07-17T07:30:26.878456711Z  INFO risingwave_meta::rpc::election_client: no restored leader, campaigning
    at src/meta/src/rpc/election_client.rs:217

  2023-07-17T07:30:26.881208203Z  INFO risingwave_meta::rpc::election_client: client meta-node-0:5690 wins election __meta_election_
    at src/meta/src/rpc/election_client.rs:239

  2023-07-17T07:30:26.881289358Z  INFO risingwave_meta::rpc::server: Shutting down follower services
    at src/meta/src/rpc/server.rs:312

  2023-07-17T07:30:26.881475708Z  INFO risingwave_meta::rpc::server: Defining leader services
    at src/meta/src/rpc/server.rs:337

  2023-07-17T07:30:26.905723384Z  WARN risingwave_meta::manager::system_param: The initializing value of ""state_store"" (hummock+s3://aicl2) differ from persisted (hummock+minio://hummockadmin:hummockadmin@minio-0:9301/hummock001), using persisted value
    at src/meta/src/manager/system_param/mod.rs:190

  2023-07-17T07:30:26.905777212Z  INFO risingwave_meta::manager::system_param: system parameters: SystemParams { barrier_interval_ms: Some(1000), checkpoint_frequency: Some(1), sstable_size_mb: Some(256), block_size_kb: Some(64), bloom_false_positive: Some(0.001), state_store: Some("hummock+minio://hummockadmin:hummockadmin@minio-0:9301/hummock001"), data_directory: Some("hummock_001"), backup_storage_url: Some("memory"), backup_storage_directory: Some("backup"), telemetry_enabled: Some(true) }
    at src/meta/src/manager/system_param/mod.rs:66

thread 'risingwave-main' panicked at 'called `Result::unwrap()` on an `Err` value: ObjectStore(dispatch failure
  backtrace of `ObjectError`:
   0: <risingwave_object_store::object::error::ObjectError as core::convert::From<risingwave_object_store::object::error::ObjectErrorInner>>::from
             at ./risingwave/src/object_store/src/object/error.rs:46:10
   1: <T as core::convert::Into<U>>::into
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/convert/mod.rs:717:9
   2: <risingwave_object_store::object::error::ObjectError as core::convert::From<aws_smithy_http::result::SdkError<E>>>::from
             at ./risingwave/src/object_store/src/object/error.rs:114:40
   3: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/result.rs:1961:27
   4: <risingwave_object_store::object::s3::S3ObjectStore as risingwave_object_store::object::ObjectStore>::list::{{closure}}
             at ./risingwave/src/object_store/src/object/s3.rs:537:26
   5: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
   6: risingwave_object_store::object::MonitoredObjectStore<OS>::list::{{closure}}::{{closure}}
             at ./risingwave/src/object_store/src/object/mod.rs:825:18
   7: risingwave_object_store::object::MonitoredObjectStore<OS>::list::{{closure}}
             at ./risingwave/src/object_store/src/object/mod.rs:828:28
   8: risingwave_object_store::object::ObjectStoreImpl::list::{{closure}}
             at ./risingwave/src/object_store/src/object/mod.rs:251:9
   9: risingwave_meta::hummock::manager::checkpoint::<impl risingwave_meta::hummock::manager::HummockManager<S>>::read_checkpoint::{{closure}}
             at ./risingwave/src/meta/src/hummock/manager/checkpoint.rs:49:14
  10: risingwave_meta::hummock::manager::HummockManager<S>::load_meta_store_state_impl::{{closure}}
             at ./risingwave/src/meta/src/hummock/manager/mod.rs:435:66
  11: risingwave_meta::hummock::manager::HummockManager<S>::load_meta_store_state::{{closure}}
             at ./risingwave/src/meta/src/hummock/manager/mod.rs:371:10
  12: risingwave_meta::hummock::manager::HummockManager<S>::new_impl::{{closure}}
             at ./risingwave/src/meta/src/hummock/manager/mod.rs:354:42
  13: risingwave_meta::hummock::manager::HummockManager<S>::new::{{closure}}
             at ./risingwave/src/meta/src/hummock/manager/mod.rs:256:10
  14: risingwave_meta::rpc::server::start_service_as_election_leader::{{closure}}
             at ./risingwave/src/meta/src/rpc/server.rs:388:6
  15: risingwave_meta::rpc::server::rpc_serve_with_store::{{closure}}::{{closure}}
             at ./risingwave/src/meta/src/rpc/server.rs:275:10
  16: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9
  22: std::panicking::try::do_call
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40
  23: std::panicking::try
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19
  24: std::panic::catch_unwind
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14
  25: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18
  26: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27
  27: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15
  28: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18
  29: tokio::runtime::task::LocalNotified<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:394:9
  30: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:464:18
  31: tokio::runtime::coop::with_budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:107:5
  32: tokio::runtime::coop::budget
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:73:5
  33: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  36: tokio::macros::scoped_tls::ScopedKey<T>::set
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/macros/scoped_tls.rs:61:9
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/task.rs:42:21
  40: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  41: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17
  42: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9
  43: tokio::runtime::task::core::Core<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13
  44: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19
  45: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9
  46: std::panicking::try::do_call
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40
  47: std::panicking::try
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19
  48: std::panic::catch_unwind
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14
  49: tokio::runtime::task::harness::poll_future
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15
  52: tokio::runtime::task::raw::RawTask::poll
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18
  53: tokio::runtime::task::UnownedTask<S>::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:431:9
  54: tokio::runtime::blocking::pool::Task::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:159:9
  55: tokio::runtime::blocking::pool::Inner::run
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:513:17
  56: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:471:13
  57: std::sys_common::backtrace::__rust_begin_short_backtrace
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:135:18
  58: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/thread/mod.rs:529:17
  59: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9
  60: std::panicking::try::do_call
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40
  61: std::panicking::try
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19
  62: std::panic::catch_unwind
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14
  63: std::thread::Builder::spawn_unchecked_::{{closure}}
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/thread/mod.rs:528:30
  64: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/ops/function.rs:250:5
  65: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1985:9
  66: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1985:9
  67: std::sys::unix::thread::Thread::new::thread_start
             at ./rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys/unix/thread.rs:108:17
  68: start_thread
             at ./nptl/pthread_create.c:442:8
  69: clone
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100
)', src/meta/src/rpc/server.rs:389:6
stack backtrace:
   0:     0x55fb2612337c - std::backtrace_rs::backtrace::libunwind::trace::hc317256c0daecb22
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55fb2612337c - std::backtrace_rs::backtrace::trace_unsynchronized::h85008105ccd95ba6
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55fb2612337c - std::sys_common::backtrace::_print_fmt::hb6e7cbe31fecc91c
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x55fb2612337c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h53097d3751045778
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55fb2018020f - core::fmt::rt::Argument::fmt::hdbe0b5dfb2c09e86
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/fmt/rt.rs:138:9
   5:     0x55fb2018020f - core::fmt::write::h4ae17c68778671a0
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/fmt/mod.rs:1094:21
   6:     0x55fb2611e27f - std::io::Write::write_fmt::hfd7c55e775c6dc60
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/io/mod.rs:1713:15
   7:     0x55fb26123185 - std::sys_common::backtrace::_print::h81eabb0036b692a0
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55fb26123185 - std::sys_common::backtrace::print::h1b1f4d1a23fac3fd
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x55fb261246f7 - std::panicking::default_hook::{{closure}}::hc17d2bd8f3da442c
  10:     0x55fb26124504 - std::panicking::default_hook::h3a32a9336113ba96
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:288:9
  11:     0x55fb25a04f7b - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h72b22edfea82ceb7
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1999:9
  12:     0x55fb25a04f7b - risingwave_rt::set_panic_hook::{{closure}}::h8e3a27eb23e52d94
                               at /risingwave/src/utils/runtime/src/lib.rs:131:9
  13:     0x55fb25a04f7b - std::panicking::update_hook::{{closure}}::h3714e8cc9549381b
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:234:47
  14:     0x55fb26124d12 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h4413820818220cb9
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1999:9
  15:     0x55fb26124d12 - std::panicking::rust_panic_with_hook::h90e4d15277396259
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:709:13
  16:     0x55fb26124ab4 - std::panicking::begin_panic_handler::{{closure}}::h2b743ea215b7c408
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:597:13
  17:     0x55fb26123756 - std::sys_common::backtrace::__rust_end_short_backtrace::h895d390908a90650
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:151:18
  18:     0x55fb26124842 - rust_begin_unwind
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:593:5
  19:     0x55fb2017ce33 - core::panicking::panic_fmt::h1cc8517ae3119d0e
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panicking.rs:67:14
  20:     0x55fb2017d2e3 - core::result::unwrap_failed::h4c9088fd4642a5e0
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/result.rs:1651:5
  21:     0x55fb2543a723 - core::result::Result<T,E>::unwrap::h9d5faaf98d7a10af
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/result.rs:1076:23
  22:     0x55fb2543a723 - risingwave_meta::rpc::server::start_service_as_election_leader::{{closure}}::he75fdcf1ee6e50fd
                               at /risingwave/src/meta/src/rpc/server.rs:381:27
  23:     0x55fb252ae7ab - risingwave_meta::rpc::server::rpc_serve_with_store::{{closure}}::{{closure}}::hf10706f1540909b4
                               at /risingwave/src/meta/src/rpc/server.rs:275:10
  24:     0x55fb252ae7ab - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h3293fae9a1ebd57e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  25:     0x55fb24dd469a - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h2ab52c5bba80f550
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17
  26:     0x55fb24dd469a - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h4f862b2d53f30011
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9
  27:     0x55fb247b1ec4 - tokio::runtime::task::core::Core<T,S>::poll::h6974f0dea2e19f3f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13
  28:     0x55fb246ab7cb - tokio::runtime::task::harness::poll_future::{{closure}}::ha80ed7b372bbe8bc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19
  29:     0x55fb246ab7cb - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdbcbc139365c8e19
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9
  30:     0x55fb246ab7cb - std::panicking::try::do_call::h19f3ac65ed3e8724
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40
  31:     0x55fb246ab7cb - std::panicking::try::h3b9cd8b73d0f4199
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19
  32:     0x55fb246ab7cb - std::panic::catch_unwind::h123dff4db0660b4e
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14
  33:     0x55fb246ab7cb - tokio::runtime::task::harness::poll_future::h310d55f54a8b9159
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18
  34:     0x55fb246ab7cb - tokio::runtime::task::harness::Harness<T,S>::poll_inner::heb02f4e6c84ebcbb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27
  35:     0x55fb246ab7cb - tokio::runtime::task::harness::Harness<T,S>::poll::hfd7e4b1fe78a6d1b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15
  36:     0x55fb261b889b - tokio::runtime::task::raw::RawTask::poll::hdc0638ec3477c651
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18
  37:     0x55fb261b889b - tokio::runtime::task::LocalNotified<S>::run::h3eca1b7b6c0ff073
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:394:9
  38:     0x55fb261b889b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::hcf503783c628c3f2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:464:18
  39:     0x55fb261b889b - tokio::runtime::coop::with_budget::h4e07396bec96a62f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:107:5
  40:     0x55fb261b889b - tokio::runtime::coop::budget::hd59c7385053dc324
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:73:5
  41:     0x55fb261b889b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h1d3cbcbc59692abb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  42:     0x55fb261b7ef3 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h967838a12dd90f89
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  43:     0x55fb261b1b4d - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h98d75ebc0a42ba04
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  44:     0x55fb261b1b4d - tokio::macros::scoped_tls::ScopedKey<T>::set::hc84c484f4468a21f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/macros/scoped_tls.rs:61:9
  45:     0x55fb261b7968 - tokio::runtime::scheduler::multi_thread::worker::run::h1ad887c56c826e53
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  46:     0x55fb2619fe8c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hfc039eb05e515bbc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  47:     0x55fb2619fe8c - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hdf2fd0cd4c9b599c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/task.rs:42:21
  48:     0x55fb2619fe8c - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h36bea44352486a86
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  49:     0x55fb261a646d - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h45ba73a378397777
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17
  50:     0x55fb261a646d - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h332f6a1dfa39f126
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9
  51:     0x55fb261a646d - tokio::runtime::task::core::Core<T,S>::poll::h39355b9e9cf5cb53
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13
  52:     0x55fb2618911f - tokio::runtime::task::harness::poll_future::{{closure}}::h220fc500dca0110a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19
  53:     0x55fb2618911f - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hf1e0a37108050006
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9
  54:     0x55fb2618911f - std::panicking::try::do_call::hafb835baf5d32b8d
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40
  55:     0x55fb2618911f - std::panicking::try::hb1c4cfbc396b16c8
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19
  56:     0x55fb2618911f - std::panic::catch_unwind::hc7d82b9e7b110d07
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14
  57:     0x55fb2618911f - tokio::runtime::task::harness::poll_future::ha98a8080fed0ce9a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18
  58:     0x55fb2618911f - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h449cb9e6326aa520
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27
  59:     0x55fb2618911f - tokio::runtime::task::harness::Harness<T,S>::poll::h89999a73a01b19fc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15
  60:     0x55fb2618752f - tokio::runtime::task::raw::RawTask::poll::hdc0638ec3477c651
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18
  61:     0x55fb2618752f - tokio::runtime::task::UnownedTask<S>::run::h3e761fbae7d5a4a1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:431:9
  62:     0x55fb2618752f - tokio::runtime::blocking::pool::Task::run::h959784bc89ce8171
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:159:9
  63:     0x55fb2618752f - tokio::runtime::blocking::pool::Inner::run::h142cc69e5014d1d7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:513:17
  64:     0x55fb261afa45 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h7149a4dd097b1f50
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:471:13
  65:     0x55fb261afa45 - std::sys_common::backtrace::__rust_begin_short_backtrace::ha34db0efeef8c979
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys_common/backtrace.rs:135:18
  66:     0x55fb26193d52 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h777b12139f0506f3
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/thread/mod.rs:529:17
  67:     0x55fb26193d52 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4ebf44199056272a
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/panic/unwind_safe.rs:271:9
  68:     0x55fb26193d52 - std::panicking::try::do_call::hb74c8c77bddfcf8a
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:500:40
  69:     0x55fb26193d52 - std::panicking::try::h0b6eea8e742313b4
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panicking.rs:464:19
  70:     0x55fb26193d52 - std::panic::catch_unwind::h57306b66870b405e
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/panic.rs:142:14
  71:     0x55fb26193d52 - std::thread::Builder::spawn_unchecked_::{{closure}}::h0614202d9a5d1595
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/thread/mod.rs:528:30
  72:     0x55fb26193d52 - core::ops::function::FnOnce::call_once{{vtable.shim}}::heb339c61fbbafa40
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/core/src/ops/function.rs:250:5
  73:     0x55fb2612aa85 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h0716d70b901d1398
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1985:9
  74:     0x55fb2612aa85 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h7c11e50091aab2e1
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/alloc/src/boxed.rs:1985:9
  75:     0x55fb2612aa85 - std::sys::unix::thread::Thread::new::thread_start::h3ae99d1c2785d9c6
                               at /rustc/f0411ffcebcd7f75ac02ed45feb53ffd07b75398/library/std/src/sys/unix/thread.rs:108:17
  76:     0x7f00b227cb43 - start_thread
                               at ./nptl/pthread_create.c:442:8
  77:     0x7f00b230dbb4 - clone
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100
  78:                0x0 - <unknown>
[root@master docker]# 

[zwang28]

2023-07-17T07:30:26.905723384Z  WARN risingwave_meta::manager::system_param: The initializing value of ""state_store"" (hummock+s3://aicl2) differ from persisted (hummock+minio://hummockadmin:hummockadmin@minio-0:9301/hummock001), using persisted value
    at src/meta/src/manager/system_param/mod.rs:190

So node actually tried to connect to minio instead of S3 unexpectedly. This was caused by the persisted cluster level system variable pointing to minio unexpectedly. I guess you have run docker-compose before, which set state_store to minio, and haven't removed containers/resource of that cluster before docker-compose-with-s3.yml?
I believe recreating the cluster from scratch, i.e. removing all previous container/volume (especially etcd), would fix the issue.
@zhifeng110

FYI the state_store system parameter is initialized once during cluster initialization and persisted in etcd. It cannot be changed later for this cluster.

[zhifeng110]

if you want to change store(minio, oss, cos...), remember to clean the cluster volume. thank you !

docker compose down -v