Cluster crush after edit configuration file

[whls]

Describe the bug
Cluster crush after add environment variables to configuration file /etc/default/rustfs.

To Reproduce
Steps to reproduce the behavior:

1. Add RUSTFS_SERVER_DOMAINS to the configuration file /etc/default/rustfs:
   RUSTFS_ACCESS_KEY=rustfsadmin
   RUSTFS_SECRET_KEY=rustfsadmin
   RUSTFS_VOLUMES="http://rustfs-node0{1...4}:9000/data/rustfs0{1...4}"
   RUSTFS_ADDRESS=":9000"
   RUSTFS_CONSOLE_ENABLE=true
   RUST_LOG=error
   RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
   RUSTFS_SERVER_DOMAINS=rustfs.eccom.com

2. Restart rustfs service and service status is running.

3. Can't access cluster:
   [root@rustfs-node01 ~]# mc ls rustfs
   mc: Unable to list folder. Not init

4. find error in log file:
   RustFS API: http://172.20.57.161:9000 http://127.0.0.1:9000
   RustFS Start Time: 2025-12-15 16:56:28
   Console WebUI Start Time: 2025-12-15 16:56:28
   Console WebUI available at: http://172.20.57.161:9001/rustfs/console/index.html
   Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
   Error: Custom { kind: Other, error: "ec init failed" }

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: CentOS Linux release 8.4.2105
• Browser: 143.0.7499.109
• Version: rustfs 1.0.0-alpha.72

Additional context
Follow the official document below to perform the operation: https://docs.rustfs.com/integration/virtual.html

[loverustfs]

What is the initial configuration?

Why is it necessary to adjust to the new configuration?

[whls]

@loverustfs
Following is the initial configuration :
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://rustfs-node0{1...4}:9000/data/rustfs0{1...4}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"

Don't have variables RUSTFS_SERVER_DOMAINS.
For change request paths mode to Virtual Host Style, need add variables RUSTFS_SERVER_DOMAINS to configuration file.

[loverustfs]

New configure file

RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://rustfs-node0{1...4}:9000/data/rustfs0{1...4}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
RUSTFS_SERVER_DOMAINS=rustfs.eccom.com

Old configure file

RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://rustfs-node0{1...4}:9000/data/rustfs0{1...4}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"

Question

• What is the error log?

• How did you access it?

• Did deleting RUSTFS_SERVER_DOMAINS restore normal operation?

[whls]

• What is the error log?
  In rustfs.log file:
  RustFS API: http://172.20.57.161:9000 http://127.0.0.1:9000
  RustFS Start Time: 2025-12-16 11:04:29
  Console WebUI Start Time: 2025-12-16 11:04:29
  Console WebUI available at: http://172.20.57.161:9001/rustfs/console/index.html
  Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
  Error: Custom { kind: Other, error: "ec init failed" }
  RustFS API: http://172.20.57.161:9000 http://127.0.0.1:9000
  RustFS Start Time: 2025-12-16 11:05:39
  Console WebUI Start Time: 2025-12-16 11:05:39
  Console WebUI available at: http://172.20.57.161:9001/rustfs/console/index.html
  Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
  Error: Custom { kind: Other, error: "ec init failed" }

More log detail please see the attached file.

rustfs_r2025-12-15_19-00-00.log

• How did you access it?
  Access cluster with mc client and Web UI, all failed.

• Did deleting RUSTFS_SERVER_DOMAINS restore normal operation?
  Yes, delete RUSTFS_SERVER_DOMAINS and restart services, cluster access and get object successful.

[loverustfs]

@whls ,

I'm sorry! I used a binary deployment and was unable to reproduce this bug on alpha.73.

[whls]

So do I need update rustfs to alpha.73 and test?
If needed, how can I update rustfs , I don't find any document about it. Just need replace binary file and restart service?

[loverustfs]

So do I need update rustfs to alpha.73 and test? If needed, how can I update rustfs , I don't find any document about it. Just need replace binary file and restart service?

Yes, replace binary and restart service.

[loverustfs]

If errors persist after the upgrade.
We can prepare four servers online for a 1:1 simulation test.

[loverustfs]

Is the startup process normal? @whls

[whls]

After upgrade to alpha.73, this problem still exists.

[root@rustfs-node01 ~]# rustfs --version
rustfs 1.0.0-alpha.73
build time : 2025-12-11 02:26:10 +00:00
build profile: release
build os : linux-x86_64
rust version : rustc 1.91.1 (ed61e7d7e 2025-11-07)
rust channel : stable-x86_64-unknown-linux-gnu
git branch :
git commit : fba201d
git tag : 1.0.0-alpha.73
git status :

[root@rustfs-node01 ~]# tail -f /var/logs/rustfs/rustfs.log
RustFS Start Time: 2025-12-17 10:34:36
Console WebUI Start Time: 2025-12-17 10:34:36
Console WebUI available at: http://172.20.57.161:9001/rustfs/console/index.html
Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
Error: Custom { kind: Other, error: "ec init failed" }
RustFS API: http://172.20.57.161:9000 http://127.0.0.1:9000
RustFS Start Time: 2025-12-17 10:35:46
Console WebUI Start Time: 2025-12-17 10:35:46
Console WebUI available at: http://172.20.57.161:9001/rustfs/console/index.html
Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
Error: Custom { kind: Other, error: "ec init failed" }
RustFS API: http://172.20.57.161:9000 http://127.0.0.1:9000
RustFS Start Time: 2025-12-17 10:36:59
Console WebUI Start Time: 2025-12-17 10:36:59
Console WebUI available at: http://172.20.57.161:9001/rustfs/console/index.html
Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
Error: Custom { kind: Other, error: "ec init failed" }
RustFS API: http://172.20.57.161:9000 http://127.0.0.1:9000

the detail log file:

rustfs_r2025-12-16_18-33-24.log

my configure file:

[root@rustfs-node01 ~]# cat /etc/default/rustfs
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://rustfs-node0{1...4}:9000/data/rustfs0{1...4}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
RUSTFS_SERVER_DOMAINS=rustfs.eccom.com

my service file:

[root@rustfs-node01 ~]# cat /etc/systemd/system/rustfs.service
[Unit]
Description=RustFS Object Storage Server
Documentation=https://rustfs.com/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main
User=rustfs-user
Group=rustfs-user

WorkingDirectory=/usr/local
EnvironmentFile=-/etc/default/rustfs
ExecStart=/usr/local/bin/rustfs $RUSTFS_VOLUMES

LimitNOFILE=1048576
LimitNPROC=32768
TasksMax=infinity

Restart=always
RestartSec=10s

OOMScoreAdjust=-1000
SendSIGKILL=no

TimeoutStartSec=30s
TimeoutStopSec=30s

NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
#ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictRealtime=true

StandardOutput=append:/var/logs/rustfs/rustfs.log
StandardError=append:/var/logs/rustfs/rustfs-err.log

[Install]
WantedBy=multi-user.target

my disk confiugre:

[root@rustfs-node01 ~]# cat /etc/fstab

/dev/mapper/cl-root / xfs defaults 0 0
UUID=d628d45d-b26b-45ca-947e-2f6adac3b6c7 /boot xfs defaults 0 0
UUID=7B83-DA95 /boot/efi vfat umask=0077,shortname=winnt 0 2
/dev/mapper/cl-swap none swap defaults 0 0
LABEL=RUSTFS01 /data/rustfs01 xfs defaults,noatime,nodiratime 0 0
LABEL=RUSTFS02 /data/rustfs02 xfs defaults,noatime,nodiratime 0 0
LABEL=RUSTFS03 /data/rustfs03 xfs defaults,noatime,nodiratime 0 0
LABEL=RUSTFS04 /data/rustfs04 xfs defaults,noatime,nodiratime 0 0

And after remove RUSTFS_SERVER_DOMAINS and restart services, cluster access and get object successful.

[loverustfs]

Okay, I will test this bug again.

Version used: alpha.73
Servers: 4
OS: ubuntu24.04
Disks: 4 drivers (4 nodes x 4 drivers)

[loverustfs]

Confirmed, same as #1201 .

Same bug. For easier management, we will track and resolve it in #1201 .