Erasure coding not working properly

[manuelfreitas11]

Hello so i have been having some issues with erasure coding, i currently have 8 nodes 4 of which are parity, however when 2 nodes go down they al appear offline in my node 1 local console ui, what can i do ,

set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

echo "[0/5] Installing dependencies"
sudo apt-get update -y
sudo apt-get install -y xfsprogs unzip curl wget jq net-tools

RUSTFS_SERVICE_PORT="${rustfs_port}"
RUSTFS_CONSOLE_PORT="${rustfs_console_port}"
RUSTFS_MOUNT_PREFIX="${rustfs_data_dir}"
RUSTFS_DISK_PATH="${rustfs_disk_path}"
RUSTFS_NODE_LIST="${rustfs_node_list}"
LABEL="RUSTFS0"
MOUNTPOINT="${RUSTFS_MOUNT_PREFIX}0"

IFS=' ' read -r -a NODE_ARRAY <<< "$RUSTFS_NODE_LIST"

echo "[1/5] Ensuring /etc/hosts contains cluster entries"
tmp_hosts=$(mktemp)
cp /etc/hosts "$tmp_hosts"

for NODE in "${NODE_ARRAY[@]}"; do
  if ! grep -q "$NODE" "$tmp_hosts"; then
    ip=$(getent hosts "$NODE" | awk '{print $1}' || true)
    if [ -n "$ip" ]; then
      echo "$ip $NODE" >> "$tmp_hosts"
    fi
  fi
done

sudo cat "$tmp_hosts" > /etc/hosts
sudo rm -f "$tmp_hosts"

echo "[2/5] Building VOLUME_URLS"
VOLUME_URLS=""

for NODE in "${NODE_ARRAY[@]}"; do
  VOLUME_URLS+=" http://${NODE}:${RUSTFS_SERVICE_PORT}${MOUNTPOINT}"
done

VOLUME_URLS="${VOLUME_URLS# }"

if ! blkid /dev/sdb >/dev/null 2>&1; then
  sudo mkfs.xfs -i size=128 -n ftype=1 -L RUSTFS0 /dev/sdb
fi


FSTAB_LINE="LABEL=$LABEL /data/rustfs0  xfs defaults,noatime,nodiratime 0 0"

echo "[3/5] Formatting and mounting $RUSTFS_DISK_PATH"
if ! grep -q "LABEL=RUSTFS0 /data/rustfs0 " /etc/fstab; then
  echo "$FSTAB_LINE" | sudo tee -a /etc/fstab >/dev/null
fi

sudo mount -a
sudo chmod 750 "$MOUNTPOINT"

echo "[2/5] Installing RustFS binary"
sudo wget https://dl.rustfs.com/artifacts/rustfs/release/rustfs-linux-x86_64-musl-latest.zip
sudo unzip rustfs-linux-x86_64-musl-latest.zip
sudo chmod +x rustfs
mv rustfs /usr/local/bin/


echo "[4/5] Writing /etc/default/rustfs"
tee /etc/default/rustfs  <<EOF
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="$VOLUME_URLS"
RUSTFS_ADDRESS=":${RUSTFS_SERVICE_PORT}"
RUSTFS_CONSOLE_ENABLE=true
RUSTFS_CONSOLE_ADDRESS=":${RUSTFS_CONSOLE_PORT}"
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
RUSTFS_ERASURE_SET_DRIVE_COUNT=8
RUSTFS_STORAGE_CLASS_STANDARD=EC:4
EOF

sudo mkdir -p /data/rustfs0 /var/logs/rustfs /opt/tls
sudo chmod -R 750 /data/rustfs* /var/logs/rustfs

echo "[5/5] Installing systemd service"
tee /etc/systemd/system/rustfs.service  <<EOF
[Unit]
Description=RustFS Object Storage Server
Documentation=https://rustfs.com/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main
User=root
Group=root

WorkingDirectory=/usr/local
EnvironmentFile=-/etc/default/rustfs
ExecStart=/usr/local/bin/rustfs \$RUSTFS_VOLUMES

LimitNOFILE=1048576
LimitNPROC=32768
TasksMax=infinity

Restart=always
RestartSec=10s

OOMScoreAdjust=-1000
SendSIGKILL=no

TimeoutStartSec=30s
TimeoutStopSec=30s

NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictRealtime=true

StandardOutput=append:/var/logs/rustfs/rustfs.log
StandardError=append:/var/logs/rustfs/rustfs-err.log

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now rustfs

[loverustfs]

Would you mind posting the full /etc/default/rustfs configuration? Please include the specific startup errors as well. Thanks!

[manuelfreitas11]

Thanks for the quick response, here is the /etc/default/rustfs as requested

me@rustfs-rustfs-01:~$ sudo cat /etc/default/rustfs
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://rustfs-rustfs-01.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-02.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-03.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-04.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-05.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-06.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-07.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0 http://rustfs-rustfs-08.c.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUSTFS_CONSOLE_ADDRESS=":9001"
RUST_LOG=info
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"

[loverustfs]

I believe this is an incorrect configuration file. The correct one should be:

RUSTFS_VOLUMES="http://rustfs-rustfs-{01....07}.ecstatic-bounty-xxxxxx-c1.internal:9000/data/rustfs0