Filer cannot upload files to the new deployed free volume

[youngcube]

Hi

I use docker-compose to setup a bunch of application for seaweedfs.
Let's just say I have 2 Linux Server(CN0, CN1) and they all contains Docker

Here's my environment:

• CN0: I deployed 5 services: master, volume, filer, kafka and etcd. The volume bind to /data/disk1 which contains 1G free space in xfs format.
• CN1: I deployed 1 service: volume. The volume bind to /data/disk1 which contains 1G free space in xfs format.

First I deploy CN0, and upload many files in order to make it full.

When CN0's volume has only 600k~800k (according to http://CN0:8080/ui/index.html), I can not upload files through filer anymore.

Then I deploy CN1 (add volume only), I want to enlarge my seaweedfs's free space.

When it's done, I still cannot upload any files through CN0's filer.

At this time, filer and master has no error log or normal log printed. But in CN0's volume, it print failed to write to local disk: write /data/1.dat: no space left on device.

It seems that filer is not upload my files to the free CN1 volume but upload to the CN0 volume which is out of space.

I've also tried restart master, filer but no luck.

I upload files through filer.

I use etcd as filer's database (also tried origin leveldb2 but no luck).

Here's my info:

docker-compose.yml

CN0

version: '3.6'

services:
  master:
    image: ${IMAGE} # use a remote image
    restart: always
    ports:
      - 9333:9333
      - 19333:19333
    command: 'master -ip=${CN0IP} -ip.bind="0.0.0.0"'
    container_name: weedfs_master
    ulimits:
      nproc: 65535
  volume:
    image: ${IMAGE} # use a remote image
    restart: always
    ports:
      - 8080:8080
      - 18080:18080
    command: 'volume -mserver="${CN0IP}:9333" -ip=${CN0IP} -ip.bind="0.0.0.0" -dataCenter=EuDataCenter -port=8080'
    container_name: weedfs_volume
    ulimits:
      nproc: 65535
    volumes:
      - type: bind
        source: ${DATAPATH}
        target: /data
    depends_on:
      - master
  filer:
    image: ${IMAGE} # use a remote image
    restart: always
    ulimits:
      nproc: 65535
    ports:
      - 8888:8888
      - 18888:18888
    command: 'filer -master="${MASTERIP}:9333" -port=8888 -dataCenter=EuDataCenter'
    container_name: weedfs_filer
    tty: true
    stdin_open: true
    volumes:
      - type: bind
        source: /data/leveldb2
        target: /data/filerleveldb2
    environment:
      RABBIT_SERVER_URL: '${RABBIT_SERVER_URL}'
    depends_on:
      - master
      - volume

CN1

version: '3.6'

services:
  volume:
    image: ${IMAGE} # use a remote image
    restart: always
    ports:
      - 8080:8080
      - 18080:18080
    command: 'volume -mserver="${CN0IP}:9333" -ip=${CN1IP} -ip.bind="0.0.0.0" -dataCenter=EuDataCenter -port=8080'
    container_name: weedfs_volume
    ulimits:
      nproc: 65535
    volumes:
      - type: bind
        source: ${DATAPATH}
        target: /data

*.tomls

notification.toml

[notification.kafka]
enabled = true
hosts = [
  "{EUDIC_KAFKA_ADDRESS}:9092" #10.117.2.80:9092
]
topic = "seaweedfs_replication"
offsetFile = "/notification/last.offset"
offsetSaveIntervalSeconds = 10

filer.toml

[filer.options]
recursive_delete = false
buckets_folder = "/" #changed, origin is /bucket
buckets_fsync = [
	"important_bucket",
	"should_always_fsync",
]

[leveldb2]
enabled = false
dir = "/data/filerldb2" # directory to store level db files

[etcd]
enabled = true
servers = "{CN0IP}:2379"
timeout = "3s"

master.toml

[master.maintenance]
scripts = """
  ec.encode -fullPercent=95 -quietFor=1h
  ec.rebuild -force
  ec.balance -force
  volume.balance -collection=ALL_COLLECTIONS -force
  volume.fix.replication
"""
sleep_minutes = 1          # changed, origin is 19

[master.filer]
default = "{CN0}:8888"    # used by maintenance scripts if the scripts needs to use fs related commands


[master.sequencer]
type = "etcd"     # changed, origin is memory. but if I change it back to memory it still the same
sequencer_etcd_urls = "http://{CN0}:2379"

replication.toml

[source.filer]
enabled = true
grpcAddress = "{CN0}:18888" #localhost:18888
# all files under this directory tree are replicated.
# this is not a directory on your hard drive, but on your filer.
# i.e., all files with this "prefix" are sent to notification message queue.
directory = "/"

[sink.filer]
enabled = true
grpcAddress = "{CN2}:18888" #localhost:18888
# all replicated files are under this directory tree
# this is not a directory on your hard drive, but on your filer.
# i.e., all received files will be "prefixed" to this directory.
directory = "/"
replication = ""
collection = ""
ttlSec = 0

Tested seaweedfs version: 1.74~1.75
What's the best practice of enlarge seaweedfs's free space and make filer to upload my files to the right volume?
Thanks for your time!

[chrislusf]

The volume size in docker image entrypoint.sh is 1GB. If you have less than 1GB disk, you can lower the number.

see https://github.com/chrislusf/seaweedfs/blob/master/docker/entrypoint.sh#L6