setting the enable_preload_parts parameter to 1 will result in a virtualWarehouses side disk capacity utilization of 100%

[FourSpaces]

Bug Report

setting the enable_preload_parts parameter to 1 will result in a virtualWarehouses side disk capacity utilization of 100%

Briefly describe the bug

Application: DB::ErrnoException: Cannot write to file /var/byconity/disks/DISK_INFOS.tmp, errno: 28, strerror: No space left on device SQLSTATE: 58030

Version

byconity/byconity:latest

[FourSpaces]

[hustnn]

Thanks for reporting, we will check the logic of preload.

[foreverneverer]

The pic show the machine disk capacity is enough but the log show no space? or the disk actually is no space when the error log happen?

[FourSpaces]

When the error occurred, the disk space was already full. I need to clean up the disk for byconsistency to function properly

[gaodayue]

Hi @FourSpaces , what's the capacity of the disk mounted to the path. Also could you share the <storage_configuration> for the byconity worker?

[foreverneverer]

When the error occurred, the disk space was already full. I need to clean up the disk for byconsistency to function properly

you may need check the config:

<disk_cache_strategies>
   <simple> // or <MergeTree></MergeTree>
     <lru_max_size>xxxx</lru_max_size> // here, the default value is 2T
    </simple>
</disk_cache_strategies>

if you don't add the config, the max cache size will be 2T and it may be full

[FourSpaces]

@gaodayue

this is worker config

  virtualWarehouses:
    - <<: *defaultWorker
      name: vw_default
      replicas: 8
      resources:
        requests:
          memory: "55Gi"
          cpu: 14
        limits:
          memory: "55Gi"
          cpu: 14
      storage:
        localDisk:
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 550Gi
            storageClassName: ebs-ssd-pl #replace to your storageClassName
        log:
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 25Gi
            storageClassName: ebs-ssd #replace to your storageClassName
      configOverwrite:
        logger:
          level: trace
        disk_cache_strategies:
          simple:
            lru_max_size: 483183820800 # 450Gi

[gaodayue]

@FourSpaces Has this problem occurred recently? If yes, cloud you also provide the output of select * from system.disks on worker?

[FourSpaces]

@gaodayue We have switched to version 0.2.0, and this error did not occur in version 0.2.0. Let's observe later and I'll raise any questions if there are any

[skyoct]

@hustnn Recurring this issue

Error:

<Error> Application: DB::ErrnoException: Cannot write to file /var/byconity/disks/DISK_INFOS.tmp, errno: 28, strerror: No space left on device SQLSTATE: 58030

Config:

[gaodayue]

@skyoct please check the output of du -sh --apparent-size /var/byconity/disk_cache_v1. If it's close to lru_max_size and significant smaller than du -sh /var/byconity/disk_cache_v1, then you have encountered the space amplification problem caused by small files. As a workaround, you can further reduce the size of lru_max_size.