[1.3][benchmark] Read/Write Performance of Large File with LeoFS

[yosukehara]

Environment

• Erlang/OTP: 17.5
• LeoFS: 1.3.0
• Test Configuration:
  • Tool: s3cmd-1.6.1
  • Object Size: 1 GB
  • Number of Objects: 500
  • Number of Threads: 20
  • Multipart Upload Part Size: 5 MiB
• LeoFS' sytem

  • of leo_gateway: 1

  • of leo_storage: 5

  • of leo_manager: 2

• Benchmark tool: basho_bench

Report

• Report: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/1.3/1.3.0/front/20160919_large1g_f500_rw_1
• Report: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/1.3/1.3.0/front/20161005_large1g_f500_rw_1

[yosukehara]

Everyone cannot understand the detail of the benchmark report. We need to add some items in the benchmark report as below:

### Summary
- Description

### Finding issues
- Subject
   - What happened and How to reproduce if possible
   - Information
       - `leofs-doctor` result
       - Summary of errors
           - Description
           - Error log files
       - [Status:[open|close]] [A link of LeoFS' issue]

### Next Action
- Subject
    - Description

[windkit]

[PUT] OK, [GET] Large Variation

https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/1.3/1.3.0/front/20161005_large1g_f500_rw_1

LeoFS Settings

• 1 gateway, 5 storage, 10Gbps Link
• No GW Cache
• SSD-backed Storage

Methodology

• Basho Bench Driver Wrapper around S3cmd
  • spawn s3cmd with os:cmd
• 20 processes in erlang spawning s3cmd
• Data Set: 500 x 1GB Files
  • Base File served at ramdisk /dev/shm
• [PUT] Partitioned Sequential
• [GET] Random

Summary

• During Loading Phase, the throughput is bounded by Disk I/O rate
  • 450MB/s * 5 (nodes) / 3 (replicas) = 800MB/s
• During Reading Phase, the throughput is going up and down wildly
  • Average around 500MB/s, Up to 700MB/s, Down to 200MB/s

Next Action

• Check if problem persists in single thread
• Detailed breakdown of latency in each processes

[windkit]

Single Threaded Result

https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/1.3/1.3.0/front/20161013_large1g_f500_rw_1

Methodology

• Basho Bench Driver Wrapper around S3cmd
  • spawn s3cmd with os:cmd
• Simple Bash Script to execute s3cmd
• [PUT] Partitioned Sequential
• [GET] Random

Summary

• During Reading Phase, decrease in throughput is observed
  • Start with 250MB/s, drop to 150MB/s after 10 minutes
  • Occasional drop to 50MB/s
• To rule out effect from basho_bench, a simple bash script is used to verify
  • Same situation

Findings

• Single threaded test show similar results with the parallel one

  • Decrease in throughput over time, abnormal drop after a while
  • A simple bash script showing similar result rules out problem from basho_bench

• As page cache is dropped before every test, try to execute once in the middle

  • At 13:00, drop page cache of all nodes (gateway, storage)
  • Throughput shows improvement immediately, back to 250MB/s from 150MB/s
  • Afterwards showing similar trend as the beginning

  $ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
  

  $ free -m
           total       used       free     shared    buffers     cached
  Mem:         31934       7826      24107          0         10       7158
  -/+ buffers/cache:        656      31277
  Swap:        32541         43      32498
  $ free -m
           total       used       free     shared    buffers     cached
  Mem:         31934      19357      12576          0         10      18671
  -/+ buffers/cache:        675      31258
  Swap:        32541         43      32498
  $ sleep 180; free -m
           total       used       free     shared    buffers     cached
  Mem:         31934      28469       3465          0         10      27813
  -/+ buffers/cache:        644      31289
  Swap:        32541         43      32498
  $ sleep 180; free -m
           total       used       free     shared    buffers     cached
  Mem:         31934      31611        322          0         10      30892
  -/+ buffers/cache:        708      31225
  Swap:        32541         43      32498
  

Next Action

• Research on why throughput is better when page cache is empty (dropped)
• Detailed breakdown of latency in each processes

[windkit]

Correlation between page cache usage (cached) and received throughput