-
Notifications
You must be signed in to change notification settings - Fork 16
Home
Welcome to the StoneNeedle wiki! Thank you for your attention, which is a powerful driver of the project!
StoneNeedle contains NVMe driver patch and proc file system patch, and both are running in kernel space.
The NVMe device driver patch code is a bunch of C code within NVMe driver. Its responsibility is to monitor I/O requests and calculate required profiling metrics value.
The Linux proc file system patch exposes the StoneNeedle controlling trigger and the metrics data from kernel space to user space. User can start/stop workload data capturing, and access the I/O profiling data via this proc file system interface.
LBA Chunk: a sequenctial LBA range. StoneNeedle can be configured to divide a whole disk's LBA space into several LBA Chunks as the statistical unit for profiling data.
StoneNeedle captures the storage workload profiling data at device level. The I/O metric items are driven by the workload pattern research needs, and new I/O metric items will be added along with the research requirements.
* Read/write count and size statistic
* WRITE_IO_COUNT
* READ_IO_COUNT
* WRITE_TOTAL_BYTES
* READ_TOTAL_BYTES
* Sequential I/O statistic
* WRITE_SEQUENTIAL_BYTES
* READ_SEQUENTIAL_BYTES
* Random I/O statistic
* WRITE_RANDOM_IO_COUNT
* READ_RANDOM_IO_COUNT
* Spatial (LBA) jump, LBA jumps for adjacent requests. The bigger the value, the more random the requests.
* WRITE_STRIDE_SECTORS
* READ_STRIDE_SECTORS
* Re-arrive time interval accumulation in each LBA Chunk
* WRITE_IO_ARRIVAL_INTERVAL
* READ_IO_ARRIVAL_INTERVAL
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET0: buckets: =0
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET1: buckets: (0, 1us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET2: buckets: (1us, 2us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET3: buckets: (2us, 4us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET4: buckets: (4us, 8us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET5: buckets: (2^3us, 2^4us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET6: buckets: (2^4us, 2^5us]
...
...
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET22: buckets: (2^20us, 2^21us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET23: buckets: (2^21us, 2^22us]
* WRITE_IO_ARRIVAL_INTERVAL_BUCKET24: buckets: (2^22us, +∞)
* Statistics of I/O skewness heatmap in each LBA Chunk
* 0KB_ WRITE_SKEWNESS ~ 124KB_WRITE_SKEWNESS: buckets:[0,4KB),[4KB,8KB),[8KB,12KB)….[124KB,128KB)
* 128KB_WRITE_SKEWNESS: bucktes: [128KB,256KB)
* 256KB_WRITE_SKEWNESS: bucktes: >=256KB
* 0KB_ READ_SKEWNESS ~ 124KB_READ_SKEWNESS: buckets:[0,4KB),[4KB,8KB),[8KB,12KB)….[124KB,128KB)
* 128KB_READ_SKEWNESS: bucktes: [128KB,256KB)
* 256KB_READ_SKEWNESS: bucktes: >=256KB
Step 1 Make sure the depend libraries are installed, such as gcc, ncurses-devel.
Step 2 Replace NVMe module source code with all files in the drivers/block directory.
1) Extract the StoneNeedle code and the original linux-3.13.tar.gz. Assume the extracted directories are $sn and $base.
`tar -zxvf linux-3.13.tar.gz`
2) Copy all the files of `drivers/block` under directory of StoneNeedle to the `linux-3.13/drivers/block` directory and overwrite the same files.
`cp $sn/Stone_Needle/drivers/block/* $base/linux-3.13/drivers/block/`
Step 3 Configure the NVMe driver module by following steps:
1) Create compiling folder and open the configuration page
`mkdir /usr/src/kernels/linux-3.13.0-kernel`
`cd $base/linux-3.13`
`make O=/usr/src/kernels/linux-3.13.0-kernel menuconfig`
Note: Check in advance that all the files are in “unix” format.
2) If you want to use STONENEEDLE module, select block device as *, choose NVM Express block device as M, and STONENEEDLE module as * in sequence.
3) Choose Save and OK to exit.
Step 4 Compile kernel by running the following command:
`cd /usr/src/kernels/linux-3.13.0-kernel`
`make -j8`
Step 5 Install and reboot machine on newly built kernel images by running the following commands:
`make modules_install`
`make install`
`reboot`
Step 6 Check whether you can see the /proc/stoneneedle/controller file after reboot
Use StoneNeedle to capture workload I/O profiling data in the following steps. Administrator priority is needed.
Step 1 Start capturing data on device “nvme0n1” by command:
`echo "start nvme0n1" > /proc/stoneneedle/controller`
Note:
- This command resets all metrics’ values under related proc file to zero
- To start capturing data on multiple NVMe SSDs, user can repeat the command on each device
Step 2 Get the IO metrics’ data by the following command:
`cat /proc/stoneneedle/nvme0n1`
Note: User can use this command to get workload I/O profiling information by StoneNeedle for the following parsing. All the numeric fields store cumulative values, so they may **overflow** and wrap. On a very busy or long-lived system, 32-bit (2^32 -1: 4294967295) or 64-bit (2^64 - 1:18446744073709551615)) type of variable may not be big enough and the value may overflow. Applications should be prepared to deal with that.
Step 3 Stop capturing data on device “nvme0n1” by command:
`echo "stop nvme0n1" > /proc/stoneneedle/controller`
Note: This command stops StoneNeedle from counting. The proc file metrics’ value are kept until StoneNeedle reset or kernel module reloads
A: Yes, it can run on both Ubuntu and CentOS. Ubuntu 14.04 LTS and CentOS 6/7 have passed the test. We can not guarantee it will be okay on other Linux distributions.
A: When nvme.ko loads, the StoneNeedle module will create proc file for each NVMe device at the same time. For example, if a server connects three NVMe SSDs, the structure of the StoneNeedle proc directory will be like:
`/proc/stoneneedle/`
`├── controller`
`└── nvme0n1`
`└── nvme1n1`
`└── nvme2n1`
A: StoneNeedle only take quite small CPU and memory resource, and has very little side effect.