writeback mode?

[syadnom]

I have a use-case where writeback mode is ideal. To clarify so we both know what I mean, I want to consider the write complete when it's written to the rdX device then write that to disk async.

I'd also like to see a timed flush and delay write.

here is the scenario

many small writes come in, enough to overwhelm a spindled disks random writes
some reads come in and that hurts random writes even more.
data is at a high enough volume that it does need written pretty soon or the cache will overflow.

want:
write to rdX, rd waits to write that to disk for a timer, then flushes to disk creating a write-deferred cache. This turns overwhelming random writes into a comfortable amount of sequential writes.

This might look like a corner case, but it's a very common scenario in security camera storage systems. Might only have ~12MBps in writes, but those are random and overcome a WD Red quickly. A Write-Deferred option solves this 100%.

There is evidence of this too. The Windows program PrimoCache does exactly this and it allows a windows server to handle 4x as many streams easily.

[pkoutoupis]

Before I rushed to reply to this one, I wanted to give it the proper amount of thought. As you are aware, RapidDisk-Cache is lacking in a write-back mode. To date, that has been by design. Traditional RAM is volatile and by enabling a write-back caching mode without some form of not losing cached data in RAM would result in user data loss or possibly corruption. If for instance, there were a BBU holding the PC in a powered state, an emergency "flush" could be invoked to persist all pending and cached data.

Although, even if that problem were easily solved, implementing a write-back solution is no trivial matter, even if you are only holding X amount of transactions at a time. If you wanted to accomplish something similar and assuming that you are using Ext4, you can easily reduce the pressure of the file system to commit journal contents less frequently. For instance, setting a commit interval of 10 seconds as opposed to the default 5 may show a nice bump in performance and will also allow the file system and in turn the block I/O subsystem to group contiguous writes into few and larger commands. This could potentially reduce the bottleneck when accessing those disk drives. Another thing worth considering is enabling delayed allocation for the same file system. What this feature does is, it holds off on allocating extents or block regions on the disk device by leaving the contents in memory cache for a bit longer. You have a higher probability of allocating larger contiguous block regions when time comes for that data to flush.

Either way, both cases, when mishandled, can and will result in data loss. At least for pending data between each sync interval. Such are the woes of leveraging DRAM...

[pkoutoupis]

Before I rushed to reply to this one, I wanted to give it the proper amount of thought. As you are aware, RapidDisk-Cache is lacking in a write-back mode. To date, that has been by design. Traditional RAM is volatile and by enabling a write-back caching mode without some form of not losing cached data in RAM would result in user data loss or possibly corruption. If for instance, there were a BBU holding the PC in a powered state, an emergency "flush" could be invoked to persist all pending and cached data.

Although, even if that problem were easily solved, implementing a write-back solution is no trivial matter, even if you are only holding X amount of transactions at a time. If you wanted to accomplish something similar and assuming that you are using Ext4, you can easily reduce the pressure of the file system to commit journal contents less frequently. For instance, setting a commit interval of 10 seconds as opposed to the default 5 may show a nice bump in performance and will also allow the file system and in turn the block I/O subsystem to group contiguous writes into few and larger commands. This could potentially reduce the bottleneck when accessing those disk drives. Another thing worth considering is enabling delayed allocation for the same file system. What this feature does is, it holds off on allocating extents or block regions on the disk device by leaving the contents in memory cache for a bit longer. You have a higher probability of allocating larger contiguous block regions when time comes for that data to flush.

Either way, both cases, when mishandled, can and will result in data loss. At least for pending data between each sync interval. Such are the woes of leveraging DRAM...

[syadnom]

"Traditional RAM is volatile and by enabling a write-back caching mode without some form of not losing cached data in RAM would result in user data loss or possibly corruption" Absolutely true, but there are valid and common use cases where a small amount of data loss isn't a big deal. ANY kind of data logging this basically applies to because the power loss that would loose 30 seconds of data also takes down the sensors. Very little concern over the difference between losing 5 minutes or 5 minutes and 30 seconds of data. Security cameras are the same deal. ext4 is a bad filesystem for small files and many directories, the kind of thing that security systems and data loggers write. XFS is by far the preferred filesystem for these workloads. *all* circumstances that use RAM caching can and will result in data loss. the question is that tolerable or not. In the circumstances I describe above, they are acceptable. This is unrelated, but I'm doing this now via a convoluted process with aufs and ramdisks. Create a ramdisk and a directory on a hard disk. Mount aufs over the top with writes going to the ramdisk. rsync --remove-source-files from ramdisk to directory in the background on a timer. This is at the file level but does result in sequential writes. The downside is aufs doesn't do a fantastic job of tracking large numbers of files in the overlay so the VMS software can get a bit confuse with files vanishing and re-appearing as the indexes get sorted. I've tweaked this to the maximum. First doing a plain rsync with a file list, waiting a few minutes, then doing the 'move' against that file list so I can let aufs get it's indexes up to date and it 'works'. This has all the downsides of RAM cache which are unavoidable, but it is very effectively a delayed write cache for the hard drive. It's got an additional downside that as the file structure on the drive grows, rsync uses more and more cpu resources to build the file list. I think rapiddisk is the closest project to accomplishing a true delayed-write cache because it's already able to do regular caching that is seamless.

…

On Wed, Dec 20, 2017 at 1:23 PM, Petros Koutoupis ***@***.***> wrote: Before I rushed to reply to this one, I wanted to give it the proper amount of thought. As you are aware, RapidDisk-Cache is lacking in a write-back mode. To date, that has been by design. Traditional RAM is volatile and by enabling a write-back caching mode without some form of not losing cached data in RAM would result in user data loss or possibly corruption. If for instance, there were a BBU holding the PC in a powered state, an emergency "flush" could be invoked to persist all pending and cached data. Although, even if that problem were easily solved, implementing a write-back solution is no trivial matter, even if you are only holding X amount of transactions at a time. If you wanted to accomplish something similar and assuming that you are using Ext4, you can easily reduce the pressure of the file system to commit journal contents less frequently. For instance, setting a commit interval of 10 seconds as opposed to the default 5 may show a nice bump in performance and will also allow the file system and in turn the block I/O subsystem to group contiguous writes into few and larger commands. This could potentially reduce the bottleneck when accessing those disk drives. Another thing worth considering is enabling delayed allocation for the same file system. What this feature does is, it holds off on allocating extents or block regions on the disk device by leaving the contents in memory cache for a bit longer. You have a higher probability of allocating larger contiguous block regions when time comes for that data to flush. Either way, both cases, when mishandled, can and will result in data loss. At least for pending data between each sync interval. Such are the woes of leveraging DRAM... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJRXZVnCOUVZbATQiufj0Q-JSmSaTsaJks5tCWzRgaJpZM4RFzGh> .

[syadnom]

"Traditional RAM is volatile and by enabling a write-back caching mode without some form of not losing cached data in RAM would result in user data loss or possibly corruption" Absolutely true, but there are valid and common use cases where a small amount of data loss isn't a big deal. ANY kind of data logging this basically applies to because the power loss that would loose 30 seconds of data also takes down the sensors. Very little concern over the difference between losing 5 minutes or 5 minutes and 30 seconds of data. Security cameras are the same deal. ext4 is a bad filesystem for small files and many directories, the kind of thing that security systems and data loggers write. XFS is by far the preferred filesystem for these workloads. *all* circumstances that use RAM caching can and will result in data loss. the question is that tolerable or not. In the circumstances I describe above, they are acceptable. This is unrelated, but I'm doing this now via a convoluted process with aufs and ramdisks. Create a ramdisk and a directory on a hard disk. Mount aufs over the top with writes going to the ramdisk. rsync --remove-source-files from ramdisk to directory in the background on a timer. This is at the file level but does result in sequential writes. The downside is aufs doesn't do a fantastic job of tracking large numbers of files in the overlay so the VMS software can get a bit confuse with files vanishing and re-appearing as the indexes get sorted. I've tweaked this to the maximum. First doing a plain rsync with a file list, waiting a few minutes, then doing the 'move' against that file list so I can let aufs get it's indexes up to date and it 'works'. This has all the downsides of RAM cache which are unavoidable, but it is very effectively a delayed write cache for the hard drive. It's got an additional downside that as the file structure on the drive grows, rsync uses more and more cpu resources to build the file list. I think rapiddisk is the closest project to accomplishing a true delayed-write cache because it's already able to do regular caching that is seamless.

…

On Wed, Dec 20, 2017 at 1:23 PM, Petros Koutoupis ***@***.***> wrote: Before I rushed to reply to this one, I wanted to give it the proper amount of thought. As you are aware, RapidDisk-Cache is lacking in a write-back mode. To date, that has been by design. Traditional RAM is volatile and by enabling a write-back caching mode without some form of not losing cached data in RAM would result in user data loss or possibly corruption. If for instance, there were a BBU holding the PC in a powered state, an emergency "flush" could be invoked to persist all pending and cached data. Although, even if that problem were easily solved, implementing a write-back solution is no trivial matter, even if you are only holding X amount of transactions at a time. If you wanted to accomplish something similar and assuming that you are using Ext4, you can easily reduce the pressure of the file system to commit journal contents less frequently. For instance, setting a commit interval of 10 seconds as opposed to the default 5 may show a nice bump in performance and will also allow the file system and in turn the block I/O subsystem to group contiguous writes into few and larger commands. This could potentially reduce the bottleneck when accessing those disk drives. Another thing worth considering is enabling delayed allocation for the same file system. What this feature does is, it holds off on allocating extents or block regions on the disk device by leaving the contents in memory cache for a bit longer. You have a higher probability of allocating larger contiguous block regions when time comes for that data to flush. Either way, both cases, when mishandled, can and will result in data loss. At least for pending data between each sync interval. Such are the woes of leveraging DRAM... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJRXZVnCOUVZbATQiufj0Q-JSmSaTsaJks5tCWzRgaJpZM4RFzGh> .

[pkoutoupis]

XFS is by far the preferred filesystem for these workloads.

In my experience, XFS also suffers with small file data, more specifically on reads. But I imagine that this is not the typical I/O profile hitting these disks anyway. They are probably very write heavy and read less frequently (on an-as-needed basis).

I think rapiddisk is the closest project to accomplishing
a true delayed-write cache because it's already able to do
regular caching that is seamless.

Well, it doesn't yet but let me give this some thought and see how much work it would take (i.e. to batch writes at set intervals).

[pkoutoupis]

XFS is by far the preferred filesystem for these workloads.

In my experience, XFS also suffers with small file data, more specifically on reads. But I imagine that this is not the typical I/O profile hitting these disks anyway. They are probably very write heavy and read less frequently (on an-as-needed basis).

I think rapiddisk is the closest project to accomplishing
a true delayed-write cache because it's already able to do
regular caching that is seamless.

Well, it doesn't yet but let me give this some thought and see how much work it would take (i.e. to batch writes at set intervals).

[syadnom]

These aren’t terribly small files, just many. Write heavy, many directories build up over time. I would estimate XFS 2. better for this particular work load.

…

On Fri, Dec 22, 2017 at 11:06 AM Petros Koutoupis ***@***.***> wrote: XFS is by far the preferred filesystem for these workloads. In my experience, XFS also suffers with small file data, more specifically on reads. But I imagine that this is not the typical I/O profile hitting these disks anyway. They are probably very write heavy and read less frequently (on an-as-needed basis). I think rapiddisk is the closest project to accomplishing a true delayed-write cache because it's already able to do regular caching that is seamless. Well, it doesn't yet but let me give this some thought and see how much work it would take (i.e. to batch writes at set intervals). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJRXZdzuQueiGXHEwaArrn4kKmSxOJtHks5tC--KgaJpZM4RFzGh> .

[syadnom]

These aren’t terribly small files, just many. Write heavy, many directories build up over time. I would estimate XFS 2. better for this particular work load.

…

On Fri, Dec 22, 2017 at 11:06 AM Petros Koutoupis ***@***.***> wrote: XFS is by far the preferred filesystem for these workloads. In my experience, XFS also suffers with small file data, more specifically on reads. But I imagine that this is not the typical I/O profile hitting these disks anyway. They are probably very write heavy and read less frequently (on an-as-needed basis). I think rapiddisk is the closest project to accomplishing a true delayed-write cache because it's already able to do regular caching that is seamless. Well, it doesn't yet but let me give this some thought and see how much work it would take (i.e. to batch writes at set intervals). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJRXZdzuQueiGXHEwaArrn4kKmSxOJtHks5tC--KgaJpZM4RFzGh> .