Add fsyncing bloomfilter

[timothycoleman]

Fixed: Improved Stream Existence Filter flushes

For #3423

While the stream existence filter is flushing (msync) there appears to be a small chance that ordinary writes can be delayed several seconds (highest observed was 25s on ZFS, GCP).

This may be ZFS specific, possibly related (though not identical) to openzfs/zfs#12662 for which reason we recently switched away from memory mapped checkpoints on zfs (#3332)

This PR adds a fsync (as opposed to msync) strategy for the persistent bloom filter. By avoiding the memory mapped file (1) we are no longer calling msync, avoiding the problem if it is ZFS related and (2) we take control of flushing directly so that we can control the rate and ensure it has a low impact on other disk operations regardless

[gregoryyoung]

@timothycoleman are you adding an unbuffered stream version of this as well?

If so is there a significant difference in how the caching needs to work?

[timothycoleman]

@gregoryyoung not for the minute, but once we're on net6 the new random file access stuff in there could have a bearing i think

[gregoryyoung]

There is the current unbufferedstream in our codebase (it exists for a reason :-D), do you want me to spend a bit tomorrow or Friday to see if I can quickly work it in as a spike?

https://github.com/EventStore/EventStore/blob/77d18b6b6b660f68e09630f80ca9a7e9ee695b79/src/EventStore.Core/TransactionLog/Unbuffered/UnbufferedFileStream.cs

Doing via it will also be immediately cross platform which other things may be less than for ... some ... time ...

[timothycoleman]

This is holding up a patch release at the moment so we'll probably take the shortest path to getting that out (i.e. the FileStream at least for the short term) but that said, the UnbufferedFileStream stuff looks interesting and like it might be a pretty easy fit. Feel free to take a look of course, but no promises on when I might get back to it. I'm back on to scavenging after this

[gregoryyoung]

Get the patch out and lets revisit.

I wish we could say :

github add comment eventstore pull3425 72 hours "wake up"

:)

Oh ... and drop me a note re: scavenging. I originally wrote most of that code and know it quite well.

[shaan1337]

it could also be looked up at runtime with sysconf(_SC_PAGESIZE) on linux and GetSystemInfo() on windows with a fallback to the usual default (4096) if there was any problem looking it up

[shaan1337]

if it is configurable, the value 0 could be used for auto-configuration (similar to how it is for other auto-configurable options)

[shaan1337]

note: if it is configurable, it should be a multiple of the cache line size (64) and disk block size (4096)

[gregoryyoung]

With the BloomFilterAccessor which is using off heap memory has it been considered to use buffers from BufferManager?

This would put all of that memory usage through the same known mechanism as other things are using.

This would have both some pros and cons associated and is not by any mean a "this is a great idea that is a definite improvement" type thing, more I am wondering if it was considered.

[timothycoleman]

@gregoryyoung I hadn’t considered the buffer manager, I’ll take a look to find out more about it, thanks for the pointer

the intptr/long thing is because allochglobal takes an intptr and addmemorypressure takes a long. Although, I wonder if we really need both constructors. I’ll check when I’m in the office

[github-actions]

@timothycoleman 👉 Created pull request targeting master: #3438