Make SF possible to read syzygy tablebases without RAM caching

[Coolchessguykevin]

Dear developers! I would like to raise an important problem with the use of SF+ syzygy tablebases in the endgames.

As you all know, wherever the tablebases are located (hard disk, ssd disk), when they are used (reading) by engine, they are very much loaded (caching) into the RAM (this is how Windows usually works with frequently accessing files). The fewer pieces we have on the board, the greater the use (TB hits) of syzygy tablebases. It is obvious.

If we use only 5-men, then there is no problem: these bases will easily fit into the RAM. If we use 6-men, then it may already be have problems: on computers, where there is a small amount of RAM, its shortage will begin when analyzing endgames (as a result, when filling, a system memory swap begins). Huge and often swap = SSD kill (my swap pagefile file is on ssd of course).

However, if we have RAM of 92 gigabytes or 128 and more, then there is no problem. All 6-men can easily fit in there.

But as soon as 7-men syzygy tablebases appeared, the problem of tablebases caching became particularly acute. I have SSD disk and only a few hundred gigabytes of 7-men bases. When SF analyzing endgame positions (the disk read speed reaches from several tens of megabytes to several hundred megabytes per second), my RAM is very quickly filled with cached data and it happens that the system stops responding (swap) very soon. This can be seen through the system resource monitor RAM data and disc work.

That is, suppose that even if I had 512 gigabytes of RAM, this would not help if I theoretically had a complete set of 7-men tablases (this is 8.5 terabytes, WDL only of course) and use them in endgames analyzes. Windows will try to cache hundreds of tablebases gygabytes.

My request is the following: is it possible to give the SF the ability to read tablebases without RAM caching? I suppose to do this through a separate UCI option in the parameters, since when using 5-6-men tables, caching is useful for speeding up the engine work (especially when tablebases located on slow hard drive). The user can choose the desired setting (check box) at his own discretion.

I suppose FILE_FLAG_NO_BUFFERING flag for CreateFile() would help in this issue:

https://support.microsoft.com/ru-ru/help/99794/info-file-flag-write-through-and-file-flag-no-buffering

Sincerely.

[noobpwnftw]

OS file cache is never supposed to use any swap/pagefiles space, instead, they should only consume free RAM and will move out of the way when memory allocation happens. Problem you have is probably due to SF memory-mapping those TB files and thus they are counted as allocated memory space but not cache. To fix that we would need to:

1. use seek & read instead of mapping, or
2. close those file mappings more often.

However, both may be very inefficient.

Probably we should introduce a parameter that controls such behavior somehow.

[DragonMist]

Some discussion on this subject from the past (there is more, most likely, it was discussed several times iirc) https://groups.google.com/d/topic/fishcooking/kto46FKCGgo/discussion

[vondele]

out of curiosity, since some of the 7men files are >100GB, is it required to have at least that much RAM+swap so that mmaping them is a good idea?

[vdbergh]

@vondele

The file on disk serves as backing store, so no swap should be required.

It seems that the problems people are reporting with mmap'ed syzygy files (also on TalkChess) are a windows issue. It would be interesting to understand it.

[sf-x]

Windows always does 32kb reads for mmapped files, leading to 8x more churn of RAM than necessary.
Possible solution for complainers is to use a more suitable OS.

[syzygy1]

The cheap solution is to disable the page file. This forces Windows to release RAM used for caching TB data when memory is needed (instead of paging other programs to the page file), which is exactly what we want. This solution may not be acceptable to everyone. (But do most users really need a page file if they already have many many GBs of physical RAM?)

A more complicated solution is to let SF regularly deallocate all TB files. This can already be done manually by first setting SyzygyPath to empty or a path that does not contain TB files and setting SyzygyPath back to the correct path.

If the TBs are deallocated while the engine is searching, the engine will likely crash. But it is relatively easy to fix this. Every search thread should get its own lock that is taken when the TB probe code is entered. A cleanup thread could then, say once a minute, take the locks of all search threads, deallocate the TBs, and release the locks.

Deallocating the TBs will not necessarily release the TB data from RAM, but it will make the OS more willing to release that data (because it is no longer part of the engine process's address space, until it is mapped back in of course).

I don't know how well this will work. For example, if 10GB of RAM is caching data from one table and the engine accesses a single position in that table immediately after a deallocation, it might be that the whole 10GB will again be effectively locked against being mapped out.

Ideally, there would be a flag to use when mmap()ing (or a system setting) that tells the OS to prefer releasing the mmap()ed data over writing data to disk.

@vondele
Luckily, even Windows does not read the whole mmap()d file into RAM.

I don't think Linux is free of this problem. (It might be less severe on Linux but even that I am not sure about. I can only say that I have no problems with 6-piece TBs on my 64 GB machine.)

[syzygy1]

I guess the problem is not so much that Windows (and Linux) are so stupid that they prefer to page out dirty pages over reclaiming clean pages, but that Windows (and Linux) start to write dirty pages to swap to clean them when memory pressure increases.

So it's not totally irrational but still unfortunate behaviour in this particular use case.