Feature Request: Ability to prevent pgbackrest from impacting heavily OS disk block cache

[Soulou]

Something which has been already raised in some issues like #822

PgBackRest is reading/writing a lot on disk and for large databases, it simply nuke the OS disk cache. However these data are not really useful in the cache.

In the following case: 16GB box, with a PG with ~300GB of data on disk, when running a pgbackrest backup, it happens that the load of the box gets really high because of the kswapd process of the kernel starting to move pages around, especially pages potentially being swapped out.

From my point of view, there is no problem in preventing pgbackrest from using the disk cache, to keep the useful data from PG into the OS cache, and use a little more disk. At least I don't take the risk seeing the server load average getting to hundreds.

There are two ways to do this usually:

• Using the flag O_DIRECT when opening a file (usually not advised)
• Using the syscall fadvise on the read/write blocks: https://linux.die.net/man/2/fadvise

For instance the dd command is using fadvise when the "nocache" flag is enabled: https://github.com/coreutils/coreutils/blob/master/src/dd.c#L1121

Seeing such configuration option in PgBackRest would be absolutely great.

Thanks a lot

[dwsteele]

Agreed, this is a good idea. We've been considering it for a while but wanted to wait for the C migration to be complete before implementing.

Thank you for the link to dd. This seems to be exactly the implementation we would want. I prefer this to using O_DIRECT as I believe it is more portable based on my research.

Does that sound good?

[Soulou]

Thanks @dwsteele for your answer, I'm glad it has been considered something useful you, it sounds absolutely great to me, I can't wait since it's quite a constant annoyance (we're a PostgreSQL hosting platform)

Do you have any idea of the ETA about the C migration? I've followed that almost all commands have been translated, should we hope this task will be engaged in 1 months, 6 months, 1 year, more? Thanks a lot

[sfrost]

Greetings,

The migration to C has been complete for a couple of months now (it was finished back in January), so that's over and done with and isn't a blocker any more. As for when we'll have a chance to implement this particular feature request, that depends on the ongoing workload involving bug fixes and maintenance, plus other feature requests that we already have and things we'd like to see done to move pgBackRest forward.

Thanks,

Stephen

[Soulou]

Ok great, thanks @sfrost for your answer, I wasn't sure the C migration was over, nicely done. We'll keep following the project closely!

[Soulou]

@sfrost would it be possible to have an update about the feasibility of this change? Thanks a lot!

[maherbeg]

We've noticed that pgbackrest based backups do indeed increase the file system cache usage and would be really interested in this feature.

Some of our users running pgbackrest backups have been seeing connection errors from their application to their postgres server while the backup is running. We haven't quite figured out why this is happening, but the memory usage and our vm.overcommit settings might be a culprit.