Repository navigation
Execution frequency of the run_periodic_tasks function for the Database sink implementation #956
|
I'm working on a custom database sink implementation and I have one question: how often is run_periodic_tasks actually called from the background thread? The reason I'm asking is that with every new write_log call I'm planning (as also suggested in the CustomSink implementation in the official docs) to cache or queue the incoming logs and sometime later perform bulk insert since that's more efficient. One option is to check in write_log if number of the cached logs has reached some predefined limit and perform database bulk insert then. The main concern here is that there are some cached logs but the write_log has not been called for quite some time that they will remain "trapped" in memory and that's where the run_periodic_tasks comes into play - but the question is, how often is this function actually called. I couldn't find something about that in the docs so I do not know how often that's gonna be executed and can this be a good place to solve this issue about pending logs... I've read some suggestions within the CustomSink implementation but still some more input could be useful. |
Replies: 5 comments 1 reply
|
With the default backend options, once the backend is idle it then sleeps for For a database sink, I would not rely on every Also note that For a database sink, the usual pattern would be: One caveat: |
|
Sorry for late reply - I had to rethink little bit about your answer and check out how does this story fit into our use case. I think the most "problematic" function is kinda flush_sink since it's difficult to distinguish between periodical and the very last one flush_sink call. Thus, I have the following approach:
To distinguish between periodical and very last call of the flush_sink function, I will use my LoggingGuard RAII wrapper. This wrapper will in its own destructor flush all sinks to avoid any potential issues when unloading DLL on WIN (that's recommendation I got from the docs and some of the previous discussions on who to resolve the issue that some pending logs might not be logged before the DLL has been unloaded). Before I'm about to call the flush_log in my LoggingGuard destructor I will simply iterate through all the database sinks and I will set the flag for them (something like enable_final_flush) which will internally prepare my database sinks for the final flush_sink. Does this sound like a reasonable way how to distinguish between periodical and very last call of the flush_sink function? Is calling flush_sink function also thread safe, since that's executed periodically from the backend thread but it's also exposed to the outside world and can be called from any thread at any point in time? Does this flag (final_flash) which I have my database sinks to distinguish between periodical and very last call of the flush_sink has to be something like std::atomic final_flush_ {false}; or the regular boolean is enough? I guess that depends on the answer from the previous question... |
|
My flush_sink function should look something like this: Please note that |
|
Yes, that approach is reasonable. Quill’s own So However, std::atomic<bool> final_flush_{false};
Then read it with .load() in flush_sink().
One detail in your example: in the non-final path, don’t call start_bulk_insert() unconditionally just because batches_ is non-empty. If the first batch may be partial, check your batching policy first:
if (final_flush_.load(std::memory_order_acquire))
{
while (!batches_.empty())
{
start_bulk_insert();
}
}
else if (has_ready_batch())
{
start_bulk_insert();
} |
|
The SinkFlushReason added in 12.2.0 should help with the above |
run_periodic_tasks()is not driven by a separate fixed timer. It is called by the backend worker when the worker reaches its idle/maintenance part of the poll loop: after it has attempted to read frontend queues and there are no cached transit events to process.With the default backend options, once the backend is idle it then sleeps for
BackendOptions::sleep_duration, which defaults to100us, so in an idle applicationrun_periodic_tasks()can be called roughly once per backend wake cycle. This is not a hard real-time interval: OS scheduling, backend work, sink work, and configured options affect it. Ifsleep_durationis set to0, it may be called very frequently; ifsleep_durationis lo…