Arrayqueue #5
Arrayqueue #5
Conversation
Rust RFC 565 deprecated the need to implement description(). Now we can inherit from fmt::Display and just use to_string() directly.
New method: We use Crossbeam ArrayQueue as a multithreaded buffer. We can write log messages into this buffer from any thread without locking. When the queue is close to full we get a lock to the underlying file, drain the queue, and write to the file. During this time other threads are still able to commit messages to the buffer. Old method. When a log had to be written the thread would get a lock to the Write, build the log entry, write to the Write, release the lock. If multiple threads have to write at the same time it would cause lock contention.
|
@DallasC I am very appreciated for your contribution! Under normal circumstances, developers mainly ignore the performance side of logging. The truth is, the locking may cause hiccups in the multi-thread system, which is harmful for system running. However, this harm could be solved by with small efforts. Your way, in fact, is in a little hard way. But I am happy to review your PR. Let's to see if we can got more interesting solution. |
|
Real good from my eye:
Some could be improve:
Would you like to try some DB works?:) I invite you to join my TensorBase BE project. When you have some nice practices in that, you will come back to find all the problems here are easy to avoid and the solution is very simple and concise to achieve! |
|
Just leave this project alone for while, hope to see you in the TensorBase. |
Still an early draft and wanted to get your opinion on it.
Summary
I used Crossbeam's ArrayQueue to create a multi-threaded buffer. This allows any thread to write log messages into the buffer without having to acquire mutex locks.
When the
ArrayQueueis full the current thread that would overfill:WriterWhile this is happening other threads can still continue to do work and write to the buffer. In the case that logs are being written to the buffer faster than it can be drained then those threads would attempt to acquire the lock for the file i/o and wait. I don't think this should be happening often though.
Comparison Overview
1 Acquire mutex lock
2 check level (warn, err..)
3 construct msg based on config
4 write msg to underlying
5 free lock
1 check level
2 construct msg
3 write msg to ArrayQueue
4. If queue is full this thread gets a
mutex lock, drains the queue, writes
to file. Other threads can still write
during this time.
With the normal simple log there can be lock contention when multiple threads want to write a log entry at the same time. In addition, it is locked not just when writing the log message to the
Writerbut also while the log message is being constructed.With the new version there is no need to acquire a lock when writing a log entry. When the
ArrayQueueis close to being full it gets drained by whatever thread puts it over it's limit. While this is happening other threads can still write without waiting.Benchmarks
I added some simple benchmarks using criterion. There are three benchmarks:
The benchmark spawns threads and on each thread just loops sending a simple log message including the thread# and msg#.
Running the benchmarks yourself
In the
benchesfolder at the top of each file there isconst NTHREADS: u32 = 4;. Right now you have to manually set this in each file to the number threads you want to spawn.After that is set from the root run:
logis where the files from the bench get written to.Right now the logs aren't automatically cleared so don't forget to clear the log folder each time before running
bench.Things still need to do
Resolve #1