Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions _data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1073,24 +1073,24 @@ admin-guide-nav:
url: /admin-guide/190_The_syslog-ng_manual_pages/002_loggen_manual
- title: "The pdbtool manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/003_pdbtool_manual
- title: "The secure-logging manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/004_secure_logging_manual
- title: "The persist-tool manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/004_persist-tool_manual
- title: "The {{ site.product.short_name }} manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/005_syslog-ng_manual
- title: "The syslog-ng.conf manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/006_syslog-ng_conf
- title: "The {{ site.product.short_name }} control tool manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/007_syslog-ng_control_tool_manual
- title: "The syslog-ng-debun manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/008_syslog-ng_debun_tool_manual
- title: "The slogencrypt manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/005_slogencrypt_manual
url: /admin-guide/190_The_syslog-ng_manual_pages/009_slogencrypt_manual
- title: "The slogkey manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/006_slogkey_manual
url: /admin-guide/190_The_syslog-ng_manual_pages/010_slogkey_manual
- title: "The slogverify manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/007_slogverify_manual
- title: "The {{ site.product.short_name }} control tool manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/008_syslog-ng_control_tool_manual
- title: "The syslog-debun manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/009_syslog_debun_tool_manual
- title: "The {{ site.product.short_name }} manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/010_syslog-ng_manual
- title: "The syslog-ng.conf manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/011_syslog-ng_conf
- title: "The persist-tool manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/012_persist-tool_manual
url: /admin-guide/190_The_syslog-ng_manual_pages/011_slogverify_manual
- title: "The secure-logging manual page"
url: /admin-guide/190_The_syslog-ng_manual_pages/012_secure_logging_manual
- title: "About us"
url: /admin-guide/200_About/README
subnav:
Expand Down
Binary file modified assets/images/admin-guide/disk-buffer-diagram-normal.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/images/admin-guide/disk-buffer-diagram-reliable.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,14 @@ The syslog-ng-cfg-helper utility has the following options:
capacity-bytes(<number>)
compaction(<yesno>)
dir(<string>)
disk-buf-size(<number>)
capacity-bytes(<number>)
flow-control-window-bytes(<nonnegative-integer>)
flow-control-window-size(<nonnegative-integer>)
front-cache-size(<nonnegative-integer>)
mem-buf-length(<nonnegative-integer>)
mem-buf-size(<nonnegative-integer>)
flow-control-window-size(<nonnegative-integer>)
flow-control-window-bytes(<nonnegative-integer>)
prealloc(<yesno>)
qout-size(<nonnegative-integer>)
front-cache-size(<nonnegative-integer>)
reliable(<yesno>)
truncate-size-ratio(<nonnegative-float>)
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,4 @@ only when:
- one destination is down, and

- the number of messages stored in the disk buffer of the destination
reaches (disk-buf-size() minus mem-buf-size()).
reaches (capacity-bytes() minus flow-control-window-bytes()).
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ destination d_BSD {
network("127.0.0.1"
port(3333)
disk-buffer(
mem-buf-size(10000)
disk-buf-size(2000000)
flow-control-window-bytes(10000)
capacity-bytes(2000000)
reliable(yes)
)
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ destination d_BSD {
network("127.0.0.1"
port(3333)
disk-buffer(
mem-buf-length(10000)
disk-buf-size(2000000)
flow-control-window-size(10000)
capacity-bytes(2000000)
reliable(no)
)
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ To process the messages from an orphan disk-buffer file using a separate
```config
network("10.21.10.20"
disk-buffer(
disk-buf-size(1048576)
capacity-bytes(1048576)
reliable(yes)
dir(/tmp/qdisk/)
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,31 +18,31 @@ Protocol (ALTP). Note that the Advanced Log Transport Protocol is
available only in syslog-ng Premium Edition version 6 LTS. Of course, using the reliable(yes) option introduces a significant performance penalty as well.

Both reliable and normal disk-buffers employ an in-memory output queue
(set in quot-size()) and an in-memory overflow queue (set in
mem-buf-size() for reliable disk-buffers, or mem-buf-length() for normal
(set in front-cache-size()) and an in-memory overflow queue (set in
flow-control-window-bytes() for reliable disk-buffers, or flow-control-window-size() for normal
disk-buffers). The difference between reliable and normal disk-buffers
is that when the reliable disk-buffer uses one of its in-memory queues,
it also stores the message on the disk, whereas the normal disk-buffer
stores the message only in memory. The normal disk-buffer only uses the
disk if the in-memory output buffer is filled up completely. This
approach has better performance (due to fewer disk I/O operations), but
also carries the risk of losing a maximum of quot-size() plus
mem-buf-length() number of messages in case of an unexpected power
also carries the risk of losing a maximum of front-cache-size() plus
flow-control-window-size() number of messages in case of an unexpected power
failure or application crash.

## Size of the queue files

Disk queue files tend to grow. Each may take up to disk-buf-size() bytes
Disk queue files tend to grow. Each may take up to capacity-bytes() bytes
on the disk. Due to the nature of reliable queue files, all the messages
traversing the queue are written to disk, constantly increasing the size
of the queue file.

The disk-buffer file\'s size should be considered as the configured
disk-buf-size() at any point of time, even if it does not have messages
capacity-bytes() at any point of time, even if it does not have messages
in it. Truncating the disk-buffer file can slow down disk I/O
operations, so {{ site.product.short_name }} does not always truncate the file when it
would be possible (see the truncate-size-ratio() option). If a large
disk-buffer file is not desirable, you should set the disk-buf-size()
disk-buffer file is not desirable, you should set the capacity-bytes()
option to a smaller value.

![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:**
Expand Down
20 changes: 10 additions & 10 deletions doc/_admin-guide/080_Log/020_Buffering/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,21 +53,21 @@ no, {{ site.product.short_name }} handles outgoing messages the following way:
the disk-buffer file during {{ site.product.short_name }} reload, restart or stop,
but they cannot be persisted if in the event of power failures, or
if {{ site.product.short_name }} crashes. By default, the output queue can hold 1000
messages (you can adjust this number using the quot-size() option).
messages (you can adjust this number using the front-cache-size() option).

- *Disk-buffer file*: Disk queue. If there is no space left in the
output queue, the message is stored on the disk-buffer file.
Messages stored here are persisted on the disk, even in case of
power failures or if {{ site.product.short_name }} crashes. Using the disk-buffer
file takes considerable amount of disk I/O and processor time. The
size of this queue can be set with the disk-buf-size() option.
size of this queue can be set with the capacity-bytes() option.

- *Overflow queue*: In-memory queue. This queue is used to trigger
flow-control if it is set. The contents of the in-memory overflow
queue are persisted to the disk-buffer file in case of {{ site.product.short_name }}
reload, restart or stop, but they are not persisted in case of power
failures or if {{ site.product.short_name }} crashes. Setting the size of the
overflow queue can be done with the mem-buf-length() option.
overflow queue can be done with the flow-control-window-size() option.

![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:**
Hazard of data loss! In case of normal disk-buffers, the messages stored
Expand All @@ -83,15 +83,15 @@ failures or if {{ site.product.short_name }} crashes.
When you use disk-based buffering, and the reliable() option is set to
yes, {{ site.product.short_name }} handles outgoing messages the following way.

The mem-buf-size() option determines when flow-control is triggered.
After the size of the disk-buffer file reaches (disk-buf-size() minus
mem-buf-size()), messages are written into both the disk-buffer file and
The flow-control-window-bytes() option determines when flow-control is triggered.
After the size of the disk-buffer file reaches (capacity-bytes() minus
flow-control-window-bytes()), messages are written into both the disk-buffer file and
the overflow queue, indicating that flow-control needs to slow down the
message source. These messages are not taken out from the control window
(governed by log-iw-size()), causing the control window to fill up.

If the control window is full, the flow-control completely stops reading
incoming messages from the source. (As a result, mem-buf-size() must be
incoming messages from the source. (As a result, flow-control-window-bytes() must be
at least as large as log-iw-size() times the average message size.)

### Figure 18: Handling outgoing messages in {{ site.product.short_name }} with the reliable disk-buffer option
Expand All @@ -106,18 +106,18 @@ at least as large as log-iw-size() times the average message size.)
faster, because {{ site.product.short_name }} can skip reading from the disk, and
deserializing the message, saving I/O and processor time. By
default, the output queue can hold 1000 messages (you can adjust it
using the quot-size() option).
using the front-cache-size() option).

- *Disk-buffer file*: Disk queue. If there is no space left in the
output queue, the message is stored on the disk-buffer file.
Messages stored here are persisted on the disk, and survive
{{ site.product.short_name }} crash or power failure. Using the disk-buffer file
takes considerable amount of disk I/O and processor time. The size
of this queue can be set with the disk-buf-size() option.
of this queue can be set with the capacity-bytes() option.

- *Overflow queue*: In-memory and disk queue. This queue is used to
trigger flow-control if it is set. Similarly to the output queue, in
case of reliable disk-buffer in addition to storing the message in
memory, it is stored directly in the disk-buffer file as well for
safety reasons. Setting the size of the overflow queue can be done
with the mem-buf-size() option.
with the flow-control-window-bytes() option.
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ destination d_lb_network {
};
destination {
network("myhost1"
disk-buffer(mem-buf-length(10000) disk-buf-size(2000000)));
disk-buffer(flow-control-window-size(10000) capacity-bytes(2000000)));
};
flags(final);
};
Expand All @@ -41,7 +41,7 @@ destination d_lb_network {

destination {
network("myhost2"
disk-buffer(mem-buf-length(10000) disk-buf-size(2000000)));
disk-buffer(flow-control-window-size(10000) capacity-bytes(2000000)));
};
flags(final);
};
Expand Down