/
persistence.xml
289 lines (289 loc) · 20.6 KB
/
persistence.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
<?xml version="1.0" encoding="UTF-8"?>
<!-- ============================================================================= -->
<!-- Copyright © 2009 Red Hat, Inc. and others. -->
<!-- -->
<!-- The text of and illustrations in this document are licensed by Red Hat under -->
<!-- a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). -->
<!-- -->
<!-- An explanation of CC-BY-SA is available at -->
<!-- -->
<!-- http://creativecommons.org/licenses/by-sa/3.0/. -->
<!-- -->
<!-- In accordance with CC-BY-SA, if you distribute this document or an adaptation -->
<!-- of it, you must provide the URL for the original version. -->
<!-- -->
<!-- Red Hat, as the licensor of this document, waives the right to enforce, -->
<!-- and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent -->
<!-- permitted by applicable law. -->
<!-- ============================================================================= -->
<chapter id="persistence">
<title>Persistence</title>
<para>In this chapter we will describe how persistence works with HornetQ and how to configure
it.</para>
<para>HornetQ ships with a high performance journal. Since HornetQ handles its own persistence,
rather than relying on a database or other 3rd party persistence engine, we have been able
to tune the journal to gain optimal performance for the persistence of messages and
transactions.</para>
<para>A HornetQ journal is an <emphasis>append only</emphasis> journal. It consists of a set of
files on disk. Each file is pre-created to a fixed size and initially filled with padding.
As operations are performed on the server, e.g. add message, update message, delete message,
records are appended to the journal. When one journal file is full we move to the next
one.</para>
<para>Because records are only appended, i.e. added to the end of the journal we minimise disk
head movement, i.e. we minimise random access operations which is typically the slowest
operation on a disk.</para>
<para>Making the file size configurable means that an optimal size can be chosen, i.e. making
each file fit on a disk cylinder. Modern disk topologies are complex and we are not in
control over which cylinder(s) the file is mapped onto so this is not an exact science. But
by minimising the number of disk cylinders the file is using, we can minimise the amount of
disk head movement, since an entire disk cylinder is accessible simply by the disk rotating
- the head does not have to move.</para>
<para>As delete records are added to the journal, HornetQ has a sophisticated file garbage
collection algorithm which can determine if a particular journal file is needed any more -
i.e. has all it's data been deleted in the same or other files. If so, the file can be
reclaimed and re-used. </para>
<para>HornetQ also has a compaction algorithm which removes dead space from the journal and
compresses up the data so it takes up less files on disk.</para>
<para>The journal also fully supports transactional operation if required, supporting both local
and XA transactions.</para>
<para>The majority of the journal is written in Java, however we abstract out the interaction
with the actual file system to allow different pluggable implementations. HornetQ ships with
two implementations:</para>
<itemizedlist>
<listitem>
<para>Java <ulink url="http://en.wikipedia.org/wiki/New_I/O">NIO</ulink>.</para>
<para>The first implementation uses standard Java NIO to interface with the file system.
This provides very good performance and runs on any platform where there's a Java 5+
runtime.</para>
</listitem>
<listitem id="aio-journal">
<para>Linux Asynchronous IO</para>
<para>The second implementation uses a thin native code wrapper to talk to the Linux
asynchronous IO library (AIO). In a highly concurrent environment, AIO can provide
better overall persistent throughput since it does not require explicit syncs to
flush operating system buffers to disk. Most disks can only support a limited number
of syncs per second, so a syncing approach does not scale well when the number of
concurrent transactions needed to be committed grows too large. With AIO, HornetQ
will be called back when the data has made it to disk, allowing us to avoid explicit
syncs altogether and simply send back confirmation of completion when AIO informs us
that the data has been persisted.</para>
<para>The AIO journal is only available when running Linux kernel 2.6 or later and after
having installed libaio (if it's not already installed). For instructions on how to
install libaio please see <xref linkend="installing-aio"/>.</para>
<para>For more information on libaio please see <xref linkend="libaio"/>.</para>
<para>libaio is part of the kernel project.</para>
</listitem>
</itemizedlist>
<para>The standard HornetQ core server uses two instances of the journal:</para>
<itemizedlist>
<listitem>
<para>Bindings journal.</para>
<para>This journal is used to store bindings related data. That includes the set of
queues that are deployed on the server and their attributes. It also stores data
such as id sequence counters. </para>
<para>The bindings journal is always a NIO journal as it is typically low throughput
compared to the message journal.</para>
</listitem>
<listitem>
<para>Message journal.</para>
<para>This journal instance stores all message related data, including the message
themselves and also duplicate id caches.</para>
<para>By default HornetQ will try and use an AIO journal. If AIO is not available, e.g.
the platform is not Linux with the correct kernel version or AIO has not been
installed then it will automatically fall back to using Java NIO which is available
on any Java platform.</para>
</listitem>
</itemizedlist>
<para>For large messages, HornetQ persists them outside the message journal. This is discussed
in <xref linkend="large-messages"/>.</para>
<para>HornetQ also pages messages to disk in low memory situations. This is discussed in <xref
linkend="paging"/>.</para>
<para>If no persistence is required at all, HornetQ can also be configured not to persist any
data at all to storage as discussed in <xref linkend="persistence.enabled"/>.</para>
<section id="configuring.bindings.journal">
<title>Configuring the bindings journal</title>
<para>The bindings journal is configured using the following attributes in <literal
>hornetq-configuration.xml</literal></para>
<itemizedlist>
<listitem>
<para><literal>bindings-directory</literal></para>
<para>This is the directory in which the bindings journal lives. The default value
is <literal>data/bindings</literal>.</para>
</listitem>
<listitem>
<para><literal>create-bindings-dir</literal></para>
<para>If this is set to <literal>true</literal> then the bindings directory will be
automatically created at the location specified in <literal
>bindings-directory</literal> if it does not already exist. The default
value is <literal>true</literal></para>
</listitem>
</itemizedlist>
</section>
<section id="configuring.message.journal">
<title>Configuring the message journal</title>
<para>The message journal is configured using the following attributes in <literal
>hornetq-configuration.xml</literal></para>
<itemizedlist>
<listitem id="configuring.message.journal.journal-directory">
<para><literal>journal-directory</literal></para>
<para>This is the directory in which the message journal lives. The default value is
<literal>data/journal</literal>.</para>
<para>For the best performance, we recommend the journal is located on its own
physical volume in order to minimise disk head movement. If the journal is on a
volume which is shared with other processes which might be writing other files
(e.g. bindings journal, database, or transaction coordinator) then the disk head
may well be moving rapidly between these files as it writes them, thus reducing
performance.</para>
<para>When the message journal is stored on a SAN we recommend each journal instance
that is stored on the SAN is given its own LUN (logical unit).</para>
</listitem>
<listitem id="configuring.message.journal.create-journal-dir">
<para><literal>create-journal-dir</literal></para>
<para>If this is set to <literal>true</literal> then the journal directory will be
automatically created at the location specified in <literal
>journal-directory</literal> if it does not already exist. The default value
is <literal>true</literal></para>
</listitem>
<listitem id="configuring.message.journal.journal-type">
<para><literal>journal-type</literal></para>
<para>Valid values are <literal>NIO</literal> or <literal>ASYNCIO</literal>.</para>
<para>Choosing <literal>NIO</literal> chooses the Java NIO journal. Choosing
<literal>AIO</literal> chooses the Linux asynchronous IO journal. If you
choose <literal>AIO</literal> but are not running Linux or you do not have
libaio installed then HornetQ will detect this and automatically fall back to
using <literal>NIO</literal>.</para>
</listitem>
<listitem id="configuring.message.journal.journal-sync-transactional">
<para><literal>journal-sync-transactional</literal></para>
<para>If this is set to true then HornetQ will make sure all transaction data is
flushed to disk on transaction boundaries (commit, prepare and rollback). The
default value is <literal>true</literal>.</para>
</listitem>
<listitem id="configuring.message.journal.journal-sync-non-transactional">
<para><literal>journal-sync-non-transactional</literal></para>
<para>If this is set to true then HornetQ will make sure non transactional message
data (sends and acknowledgements) are flushed to disk each time. The default
value for this is <literal>true</literal>.</para>
</listitem>
<listitem id="configuring.message.journal.journal-file-size">
<para><literal>journal-file-size</literal></para>
<para>The size of each journal file in bytes. The default value for this is <literal
>10485760</literal> bytes (10MiB).</para>
</listitem>
<listitem id="configuring.message.journal.journal-min-files">
<para><literal>journal-min-files</literal></para>
<para>The minimum number of files the journal will maintain. When HornetQ starts and
there is no initial message data, HornetQ will pre-create <literal
>journal-min-files</literal> number of files.</para>
<para>Creating journal files and filling them with padding is a fairly expensive
operation and we want to minimise doing this at run-time as files get filled. By
precreating files, as one is filled the journal can immediately resume with the
next one without pausing to create it.</para>
<para>Depending on how much data you expect your queues to contain at steady state
you should tune this number of files to match that total amount of data.</para>
</listitem>
<listitem id="configuring.message.journal.journal-max-io">
<para><literal>journal-max-io</literal></para>
<para>Write requests are queued up before being submitted to the system execution.
This parameter controls the maximum number of write requests that can be in the
IO queue at any one time. If the queue becomes full then writes will block until
space is freed up. </para>
<para>When using NIO, this value should always be equal to <literal
>1</literal></para>
<para>When using AIO, the default should be <literal>500</literal>.</para>
<para>The system maintains different defaults for this parameter depening on whether
it's NIO or AIO (default for NIO is 1, default for AIO is 500)</para>
<para>There is a limit and the total max AIO can't be higher than what is configured
at the OS level (/proc/sys/fs/aio-max-nr) usually at 65536.</para>
</listitem>
<listitem id="configuring.message.journal.journal-buffer-timeout">
<para><literal>journal-buffer-timeout</literal></para>
<para>Instead of flushing on every write that requires a flush, we maintain an
internal buffer, and flush the entire buffer either when it is full, or when a
timeout expires, whichever is sooner. This is used for both NIO and AIO and
allows the system to scale better with many concurrent writes that require
flushing.</para>
<para>This parameter controls the timeout at which the buffer will be flushed if it
hasn't filled already. AIO can typically cope with a higher flush rate than NIO,
so the system maintains different defaults for both NIO and AIO (default for NIO
is 3333333 nanoseconds - 300 times per second, default for AIO is 500000
nanoseconds - ie. 2000 times per second).</para>
<para>By increasing the timeout, you may be able to increase system throughput at
the expense of latency, the default parameters are chosen to give a reasonable
balance between throughput and latency.</para>
</listitem>
<listitem id="configuring.message.journal.journal-buffer-size">
<para><literal>journal-buffer-size</literal></para>
<para>The size of the timed buffer on AIO. The default value is <literal
>490KiB</literal>.</para>
</listitem>
<listitem id="configuring.message.journal.journal-compact-min-files">
<para><literal>journal-compact-min-files</literal></para>
<para>The minimal number of files before we can consider compacting the journal. The
compacting algorithm won't start until you have at least <literal
>journal-compact-min-files</literal></para>
<para>The default for this parameter is <literal>10</literal></para>
</listitem>
<listitem id="configuring.message.journal.journal-compact-percentage">
<para><literal>journal-compact-percentage</literal></para>
<para>The threshold to start compacting. When less than this percentage is
considered live data, we start compacting. Note also that compacting won't kick
in until you have at least <literal>journal-compact-min-files</literal> data
files on the journal</para>
<para>The default for this parameter is <literal>30</literal></para>
</listitem>
</itemizedlist>
</section>
<section id="disk-write-cache">
<title>An important note on disabling disk write cache.</title>
<warning>
<para>Most disks contain hardware write caches. A write cache can increase the apparent
performance of the disk because writes just go into the cache and are then lazily
written to the disk later. </para>
<para>This happens irrespective of whether you have executed a fsync() from the operating
system or correctly synced data from inside a Java program!</para>
<para>By default many systems ship with disk write cache enabled. This means that even after
syncing from the operating system there is no guarantee the data has actually made it to
disk, so if a failure occurs, critical data can be lost.</para>
<para>Some more expensive disks have non volatile or battery backed write caches which won't
necessarily lose data on event of failure, but you need to test them!</para>
<para>If your disk does not have an expensive non volatile or battery backed cache and it's
not part of some kind of redundant array, and you value your data integrity you need to
make sure disk write cache is disabled.</para>
<para>Be aware that disabling disk write cache can give you a nasty shock performance wise.
If you've been used to using disks with write cache enabled in their default setting,
unaware that your data integrity could be compromised, then disabling it will give you
an idea of how fast your disk can perform when acting really reliably.</para>
<para>On Linux you can inspect and/or change your disk's write cache settings using the
tools <literal>hdparm</literal> (for IDE disks) or <literal>sdparm</literal> or <literal
>sginfo</literal> (for SDSI/SATA disks)</para>
<para>On Windows you can check / change the setting by right clicking on the disk and
clicking properties.</para>
</warning>
</section>
<section id="installing-aio">
<title>Installing AIO</title>
<para>The Java NIO journal gives great performance, but If you are running HornetQ using
Linux Kernel 2.6 or later, we highly recommend you use the <literal>AIO</literal>
journal for the best persistence performance especially under high concurrency.</para>
<para>It's not possible to use the AIO journal under other operating systems or earlier
versions of the Linux kernel.</para>
<para>If you are running Linux kernel 2.6 or later and don't already have <literal
>libaio</literal> installed, you can easily install it using the following
steps:</para>
<para>Using yum, (e.g. on Fedora or Red Hat Enterprise Linux):
<programlisting>sudo yum install libaio</programlisting></para>
<para>Using aptitude, (e.g. on Ubuntu or Debian system):
<programlisting>sudo apt-get install libaio</programlisting></para>
</section>
<section id="persistence.enabled">
<title>Configuring HornetQ for Zero Persistence</title>
<para>In some situations, zero persistence is sometimes required for a messaging system.
Configuring HornetQ to perform zero persistence is straightforward. Simply set the
parameter <literal>persistence-enabled</literal> in <literal
>hornetq-configuration.xml</literal> to <literal>false</literal>. </para>
<para>Please note that if you set this parameter to false, then <emphasis>zero</emphasis>
persistence will occur. That means no bindings data, message data, large message data,
duplicate id caches or paging data will be persisted.</para>
</section>
</chapter>