Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 857 lines (727 sloc) 37.515 kb
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
1 ;;; BKNR Datastore
2
3 ;;;# Introduction
4 ;;;
5 ;;;## The prevalence model
6 ;;;
7 ;;; The BKNR datastore is a persistence solution for Lisp data. It
8 ;;; uses the prevalence model, which is based on the following
9 ;;; assumptions:
10 ;;;
11 ;;; All data is held in RAM.
12 ;;;
13 ;;; Data can be saved to disk at once into a snapshot file and is read
14 ;;; from that file at startup time.
15 ;;;
16 ;;; Changes to persistent data are written to a transaction log file
17 ;;; immediately, which can be replayed to restore all changes that
18 ;;; occured since the last snapshot was saved.
19 ;;;
20 ;;; Every kind of operation that needs to be logged is called a
21 ;;; "transaction", and such transactions are made explicit in the
22 ;;; program code. This is different from object-oriented databases,
23 ;;; where the fundamental transactions are object creation, object
24 ;;; deletion and slot access, which are not special cases in the
25 ;;; prevalence model at all.
26 ;;;
27 ;;; Isolation of transactions is achieved using thread locks. In the
28 ;;; simplest model used by the `mp-store', transactions are serialized
29 ;;; using a global lock.
30 ;;;
31 ;;; The transaction system is responsible for providing replay of
32 ;;; committed transactions after a server crash, but not for rollback
33 ;;; of failed transactions in a running server, except that failing
34 ;;; transactions are simply not logged onto disk. To roll back
35 ;;; transactions at points where exceptions might be excepted, use
36 ;;; ordinary Lisp programming techniques involving `unwind-protect'
37 ;;; and similar.
38 ;;;
39 ;;;## BKNR Datastore Design
40 ;;;
41 ;;; The design of the datastore aims to make explicit the
42 ;;; orthogonality of object system access (unlogged) and logging of
43 ;;; transactions (essentially independent of the object system). The
44 ;;; interface between transaction system and object system is
45 ;;; documented and allows for the implementation of alternative object
46 ;;; systems. For example, the blob subsystem is using the same
47 ;;; interface as the object subsystem.
48 ;;;
49 ;;; Previous versions of the BKNR Datastore allowed the creation of
50 ;;; multiple datastores in a single LISP process. However, this
51 ;;; feature was seldom used, and could be very confusing while
52 ;;; developing applications. The new version of the BKNR Datastore
53 ;;; supports only a single datastore, which is referenced by the
54 ;;; special variable `*STORE*'.
55 ;;;
56 ;;;## BKNR Object Datastore
57 ;;;
58 ;;; In addition to the transaction layer (in the file `txn.lisp'), the
59 ;;; BKNR datastore provides persistent CLOS objects using the
60 ;;; Metaobject Protocol. It provides a metaclass with which slots can
61 ;;; be defined as persistent (stored on snapshot) or transient. The
62 ;;; metaclass also prohibits slot accesses outside transactions,
63 ;;; provides unique IDs for all objects, and provides standard query
64 ;;; functions like `STORE-OBJECTS-WITH-CLASS' and
65 ;;; `STORE-OBJECTS-OF-CLASS'. The object datastore can be seamlessly
66 ;;; combined with BKNR indices and XML import/export.
67
68 ;;;# Obtaining and loading BKNR Datastore
69 ;;;
70 ;;; You can obtain the current CVS sources of BKNR by following the
71 ;;; instructions at `http://bknr.net/blog/bknr-devel'. Add the `experimental'
72 ;;; directory of BKNR to your `asdf:*central-registry*', and load the
73 ;;; indices module by evaluating the following form:
74
d9a215c @hanshuebner Multi-store fixes and tutorial update.
authored
75 (asdf:oos 'asdf:load-op :bknr.datastore)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
76
77 ;;; Then switch to the `bknr.datastore' package to try out the tutorial.
78
79 (in-package :bknr.datastore)
80
81 ;;;# A transaction system example
82
83 ;;; The first datastore we will build is very simple. We have a
84 ;;; counter variable for the store, and this counter variable can be
85 ;;; decremented and indecremented. We want this variable to be
86 ;;; persistent, so decrementing and incrementing it has to be done
87 ;;; through transactions that will be logged by the datastore. We also
88 ;;; define a `:BEFORE' method for the generic function `RESTORE-STORE'
89 ;;; to set the counter to `0' initially. This method will be called
90 ;;; every time the store is created or restored from disk.
91
92 (defclass tutorial-store (mp-store)
93 ((counter :initform 0 :accessor tutorial-store-counter)))
94
95 (defmethod restore-store :before ((store tutorial-store) &key until)
96 (declare (ignore until))
97 (setf (tutorial-store-counter store) 0))
98
99 ;;; The two transactions are declared like normal functions, but using
100 ;;; the `DEFTRANSACTION' macro.
101
102 (deftransaction incf-counter ()
103 (incf (tutorial-store-counter *store*)))
104
105 (deftransaction decf-counter ()
106 (decf (tutorial-store-counter *store*)))
107
108 ;;; When looking at the macro-expanded form of `DEFTRANSACTION', we
109 ;;; see that `DEFTRANSACTION' defines two functions, a toplevel
110 ;;; function that creates a transaction object and calls the method
111 ;;; `EXECUTE' on it, and a function that contains the actual
112 ;;; transaction code and that will be called in the context of the
113 ;;; transaction, and logged to disk.
114
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
115 (PROGN (DEFUN TX-DECF-COUNTER ()
116 (UNLESS (IN-TRANSACTION-P) (ERROR 'NOT-IN-TRANSACTION))
117 (DECF (TUTORIAL-STORE-COUNTER *STORE*)))
118 (DEFUN DECF-COUNTER ()
119 (EXECUTE (MAKE-INSTANCE 'TRANSACTION
120 :FUNCTION-SYMBOL 'TX-DECF-COUNTER
121 :TIMESTAMP (GET-UNIVERSAL-TIME)
122 :ARGS (LIST)))))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
123
124 ;;; The new datastore only supports a single datastore instance per
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
125 ;;; LISP instance. When creating a `STORE' object, the `*STORE*'
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
126 ;;; special variable is modified to point to the datastore. Thus, we
127 ;;; can create our simple datastore by creating an object of type
128 ;;; `TUTORIAL-STORE'. The transaction log will be store in the
129 ;;; directory "/tmp/tutorial-store".
130
d9a215c @hanshuebner Multi-store fixes and tutorial update.
authored
131 (close-store)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
132 (make-instance 'tutorial-store :directory "/tmp/tutorial-store/"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
133 :subsystems nil)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
134 ; Warning: restoring #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
135 ; => #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
136
137 (tutorial-store-counter *store*)
138 ; => 0
139 (incf-counter)
140 ; => 1
141 (incf-counter)
142 ; => 2
143 (decf-counter)
144 ; => 1
145
146 ;;; The three transactions have been logged to the transaction log in
147 ;;; "/tmp/tutorial-store/", as we can see:
148
149 (with-open-file (s "/tmp/tutorial-store/current/transaction-log"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
150 :direction :input)
151 (file-length s))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
152 ; => 126
153 (incf-counter)
154 ; => 2
155 (with-open-file (s "/tmp/tutorial-store/current/transaction-log"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
156 :direction :input)
157 (file-length s))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
158 ; => 168
159
160 ;;; The transaction log is kept in a directory called "current", which
161 ;;; is where the currently active version of the snapshots and log
162 ;;; files are kept. When a datastore is snapshotted, the "current"
163 ;;; directory is backupped to another directory with the current date,
164 ;;; and snapshots are created in the new "current" directory. However,
165 ;;; we cannot snapshot our tutorial datastore, as we cannot snapshot
166 ;;; the persistent data (the counter value).
167
168 (snapshot)
169 ; => Error in function (METHOD SNAPSHOT-STORE NIL (STORE)):
170 ; => Cannot snapshot store without subsystems...
171 ; => [Condition of type SIMPLE-ERROR]
172
173 ;;; We can close the store by using the function `CLOSE-STORE'.
174 *store*
175 ; => #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
176 (close-store)
177 ; => NIL
d9a215c @hanshuebner Multi-store fixes and tutorial update.
authored
178 (boundp '*store*)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
179 ; => NIL
180
181 ;;; The store can then be recreated, and the transaction log will be
182 ;;; read and executed upon restore.
183 (make-instance 'tutorial-store :directory "/tmp/tutorial-store/"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
184 :subsystems nil)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
185
186 ; Warning: restoring #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
187 ; Warning: loading transaction log
188 ; /tmp/tutorial-store/current/transaction-log
189 ; => #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
190 (tutorial-store-counter *store*)
191 ; => 2
192
193 ;;; The store can also be restored in a later LISP session. Make sure
194 ;;; that all the code necessary to the execution of the transaction
195 ;;; log has been loaded before restoring the datastore. A later
196 ;;; version of the datastore will log all the code necessary in the
197 ;;; datastore itself, so that code and data are synchronized.
198
199 ;;;## Debugging the datastore
200 ;;;
201 ;;; By setting the `*STORE-DEBUG*' special variable to `T', the
202 ;;; datastore prints a lot of useful warnings. For example
203
204 ;;; You can also restore to a certain point in time, by specifying the
205 ;;; `UNTIL' argument of `RESTORE-STORE'.
206
207 (setf *store-debug* t)
208 ; => T
209 (restore-store *store*)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
210 ; restoring #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
211 ; loading transaction log /tmp/tutorial-store/current/transaction-log
212 ; executing transaction #<TRANSACTION 21.04.2008 07:08:22 TX-INCF-COUNTER > at timestamp 3417743302
213 ; executing transaction #<TRANSACTION 21.04.2008 07:08:25 TX-INCF-COUNTER > at timestamp 3417743305
214 ; executing transaction #<TRANSACTION 21.04.2008 07:08:26 TX-DECF-COUNTER > at timestamp 3417743306
215 ; executing transaction #<TRANSACTION 21.04.2008 07:08:34 TX-INCF-COUNTER > at timestamp 3417743314
216 ; => NIL
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
217 (tutorial-store-counter *store*)
218 ; => 2
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
219 ; !! Update the timestamp below to correspond to the fist transaction executed above !!
220 (restore-store *store* :until 3417743302) ...
221 ; restoring #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
222 ; loading transaction log /tmp/tutorial-store/current/transaction-log
223 ; executing transaction #<TRANSACTION 21.04.2008 07:08:22 TX-INCF-COUNTER > at timestamp 3417743302
224 ; creating log file backup: /tmp/tutorial-store/current/transaction-log.backup
225 ; truncating transaction log at position 42.
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
226 (tutorial-store-counter *store*)
227 ; => 1
228
229 ;;;## Adding a subsystem
230
231 ;;; Now that we can restore the counter state by loading the
232 ;;; transaction log, we want to add a subsystem to be able to snapshot
233 ;;; the state of the counter. Thus, we won't need to execute every
234 ;;; single incrementing or decrementing transaction to restore our
235 ;;; persistent state.
236
237 ;;; To do this, we have to create a store-subsystem that will be able
238 ;;; to write the counter number to a file and to reload it on restore.
239
240 (defclass counter-subsystem ()
241 ())
242
243 ;;; Three methods are used to interact with the subsystem.
244 ;;; The first method is `INITIALIZE-SUBSYSTEM', which is called after
245 ;;; the store has been created and restored. It is used to initialize
246 ;;; certain parameters of the subsystem. We won't use this method
247 ;;; here, as our subsystem is very simple.
248 ;;; The second method is `SNAPSHOT-SUBSYSTEM', which is called when
249 ;;; the store is snapshotted. The subsystem has to store the
250 ;;; persistent data it handles to a snapshot file inside the current
251 ;;; directory of the store. Our `COUNTER-SUBSYSTEM' writes the current
252 ;;; value of the counter to a file named "counter" in the current
253 ;;; directory of the store (the old directory has been renamed).
254
255 (defmethod snapshot-subsystem ((store tutorial-store)
256 (subsystem counter-subsystem))
257 (let* ((store-dir (ensure-store-current-directory store))
258 (counter-pathname
259 (make-pathname :name "counter" :defaults store-dir)))
260 (with-open-file (s counter-pathname :direction :output)
261 (write (tutorial-store-counter store) :stream s))))
262
263 ;;; Finally, the method `RESTORE-SUBSYSTEM' is called at restore time
264 ;;; to tell the subsystem to read back its persistent state from the
265 ;;; current directory of the store. Our `COUNTER-SUBSYSTEM' reads back
266 ;;; the counter value from the file named "counter". If it can't find
267 ;;; the file (for example if this is the first time that our datastore
268 ;;; is created, the file won't be there, so we issue a warning and set
269 ;;; the counter value to 0.
270
271 (defmethod restore-subsystem ((store tutorial-store)
272 (subsystem counter-subsystem) &key
273 until)
274 (declare (ignore until))
275 (let* ((store-dir (ensure-store-current-directory store))
276 (counter-pathname
277 (make-pathname :name "counter" :defaults store-dir)))
278 (if (probe-file counter-pathname)
279 (with-open-file (s counter-pathname :direction :input)
280 (let ((counter (read s)))
281 (setf (tutorial-store-counter store) counter)))
282 (progn
283 (warn "Could not find store counter value, setting to 0.")
284 (setf (tutorial-store-counter store) 0)))))
285
286 ;;; Now we can close our current store, and instantiate it anew with a
287 ;;; `COUNTER-SUBSYSTEM'.
288
289 (close-store)
290 ; => NIL
291 (make-instance 'tutorial-store :directory "/tmp/tutorial-store/"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
292 :subsystems (list (make-instance 'counter-subsystem)))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
293 ; Warning: restoring #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
294 ; Warning: Could not find store counter value, setting to 0.
295 ; Warning: loading transaction log
296 ; /tmp/tutorial-store/current/transaction-log
297 ; => #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
298 (snapshot)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
299 ; Snapshotting subsystem #<COUNTER-SUBSYSTEM #xE65F866> of #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
300 ; Successfully snapshotted #<COUNTER-SUBSYSTEM #xE65F866> of #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
301 ; => NIL
302 (restore)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
303 ; restoring #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
304 ; Restoring the subsystem #<COUNTER-SUBSYSTEM #xE65F866> of #<TUTORIAL-STORE DIR: "/tmp/tutorial-store/">
305 ; => NUL
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
306
307 ;;;# An object store example
308
309 ;;; The BKNR object datastore is implemented using a special subsystem
310 ;;; `STORE-OBJECT-SUBSYSTEM'. Every object referenced by the store
311 ;;; object subsystem has a unique ID, and must be of the class
312 ;;; `STORE-OBJECT'. The ID counter in the store-object subsystem is
313 ;;; incremented on every object creation.
314 ;;;
315 ;;; All store objects have to be of the metaclass `PERSISTENT-CLASS',
316 ;;; which will ensure the object is referenced in the base indices of
317 ;;; the object datastore, and that slot access is only done inside a
318 ;;; transaction. The subsystem makes heavy use of BKNR indices, and
319 ;;; indexes object by ID and by class.
320
321 ;;; The ID index can be queried using the functions
322 ;;; `STORE-OBJECT-WITH-ID', which returns the object with the
323 ;;; requested ID, `ALL-STORE-OBJECTS' which returns all current store
324 ;;; objects, and `MAP-STORE-OBJECTS', which applies a function
325 ;;; iteratively to each store object. The class index can be queried
326 ;;; using the functions `ALL-STORE-CLASSES', which returns the names
327 ;;; of all the classes currently present in the datastore, and
328 ;;; `STORE-OBJECTS-WITH-CLASS', which returns all the objects of a
329 ;;; specific class (across superclasses also, so
330 ;;; `(STORE-OBJECTS-WITH-CLASS \'STORE-OBJECT)' returns all the
331 ;;; existing store objects.
332
333 ;;;## Store and object creation
334
335 ;;; We can create an object datastore by creating a `STORE' with the
336 ;;; subsystem `STORE-OBJECT-SUBSYSTEM'.
337
d9a215c @hanshuebner Multi-store fixes and tutorial update.
authored
338 (close-store)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
339 (make-instance 'mp-store :directory "/tmp/object-store/"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
340 :subsystems (list
341 (make-instance 'store-object-subsystem)))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
342
343 ; Warning: restoring #<MP-STORE DIR: "/tmp/object-store/">
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
344 ; restoring #<MP-STORE DIR: "/tmp/object-store/">
345 ; Restoring the subsystem #<STORE-OBJECT-SUBSYSTEM #xE63F866> of #<MP-STORE DIR: "/tmp/object-store/">
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
346 (all-store-objects)
347 ; => NIL
348
349 ;;; We can now create a few store objects (which is not very
350 ;;; interesting in itself). Store objects have to be created inside a
351 ;;; transaction so that the object creation is logged into the
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
352 ;;; transaction log. If `MAKE-INSTANCE' is used outside of a
353 ;;; transaction, the datastore will create the object in a new,
354 ;;; separate transaction.
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
355
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
356 (make-instance 'store-object)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
357 ; => #<STORE-OBJECT ID: 0>
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
358 (make-instance 'store-object)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
359 ; => #<STORE-OBJECT ID: 1>
360 (all-store-objects)
361 ; => (#<STORE-OBJECT ID: 0> #<STORE-OBJECT ID: 1>)
362 (all-store-classes)
363 ; => (STORE-OBJECT)
364
365 ;;; Object deletion also has to be done through the transaction
366 ;;; `DELETE-OBJECT', which will log the deletion of the object in the
367 ;;; transaction log, and remove the object from all its indices.
368
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
369 (make-instance 'store-object)
370 ; executing transaction #<TRANSACTION 21.04.2008 08:02:10 MAKE-INSTANCE STORE-OBJECT ID 2> at timestamp 3417746530
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
371 ; => #<STORE-OBJECT ID: 12>
d9a215c @hanshuebner Multi-store fixes and tutorial update.
authored
372 (store-object-with-id 2)
373 ; => #<STORE-OBJECT ID: 2>
374 (delete-object (store-object-with-id 2))
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
375 ; executing transaction #<TRANSACTION 21.04.2008 08:52:14 TX-DELETE-OBJECT 2> at timestamp 3417749534
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
376 ; => T
d9a215c @hanshuebner Multi-store fixes and tutorial update.
authored
377 (store-object-with-id 2)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
378 ; => NIL
379
380 ;;;## Defining persistent classes
381
382 ;;; A more interesting thing is to create our own persistent class,
383 ;;; which we will call `TUTORIAL-OBJECT'.
384
385 (defclass tutorial-object (store-object)
386 ((a :initarg :a :reader tutorial-object-a))
387 (:metaclass persistent-class))
388
389 ;;; We can also use the `DEFINE-PERSISTENT-CLASS' to define the class
390 ;;; `TUTORIAL-OBJECT':
391
392 (define-persistent-class tutorial-object ()
393 ((a :read)))
394
395 ;;; This gets macroexpanded to the following form.
396
397 (DEFINE-BKNR-CLASS TUTORIAL-OBJECT
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
398 (STORE-OBJECT)
399 ((A :READ))
400 (:METACLASS PERSISTENT-CLASS))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
401
402 ;;; The macro DEFINE-BKNR-CLASS is a short form of the macro DEFCLASS,
403 ;;; it expands to the following code. The `EVAL-WHEN' is
404 ;;; there to ensure timely definition of the accessor methods.
405
406 (EVAL-WHEN (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE)
407 (DEFCLASS TUTORIAL-OBJECT
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
408 (STORE-OBJECT)
409 ((A :READER TUTORIAL-OBJECT-A :INITARG :A))
410 (:METACLASS PERSISTENT-CLASS)))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
411
412 ;;; We can now create a few instance of `TUTORIAL-OBJECT':
413
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
414 (make-instance 'tutorial-object :a 2)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
415 ; => #<TUTORIAL-OBJECT ID: 3>
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
416 (make-instance 'tutorial-object :a 2)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
417 ; => #<TUTORIAL-OBJECT ID: 4>
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
418 (make-instance 'tutorial-object :a 2)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
419 ; => #<TUTORIAL-OBJECT ID: 5>
420
421 (store-object-with-id 5)
422 ; => #<TUTORIAL-OBJECT ID: 5>
423
424 (all-store-classes)
425 ; => (STORE-OBJECT TUTORIAL-OBJECT)
426
427 (store-objects-with-class 'tutorial-object)
428 ; => (#<TUTORIAL-OBJECT ID: 3> #<TUTORIAL-OBJECT ID: 4>
429 ; #<TUTORIAL-OBJECT ID: 5>)
430 (store-objects-with-class 'store-object)
431 ; => (#<STORE-OBJECT ID: 0> #<STORE-OBJECT ID: 1>
432 ; #<FOO ID: 2> #<TUTORIAL-OBJECT ID: 3>
433 ; #<TUTORIAL-OBJECT ID: 4> #<TUTORIAL-OBJECT ID: 5>)
434
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
435 ;;; In order to change the slot values of persistent object, the
436 ;;; application needs to be in a transaction context. This can be
437 ;;; done either by invoking a named transaction as above, or by
438 ;;; creating an anonymous transaction. In an anonymous transaction,
439 ;;; all write accesses to persistent objects are logged.
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
440
441 (define-persistent-class tutorial-object2 ()
442 ((b :update)))
443
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
444 (make-instance 'tutorial-object2 :b 3)
445 ; executing transaction #<TRANSACTION 21.04.2008 08:03:27 MAKE-INSTANCE TUTORIAL-OBJECT2 ID 6 B 3> at timestamp 3417746607
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
446 ; => #<TUTORIAL-OBJECT2 ID: 6>
447 (setf (slot-value (store-object-with-id 6) 'b) 4)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
448 ; => Error
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
449 ; Attempt to set persistent slot B of #<TUTORIAL-OBJECT2 ID: 6> outside of a transaction
450 (with-transaction ()
451 (setf (slot-value (store-object-with-id 6) 'b) 4))
452 ; => 4
453 (tutorial-object2-b (store-object-with-id 6))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
454 ; => 4
455
456 ;;;## Object creation and deletion protocol
457
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
458 ;;; Persistent objects have the metaclass `PERSISTENT-CLASS'. They
459 ;;; are created using `MAKE-INSTANCE' just like any other CLOS object.
460 ;;; The datastore allocates a unique ID for the object and initializes
461 ;;; it using the usual CLOS protocol. In order to perform any
462 ;;; additional actions during initialization of the persistent
463 ;;; instance, the `INITIALIZE-INSTANCE' function must be specialized
464 ;;; for the persistent class. In addition, the datastore calls the
465 ;;; `INITIALIZE-TRANSIENT-INSTANCE' for the new object when it is
466 ;;; initially created as well as when it is restored from the
467 ;;; transaction log or snapshot file. It may be used to initialize
468 ;;; transient aspects of the objects, but it must not alter the
469 ;;; persistent state of the object. `INITIALIZE-TRANSIENT-INSTANCE'
470 ;;; does not have access to the initargs used to initially create the
471 ;;; persistent instance.
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
472 ;;;
473 ;;; We can define the following class with a transient and a
474 ;;; persistent slot.
475
476 (define-persistent-class protocol-object ()
477 ((a :update :transient t)
478 (b :update)))
479
480 ;;; We can modify the slot `A' outside a transaction:
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
481 (make-instance 'protocol-object :a 1 :b 2)
482 ; executing transaction #<TRANSACTION 21.04.2008 08:10:49 MAKE-INSTANCE PROTOCOL-OBJECT ID 7 A 1 B 2> at timestamp 3417747049
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
483 ; => #<PROTOCOL-OBJECT ID: 7>
484 (setf (protocol-object-a (store-object-with-id 7)) 2)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
485 ; => 2
486
487 ;;; However, we cannot modify the slot `B', as it is persistent and
488 ;;; has to be changed inside a transaction.
489
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
490 (setf (protocol-object-b (store-object-with-id 7)) 4)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
491 ; => Error
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
492 ; Attempt to set persistent slot B of #<PROTOCOL-OBJECT ID: 7> outside of a transaction
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
493
494 ;;; An object can be removed from the datastore using the transaction
495 ;;; `DELETE-OBJECT', which calls the method `DESTROY-OBJECT' on the
496 ;;; object. Special actions at deletion time have to be added by
497 ;;; overriding `DESTROY-OBJECT'. The basic action is to remove the
498 ;;; object from all its indices.
499
500 ;;;## Snapshotting an object datastore
501
502 ;;; We can snapshot the persistent state of all created objects by
503 ;;; using `SNAPSHOT'.
504 (snapshot)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
505 ; Snapshotting subsystem #<STORE-OBJECT-SUBSYSTEM #xE54991E> of #<MP-STORE DIR: "/tmp/object-store/">
506 ; Successfully snapshotted #<STORE-OBJECT-SUBSYSTEM #xE54991E> of #<MP-STORE DIR: "/tmp/object-store/">
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
507
508 ;;; This will create a backup directory containing the old transaction
509 ;;; log, and the creation of a snapshot file in the "current"
510 ;;; directory.
511
512 (directory "/tmp/object-store/**/*.*")
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
513 ; => (#P"/tmp/object-store/20080421T061210/random-state"
514 ; #P"/tmp/object-store/20080421T061210/transaction-log"
515 ; #P"/tmp/object-store/current/random-state"
516 ; #P"/tmp/object-store/current/store-object-subsystem-snapshot"
517 ; #P"/tmp/object-store/current/transaction-log")
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
518
519 ;;; The snapshot file contains all persistent objects present at
520 ;;; snapshotting time, and the value of their persistent
521 ;;; slots. Further transaction are recorded in a new transaction log.
522
523 ;;;## Adding indices to store objects
524
525 ;;; The object datastore builds upon the functionality of the BKNR
526 ;;; indices system. All store objects are of the metaclass
527 ;;; `INDEXED-CLASS', so adding indices is seamless. Indices are
528 ;;; transient, and are rebuilt every time the datastore is
529 ;;; restored. Adding an index on a transient slot or on a persistent
530 ;;; slot makes no difference.
531
532 (define-persistent-class gorilla ()
533 ((name :read :index-type string-unique-index
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
534 :index-reader gorilla-with-name
535 :index-values all-gorillas)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
536 (mood :read :index-type hash-index
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
537 :index-reader gorillas-with-mood
538 :index-keys all-gorilla-moods)))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
539
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
540 (make-instance 'gorilla :name "lucy" :mood :aggressive)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
541 ; => #<GORILLA ID: 8>
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
542 (make-instance 'gorilla :name "john" :mood :playful)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
543 ; => #<GORILLA ID: 9>
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
544 (make-instance 'gorilla :name "peter" :mood :playful)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
545 ; => #<GORILLA ID: 10>
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
546 (gorilla-with-name "lucy")
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
547 ; => #<GORILLA ID: 8>
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
548 (gorillas-with-mood :playful)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
549 ; => (#<GORILLA ID: 10> #<GORILLA ID: 9>)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
550
551 ;;;## Adding blobs
552
553 ;;; A blob is a Binary Large OBject, that means it is a normal
554 ;;; persistent object with an associated binary data (that most of the
555 ;;; time is quite large). The object datastore supports storing this
556 ;;; large binary data outside the transaction log and the snapshot
557 ;;; file in order not to strain the store memory footprint too much,
558 ;;; and to be able to access the binary data from outside the LISP
559 ;;; session. This can be useful in order to copy the binary data using
560 ;;; the operating system calls directly. Blobs are used to store
561 ;;; images in the BKNR Web Framework (in fact, eboy.com contains more
562 ;;; than 40000 images). They have also been used to store MP3 files
563 ;;; for the GPN interactive DJ.
564 ;;;
565 ;;; In addition to the binary data, a blob object also holds a `TYPE'
566 ;;; and a `TIMESTAMP'. The type of a blob object is a keyword somehow
567 ;;; identifying the type of binary data it stores. For example, for
568 ;;; the images of the eboy datastore, we used the keywords `:JPEG',
569 ;;; `:PNG', `:GIF' to identity the different file formats used to
570 ;;; store images. The timestamp identifies the time of creation of the
571 ;;; blob object (this can be useful to cache binary data of blob
572 ;;; objects in a web server context).
573 ;;;
574 ;;; Stores are implemented in a custom subsystem, which takes as key
575 ;;; argument `:DIRECTORY' the name of a directory where the binary
576 ;;; data of the blob objects is stored as a simple file. This
577 ;;; directory can be further partitioned dynamically by the datastore,
578 ;;; when provided with the argument `:N-BLOBS-PER-DIRECTORY'. The
579 ;;; value of this argument is stored in the directory of the
580 ;;; datastore, so that a future instance of the blob subsystem is
581 ;;; initialised correctly.
582 ;;;
583 ;;; We can now add blob support to our existing object datastore by
584 ;;; adding the blob subsystem to its list of subsystems.
585
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
586 (close-store)
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
587 (make-instance 'mp-store :directory "/tmp/object-store/"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
588 :subsystems (list
589 (make-instance 'store-object-subsystem)
590 (make-instance 'blob-subsystem)))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
591
592 ;;; The blob subsystem provides a few functions and transactions to
593 ;;; work with blobs. To show how to use these functions, we will
594 ;;; define a blob class in our example store. A photo is simply a
595 ;;; binary object with a name.
596
597 (define-persistent-class photo (blob)
598 ((name :read)))
599
600 ;;; A blob can be created using the function `MAKE-BLOB-FROM-FILE',
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
601 ;;; which is a wrapper around `MAKE-INSTANCE' and the function
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
602 ;;; `BLOB-FROM-FILE'. The method `BLOB-FROM-FILE' fills the binary
603 ;;; data of a blob object by reading the content of a file. This
604 ;;; binary data is then stored in a file named after the ID of the
605 ;;; object in the blob root directory of the blob subsystem.
606
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
607 (make-blob-from-file "/tmp/bla.jpg" 'photo :name "foobar"
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
608 :type :jpg)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
609 ; => #<PHOTO ID: 11, TYPE: jpg>
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
610
611 ;;; We can work with the photo object in the same way as when we work
612 ;;; with a normal object. However, we can access the binary data using
613 ;;; the methods `BLOB-PATHNAME', which returns the pathname to the
614 ;;; file in the blob root that holds the binary data of the
615 ;;; object.
616
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
617 (blob-pathname (store-object-with-id 11))
618 ; => #P"/tmp/object-store/blob-root/11"
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
619
620 ;;; The method `BLOB-TO-FILE' and `BLOB-TO-STREAM' write the binary
621 ;;; data of the object to the specified file or stream (the stream has
622 ;;; to be of the type `(UNSIGNED-BYTE 8)'). The macro `WITH-OPEN-BLOB'
623 ;;; is provided as wrapper around the `WITH-OPEN-FILE' macro.
624
625 ;;;## Relaxed references
626
627 ;;; It sometimes happens that a persistent object is deleted while it
628 ;;; still is referenced by another object. This can lead to problems
629 ;;; when snapshotting and restoring the datastore, as the referenced
630 ;;; object is not available anymore.
631 ;;;
632 ;;; When a slot is specified as being a relaxed object reference slot
633 ;;; using the slot option `:RELAXED-OBJECT-REFERENCE', a reference to
634 ;;; an unexistent object can be encoded during snapshot. The object
635 ;;; subsystem issues a warning when a reference to a non-existent
636 ;;; object is encoded. When a reference to a deleted object is decoded
637 ;;; form the snapshot file, a `NIL' value is returned if the slot from
638 ;;; where the object is referenced supports relaxed references. Else,
639 ;;; an error is thrown.
640
641 (define-persistent-class relaxed-object ()
642 ((a :update :relaxed-object-reference t)))
643
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
644 (make-instance 'relaxed-object)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
645 ; => #<RELAXED-OBJECT ID: 12>
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
646 (make-instance 'relaxed-object)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
647 ; => #<RELAXED-OBJECT ID: 13>
648 (with-transaction ()
649 (setf (slot-value (store-object-with-id 12) 'a) (store-object-with-id 13)))
650 ; => #<RELAXED-OBJECT ID: 13>
651 (delete-object (store-object-with-id 13))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
652 ; => T
653 (snapshot)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
654 ; Warning: Backup of the datastore in /tmp/object-store/20080421T064811/.
655 ; While executing: (:INTERNAL (SNAPSHOT-STORE (STORE))), in process worker(1750).
656 ; Snapshotting subsystem #<STORE-OBJECT-SUBSYSTEM #xE6069EE> of #<MP-STORE DIR: "/tmp/object-store/">
657 ; Warning: Encoding reference to destroyed object with ID 13 from slot A of object RELAXED-OBJECT with ID 12.
658 ; While executing: #<STANDARD-METHOD ENCODE-OBJECT (STORE-OBJECT T)>, in process worker(1750).
659 ; Successfully snapshotted #<STORE-OBJECT-SUBSYSTEM #xE6069EE> of #<MP-STORE DIR: "/tmp/object-store/">
660 ; Snapshotting subsystem #<BLOB-SUBSYSTEM #xE6069CE> of #<MP-STORE DIR: "/tmp/object-store/">
661 ; Successfully snapshotted #<BLOB-SUBSYSTEM #xE6069CE> of #<MP-STORE DIR: "/tmp/object-store/">
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
662 ; => NIL
663 (restore)
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
664 ; restoring #<MP-STORE DIR: "/tmp/object-store/">
665 ; Restoring the subsystem #<STORE-OBJECT-SUBSYSTEM #xE6069EE> of #<MP-STORE DIR: "/tmp/object-store/">
666 ; loading snapshot file /tmp/object-store/current/store-object-subsystem-snapshot
667 ; Warning: internal inconsistency during restore: can't find store object 13 in loaded store
668 ; While executing: %DECODE-STORE-OBJECT, in process worker(1754).
669 ; Warning: Reference to inexistent object with id 13 from unnamed container, returning NIL.
670 ; While executing: %DECODE-STORE-OBJECT, in process worker(1754).
671 ; Restoring the subsystem #<BLOB-SUBSYSTEM #xE6069CE> of #<MP-STORE DIR: "/tmp/object-store/">
672 ; loading transaction log /tmp/object-store/current/transaction-log
673 ; executing transaction #<ANONYMOUS-TRANSACTION 21.04.2008 08:48:48 PREPARE-FOR-SNAPSHOT NIL> at timestamp 3417749328
674 (relaxed-object-a (store-object-with-id 12))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
675 ; => NIL
676
677 ;;;# Store internals
678
679 ;;;## Binary data files
680 ;;;
681 ;;; This implementation of the BKNR datastore uses a binary encoding
682 ;;; of Lisp data. The encoding library is used by both the transaction
683 ;;; system and the object system and is mostly independent of
684 ;;; them. Users need not be aware of the details of this encoding,
685 ;;; except that (1) primitive data stored needs to be supported by the
686 ;;; encoding library and (2) user-defined object systems need to
687 ;;; register their own encoder and decoder methods to allow their
688 ;;; objects to be used as part of transaction arguments.
689
690 Function ENCODE (OBJECT STREAM)
691
692 Function DECODE (STREAM) =< OBJECT
693
694 ;;; The `STREAM' must be specialized on `(unsigned-byte 8)'.
695
696 ;;; The object store subsystem uses the encoding library to encode the
697 ;;; persistent state of all the objects in the store. It does this by
698 ;;; first serializing the layout of a class (which is a list of
699 ;;; slot-names), then by first serializing the class and the id of
700 ;;; each object, and finally by serializing the slots of each
701 ;;; object. This two-step system is necessary to correctly serialize
702 ;;; circular of forward references.
703
704 ;;; When the snapshot is loaded, an empty instance of each object is
705 ;;; created, and can be referenced only using the `ID'. After each
706 ;;; object has been instantiated, it can be referenced by another
707 ;;; object. The objects are serialized in the order they have been
708 ;;; created.
709
710 ;;;## Datastore state
711
712 ;;; The store always is in one of the following states:
713
714 ;;; :closed - No store is open, no persistent operations are
715 ;;; possible. It is an error, which will eventually be signaled, to
716 ;;; execute transactions in this state.
717
718 ;;; :restore - The store is currently recovering from a restart
719
720 ;;; :idle - The store has been opened and is prepared to execute
721 ;;; transactions.
722
723 ;;; :transaction - The store is currently executing a transaction
724
725 ;;; Note that all transactions are serialized and thus only one
726 ;;; transaction can be active at any time.
727
728 ;;; :snapshot - The store is currently writing a snapshot.
729
730 ;;;## Transactions
731
732 ;;; Transactions are objects of the class `TRANSACTION', and have a
733 ;;; slot containing the symbol of their transaction function, as well
734 ;;; as a list of the arguments that have to be passed to this
735 ;;; function. When a transaction is executed, a timestamp
736 ;;; of the execution time is stored in the object.
737
738 ;;; Transactions can be grouped by enclosing them with a
739 ;;; with-transaction block. All subtransactions will be logged to the
740 ;;; transaction log as a group and only re-executed on restore time
741 ;;; when the complete group could be read from the file.
742
743 (defun foo-bar ()
744 (with-transaction ()
745 (invoke-transaction-a)
746 (invoke-transaction-b)))
747
748 ;;; The persistent object store further supports the use of anonymous
749 ;;; transactions by automatically converting instance creation and
750 ;;; slot writes to transaction invocations which are atomically
751 ;;; executed upon restore time.
752
753 ;;;## Snapshot and restore procedures
754
755 ;;; When the datastore is snapshotted, the transaction layer ensures
756 ;;; that the store is opened, and that there are subsystems in the
757 ;;; store. Without subsystems, the transaction log is the only way for
758 ;;; the store to achieve persistence, and no snapshot can be made. The
759 ;;; store is then switched to read-only, and a backup directory is
760 ;;; created, containing the current transaction log and previous
761 ;;; snapshot files. This way, the older state of the datastore is not
762 ;;; lost. Then, each subsystem is asked to save its persistent state
763 ;;; by calling the method `SNAPSHOT-SUBSYSTEM'. When an error is
764 ;;; thrown during the snapshot of the subsystems, the backup directory
765 ;;; is renamed to be the current directory, and the store is
766 ;;; reopened.
767
768 ;;; When the datastore is restored, the store is switched to read
769 ;;; only, and each subsystem is asked to restore its persistent
770 ;;; state. Note that the subsystems are restored in the order in which
771 ;;; they are listed in the `SUBSYSTEMS' slot of the store, so that
772 ;;; dependent subsystems are restored last. When an error is thrown
773 ;;; while restoring the subsystems, the store is closed, and already
774 ;;; opened subsystems are closed using the method
775 ;;; `CLOSE-SUBSYSTEM'. After the restoring of all the subsystems, the
776 ;;; transaction log file is read, and each transaction recorded is
777 ;;; executed. This is where the `UNTIL' parameter comes into
778 ;;; play. Transactions that have been executed after the time of
779 ;;; `UNTIL' are discarded.
780
781 ;;;## Filesystem syncing
782
783 ;;; By default, the transaction log file is synced after a transaction
784 ;;; has been executed, so that all the data is correctly written on
785 ;;; disk. However, this can be a major performance stopper when
786 ;;; executing a big batch of transactions (for example, deleting a few
787 ;;; thousands objects). You can disable the mandatory syncing by
788 ;;; executing your transactions inside the form `WITHOUT-SYNC'.
789
790 (without-sync ()
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
791 (execute-a-lot-of-transactions))
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
792
793 ;;;## Snapshotting and restoring the object subsystem
794
795 ;;; Snapshotting and restoring the object subsystem is a bit tricky,
796 ;;; as additional systems come into play. When the object subsystem is
797 ;;; snapshotted using the method `SNAPSHOT-SUBSYSTEM', a snapshot file
798 ;;; containing a binary dump of all the current store objects is
799 ;;; created. First, the layouts of the objects (the name of their
800 ;;; slots) is stored, and minimal information about each object is
801 ;;; stored in the order of their creation (the minimal information
802 ;;; consists of the class of the object, and its ID). After this
803 ;;; information has been stored for each object, the slot values of
804 ;;; the store objects (again in the order of their creation) are
805 ;;; stored in the snapshot file. These two phases are necessary to
806 ;;; allow the snapshotting of circular or forward-referencing
807 ;;; structures.
808
809 ;;; When the object subsystem is restored, all the indices for classes
810 ;;; contained in the store are cleared in order to accomodate for the
811 ;;; new objects. Be very careful when using class indices that are not
812 ;;; related to store objects. The ID counter of the store subsystem is
813 ;;; reset to 0, and the class-layouts are read from the snapshot
814 ;;; file. Then, the minimal information for each object is read, and
815 ;;; an "empty version" of each object is instantiated. Thus, the
816 ;;; objects can be referenced by their ID. Then, the slot values for
817 ;;; each object are read from the snapshot file, references are
818 ;;; resolved (check the section about relaxed references). Finally,
819 ;;; after each slot value has been set, the method
820 ;;; `INITIALIZE-TRANSIENT-INSTANCE' is called for each created
4fc2500 @hanshuebner Merge from anon-transaction-fixes-2 branch. This changeset removes
authored
821 ;;; object.
cd97ded @hanshuebner Replaced trunk with reorganized branch.
authored
822
823 ;;;## Garbage collecting blobs
824
825 ;;; The binary data of deleted blob objects is kept in the blob root
826 ;;; directory by default. If you want to purge the binary data of
827 ;;; deleted objects, you can use the function
828 ;;; `DELETE-ORPHANED-BLOB-FILES'. However, take note that you won't be
829 ;;; able to restore the persistent state anteriour to the deletion of
830 ;;; the blobs, as their binary data is not stored in the transaction
831 ;;; log and not backed up by the snapshot method of the blob
832 ;;; subsystem.
833
834 ;;;## Schema evolution in the datastore
835
836 ;;; The transaction log only stores when a transaction is called, and
837 ;;; with which arguments. However, it doesn't store the definition of
838 ;;; the transaction itself. When the transaction definition is
839 ;;; changed, the transaction log may be restored in a different way,
840 ;;; according to the changes made in the code.
841 ;;;
842 ;;; In the same way, class definition changes are not recorded in the
843 ;;; transaction log. When a class definition is changed (for example a
844 ;;; slot initform is changed), the existing instances of the class are
845 ;;; updated accordingly. However, when the snapshot is restored in a
846 ;;; future session, the objects may be different than those created at
847 ;;; the last restore.
848 ;;;
849 ;;; The only way to cleanly upgrade transaction definitions and class
850 ;;; definitions is to make a snapshot after the changes have been
851 ;;; made. In a future version of the datastore, we hope to store all
852 ;;; the application sourcecode, so that a restore to a certain point
853 ;;; in time does not depend on the latest version of the code. The
854 ;;; object subsystem warns when a class definition is changed, and
855 ;;; urges the developer to make a snapshot of the database. Please be
6cf7c53 @hanshuebner Update tutorials to reflect reality better
authored
856 ;;; careful, this can be a pretty tricky source of bugs.
Something went wrong with that request. Please try again.