Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 387 lines (305 sloc) 13.945 kb
11ef942 @cruppstahl DAM is no longer persistent (changes compile, but are not yet tested)
authored
1
14c2c75 @cruppstahl git-svn-id: svn+ssh://hamsterdb.com/home/chris/repos/hamsterdb/trunk@…
authored
2 I Am Legend:
8f5a9a1 @cruppstahl added legend
authored
3
14c2c75 @cruppstahl git-svn-id: svn+ssh://hamsterdb.com/home/chris/repos/hamsterdb/trunk@…
authored
4 Items are sorted by priority (highest on top).
5 o a pending TODO item (for the current release)
6 . a pending TODO item (for future releases)
8f5a9a1 @cruppstahl added legend
authored
7 x a finished TODO item
8
9 -----------------------------------------------------------------------------
2105ba0 new TODO file
Christoph Rupp authored
10 This Branch Is About Integrating The hamsterdb2 Functionality!!!!!
304aac2 updating visual studio projects to 2009; added projects for server an…
U-WINDOOF\chris authored
11 -----------------------------------------------------------------------------
34805bf @cruppstahl refactoring; introduced Environment class. no functional changes
authored
12 The big headline is:
2105ba0 new TODO file
Christoph Rupp authored
13 As a user i want to run many Transactions in parallel with high performance.
6bed4c8 @cruppstahl the Page class is now fully C++-ified
authored
14 I'm using multiple threads b/c my CPU has multiple cores, and expect hamsterdb
15 to scale with the number of cores.
2105ba0 new TODO file
Christoph Rupp authored
16 ==============================================================================
17
96f7517 @cruppstahl updated TODO file
authored
18 x update doxygen documentation on the webpage
19
20 x update prebuilt libraries on the webpage (include boost)
21
22 x add Ger's freelist patch
3c4fca6 @cruppstahl updated TODO
authored
23
0210a26 @cruppstahl Fixed a memory leak in the test
authored
24 x unittest: valgrind reports leaks in the test
96f7517 @cruppstahl updated TODO file
authored
25 ==22694== 150 bytes in 5 blocks are definitely lost in loss record 1 of 6
26 ==22694== at 0x4C28B35: operator new(unsigned long)
27 (vg_replace_malloc.c:261)
28 ==22694== by 0x575B058: std::string::_Rep::_S_create(unsigned long, unsigned
29 long, std::allocator<char> const&) (in
30 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16)
31 ==22694== by 0x575B249: std::string::_M_mutate(unsigned long, unsigned long,
32 unsigned long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16)
33 ==22694== by 0x575B3EB: std::string::_M_replace_safe(unsigned long, unsigned
34 long, char const*, unsigned long) (in
35 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16)
36 ==22694== by 0x422671: LogEntry::LogEntry(unsigned long, unsigned long,
37 unsigned int, unsigned short, char const*) (journal.cpp:46)
38 ==22694== by 0x42C809: JournalTest::iterateOverLogMultipleEntryTest()
39 (journal.cpp:587)
40
28576d6 @cruppstahl Fixed a dotnet unittest regression (thanks, Jason)
authored
41 x email from Jason: Great product as usual. 20.2 seems to have broken the .NET
96f7517 @cruppstahl updated TODO file
authored
42 unit test "GetError" in DatabaseTest.cs. It ultimately calls
43 ham_get_error which causes an exception
c8e331e @cruppstahl temporary key->data and record->data storage is now managed by the Tr…
authored
44
c3d8264 @cruppstahl Backend is now C++-ified
authored
45 x btree_read_key: calls db->get_extended_key, which should use an
46 existing ByteArray to avoid memory allocations
47 - no, cannot because the pointer is cached
6922cec @cruppstahl minor fixes and valgrind suppressions
authored
48
c3d8264 @cruppstahl Backend is now C++-ified
authored
49 x db_erase_txn: calls check_erase_conflict, which does a btree lookup.
c8e331e @cruppstahl temporary key->data and record->data storage is now managed by the Tr…
authored
50 but the lookup then happens again when doing the final erase, or not?
c3d8264 @cruppstahl Backend is now C++-ified
authored
51 - no, everything is fine
52
1d3cd97 @cruppstahl updated TODOs
authored
53 x extend the FAQ and the tutorial
54 x thread safety
55 x rec.data scope
56
57 x extend the documentation with information about the new Transactions
58
59 o Mark reported an error after surprise-shutdown
60 o with or without logging?
61 o get access to the corrupt file
62 o ...
63
64 o why do tests with 20 threads fail with oom when using mmap? the cache limits
65 should still work
66
a4ffa37 @cruppstahl Updated TODO
authored
67 o look at Peter's mail - sometimes, ham_env_create_db returns
4862d1f @cruppstahl device refactoring (done)
authored
68 HAM_INTERNAL_ERROR (only win32, C++ API, Windows 7)
a4ffa37 @cruppstahl Updated TODO
authored
69
a8926a4 @cruppstahl minor cleanups
authored
70 o more fine-grained locking
66e5f26 @cruppstahl cache refactoring
authored
71 x cache
72 x make sure it's perfectly abstracted
73 x protect with mutex
3794f98 @cruppstahl extkey-cache refactoring
authored
74 x extkey-cache
75 x make sure it's perfectly abstracted
76 x protect with mutex
5bca10f @cruppstahl log refactoring
authored
77 x log
78 x make sure it's perfectly abstracted
79 x protect with mutex
3a87831 @cruppstahl journal refactoring
authored
80 x journal
81 x make sure it's perfectly abstracted
82 x protect with mutex
0c41058 @cruppstahl device refactoring (WIP)
authored
83 x changeset
740d85b @cruppstahl changeset refactoring
authored
84 x make sure it's perfectly abstracted
85 x protect with mutex
0c41058 @cruppstahl device refactoring (WIP)
authored
86 x device
87 x make sure it's perfectly abstracted
4862d1f @cruppstahl device refactoring (done)
authored
88 x protect with mutex
d76cfb8 @cruppstahl updated TODO items
authored
89 o freelist
90 o remove legacy code (1.x)
91 o create c++ class
4862d1f @cruppstahl device refactoring (done)
authored
92 o freelist class takes ownership of freelist_cache (from device)
d76cfb8 @cruppstahl updated TODO items
authored
93 o protect with mutex
94 o db
95 o completely hide the local/remote implementation in the database
96 o move more members from Database to the Implementation classes
97 o improve abstraction
98 o protect with mutex
99 o env
100 o completely hide the local/remote implementation in the Environment
101 o move more members from Environment to the Implementation classes
102 o improve abstraction
103 o backend
104 o replace with C++ class and inheritance
105 o BtreeBackend does not require to store in little/big-endian!
106 o remove be_set_dirty, replace with be->_fun_flush()
107 o btree has keydata1, keydata2 - why? better use DynamicArray?
108 o improve abstraction
109 o assert that the Database is locked
110 o statistics
111 o split into env/db/freelist
112 o remove unused code
113 o protect env and freelist with mutex
114 o in db: assert that the db is locked
115 o txn
116 o convert to clean c++ code
117 o protect with a mutex
118 o when changing the transaction queue: assert that the env is locked
119 o configuration/env-header-page
120 o create configuration object
121 o protect with mutex
122 o file filters -- why? the filter list is not modified in the selected
123 functions; but it needs to be documented that it has to be
124 threadsafe, therefore verify that the AES filter *is* threadsafe
125 o convert to C++ visitor pattern?
0c41058 @cruppstahl device refactoring (WIP)
authored
126 o page
127 o do we also need to lock pages?? or are they locked implicitly by
128 adding them to the changeset? if yes then make sure that this is
129 failsafe! and assert that there can be only one changeset at a
130 time
131 o changeset::contains is not thread safe
132 only used in Cache::purge
133 o the pages are only used in the btree (with one exception: header
134 page); should the cache therefore be limited to btree functions?
135 then purge and other stuff can also move to the btree level and
136 everything gets simplified
d76cfb8 @cruppstahl updated TODO items
authored
137
138 o all other functions/modules: assert that the env-mutex is locked!
139
a8926a4 @cruppstahl minor cleanups
authored
140 o use read/write lock for Environment
141 o introduce a new lock in the Database
142 o the following functions lock the Environment for reading, but the
143 Database for writing:
144 o ham_insert
145 o ham_find
146 o ham_erase
147 o ham_cursor_*
148 o any others?
66e5f26 @cruppstahl cache refactoring
authored
149 o make sure that the cache does not flush pages that are currently locked
150 by any Transaction
c3d8264 @cruppstahl Backend is now C++-ified
authored
151
152 o also remove locking from C# and Java APIs
c8e331e @cruppstahl temporary key->data and record->data storage is now managed by the Tr…
authored
153
a8926a4 @cruppstahl minor cleanups
authored
154 . look at ssb's clang output and try to integrate it into the build process
155
156 . need a way to graphically display performance metrics
1d3cd97 @cruppstahl updated TODOs
authored
157 o output in a text file
158 version/configuration/test-id
159 each test is run in multiple threads
160 performance-metrics based on time(), not real duration
161 memory-metrics
162 disk size
163 same for berkeleydb
164 o skript that generates a static html page from this file,
165 w/ graphs over the different versions
166
c8e331e @cruppstahl temporary key->data and record->data storage is now managed by the Tr…
authored
167 o implementation of hamsterdb should move into a namespace; otherwise
168 there are conflicts if users have a C++ class called Database or
169 Environment etc
170
f5fc536 @cruppstahl code is now threadsafe; some tests are still missing though
authored
171 o clean up the "close" functions
172 o ham_close - move all functionality to Database::close
173 o ham_env_close - move all functionality to Environment::close
174 o ~Database: call close(), then simplify all code
175 o ~Environment: call close(), then simplify all code
176 o Cursor::close: currently (nearly) empty; merge with ~Cursor and
177 Database::close_cursor()
178
79d4cae @cruppstahl updated TODO items
authored
179 o python API - update and integrate
6922cec @cruppstahl minor fixes and valgrind suppressions
authored
180 o rewrite with boost::python??
79d4cae @cruppstahl updated TODO items
authored
181 o also add to win32 package
182
d76cfb8 @cruppstahl updated TODO items
authored
183 <<<<<<< Updated upstream
1b9a579 @cruppstahl data pointers are now stored in thread local storage
authored
184 o continue with c++-ification of db.h/db.cc
185
186 o c++-ify the backend
c3d8264 @cruppstahl Backend is now C++-ified
authored
187 o replace with C++ class and inheritance
188 o BtreeBackend does not require to store in little/big-endian!
1b9a579 @cruppstahl data pointers are now stored in thread local storage
authored
189 o remove be_set_dirty, replace with be->_fun_flush()
c3d8264 @cruppstahl Backend is now C++-ified
authored
190 o btree has keydata1, keydata2 - why? better use DynamicArray?
d76cfb8 @cruppstahl updated TODO items
authored
191 =======
192 >>>>>>> Stashed changes
79d4cae @cruppstahl updated TODO items
authored
193
6d40855 @cruppstahl always flush transactions before closing the database
authored
194
195
196
197
198 o move flushing of transactions in background
199 o new flag HAM_DISABLE_ASYNC_COMMITS
200 o need new test: n threads; each thread inserts keys [i*n, i*(n+1))
d76cfb8 @cruppstahl updated TODO items
authored
201 o make sure that each Database accesses its own pages; i.e. do not share
202 blob pages between databases. Each database stores the address of the
203 previously used blob page in order to reduce freelist access
204 o when flushing and values are written to multiple databases: write them
205 in parallel
2885e54 @cruppstahl More C++-ification of the Environment class; no functional changes
authored
206
79d4cae @cruppstahl updated TODO items
authored
207 o need a function to get the txn of a conflict (same as in v2)
208 ham_status_t ham_txn_get_conflicting_txn(ham_txn_t *txn, ham_txn_t **other);
5fac4f3 @cruppstahl updated TODO items
authored
209 oder: txn-id zurückgeben?
79d4cae @cruppstahl updated TODO items
authored
210 o also add to c++ API
211 o add documentation (header file)
212 o add documentation (wiki)
8bcfa94 @cruppstahl updated TODO file
authored
213
214 o recovery: re-create pending transactions (if required)
215 o needs a function to enumerate them
216
6d40855 @cruppstahl always flush transactions before closing the database
authored
217 o c++-ify the transaction
218
219
220
221
222
223
224 o allow transactions in-memory
225
226 o allow transactions w/o journal
227
228 o allow transactions w/o recovery
229
230 o rename HAM_WRITE_THROUGH to HAM_ENABLE_FSYNC
231
232 o new node format:
233 iiikkkkkkkkkrrrrrrrr
234
235 iii: fixed size index (skip-list)
236 kkk: keys
237 rrr: records
238 keys can have different compressions (bitmap, suppress null, ...)
239 records are compressed; will be multiplied by BLOCKSIZE (32)
240 each key consists of { char flag; short record_id; char data[1]; }
241 where record_id is an offset into rrrrrrrrr
242 keys are sorted lazily (i.e. inserted at the end and only sorted
243 when flushed to disk or when searched
79d4cae @cruppstahl updated TODO items
authored
244
8bcfa94 @cruppstahl updated TODO file
authored
245 . android port (needs new java api) in /contrib directory (it's on a separate
246 branch)
247
3412f64 @cruppstahl updated TODO list; started with removing idempotent operations from t…
authored
248 . new test case for cursors
249 insert (1, a)
250 insert (1, b) (duplicate of 1)
251 move (last) (-> 1, b)
252 insert (1, c)
253 move (last) (-> 1, c)? is the dupecache updated correctly?
254
255 . look for someone who can write a PHP or Perl or Ruby wrapper
e6f6f92 @cruppstahl updated TODO file
authored
256
8bcfa94 @cruppstahl updated TODO file
authored
257 . implement support for partial keys
e6f6f92 @cruppstahl updated TODO file
authored
258
dbd7802 @cruppstahl ham_db_t is now C++-ified (WIP)
authored
259 . test with tcmalloc; if it works then also use it in the master branch, but
260 make sure that memory consumption does not increase significantly
3412f64 @cruppstahl updated TODO list; started with removing idempotent operations from t…
authored
261
b2ab1a5 @cruppstahl updated TODO file regarding performance improvements
authored
262 . there are a couple of areas where a btree cursor is uncoupled, just to
263 retrieve the key and to couple the txn-key. that's not efficient
264 db.c:__btree_cursor_points_to
265 db.c:__compare_cursors
266 txn_cursor.c:cursor_sync
267 txn_cursor.c:cursor_overwrite
268 o move to a separate function
269 o try to optimize
e6f6f92 @cruppstahl updated TODO file
authored
270
4ee7a3d @cruppstahl cleanups of the cursor code; no functional changes
authored
271 . hash-table.h: the foreach/remove_if/visitor pattern is clumsy. use
272 functor or class w/ operator() instead
273 . changeset: use a generic hash table for fast lookup (but leave list in place
274 for everything else)
275 . cache: use a generic hash table
276
6302799 @cruppstahl updated TODO file
authored
277 . add tests to verify that the cursor is not modified if an operation fails!
278 (in cursor.cpp:LongTxnCursorTest are some wrapper functions to move or
279 insert the cursor; that's a good starting point)
280
dbd7802 @cruppstahl ham_db_t is now C++-ified (WIP)
authored
281 . the whole c++ protocol should be c++-ified
f4dbd73 @cruppstahl inlining a couple of functions to improve performance
authored
282
6302799 @cruppstahl updated TODO file
authored
283 . move the whole configuration (key sizes, parameters, page size, etc) into a
284 separate class which is instantiated by the env
285
286 . c++-ify the btree node representation;
287 o include duplicates as well! disentangle duplicates from blob-handling
dbd7802 @cruppstahl ham_db_t is now C++-ified (WIP)
authored
288
289 . c++-ify the blob handling and split off the duplicates
290
3412f64 @cruppstahl updated TODO list; started with removing idempotent operations from t…
authored
291 . cleanup db.h/db.cc - move functions into Database or
b99a85b @cruppstahl increased version to 20.0.0.rc3
authored
292 DatabaseImplementationLocal namespace - but take care b/c these functions
293 are also used by Cursor or other modules which don't necessarily have
294 access to the Local stuff
295 o db_get_key_count
296 o db_alloc_page
297 o db_fetch_page
298 o db_insert_txn
299 o db_erase_txn
300 o db_find_txn
301 o db_check_insert_conflicts
302 o db_check_erase_conflicts
303 o __increment_dupe_index
304
dbd7802 @cruppstahl ham_db_t is now C++-ified (WIP)
authored
305 . c++-ify everything else
306
3412f64 @cruppstahl updated TODO list; started with removing idempotent operations from t…
authored
307 . device, page and os shold no longer return errors but throw exceptions
6302799 @cruppstahl updated TODO file
authored
308
3412f64 @cruppstahl updated TODO list; started with removing idempotent operations from t…
authored
309 XXXXXXXXXXXXXXXXXXXXX release 2.0.0 STABLE !!! XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cac668e @cruppstahl Cleaning up the cache usage, removed unnecessary code
authored
310
a95e0b7 @cruppstahl updated TODO file
authored
311 . new flag for transactions: HAM_TXN_WILL_COMMIT
e3e99cc @cruppstahl added unittest for cursor_get_dupecache_count
authored
312 if this flag is set, then write all records directly to the blob, not
313 to the log. the log will only contain the rid.
314 o document this (and also the drawback - that an abort will lose the
315 blobs and they cannot be reused
316 -> this affects all temporary ham_insert-transactions
317 (not sure if this should get high priority)
cac668e @cruppstahl Cleaning up the cache usage, removed unnecessary code
authored
318
cf9c6c7 @cruppstahl fixed a few regressions
authored
319 o hamsterdb.com
320 x add twitter feed
321 o API documentation: don't link to "modules" but to startup page, update
322 with newest version
323 o crupp.de: do a backup of the database
324 . google +1 button
325 . can we use something like Aller.font?
326
b98155a @cruppstahl updated header file with documentation (and new copyright year)
authored
327 o update documentation
328 x in header file
329 o in the wiki
f9d3746 @cruppstahl HAM_SORT_DUPLICATES is now forbidden if Transactions are enabled
authored
330 o don't forget to list all functions that are currently disabled
331 w/ txn's -> sorting dupes, approx. matching, ...
3e66c05 @cruppstahl updated TODO list
authored
332 o transactional behavior/conflicts of duplicate keys
9220b03 @cruppstahl improved the doxygen documentation and added a 'mainpage' tag
authored
333 o in the wiki: start with internal documentation
334 o transactions
335 o architecture
336 o btree
337 o journal/log
338 o cache
339 o I/O
e9b49fc @cruppstahl fixed several smaller memory leaks
authored
340 o unittests
2e8ec8b @cruppstahl some refactoring of _local_Cursor_insert; no functional changes
authored
341 o cursor(s)
7a2551d @cruppstahl updated TODO items
authored
342 o monstertests - how to use them?
9220b03 @cruppstahl improved the doxygen documentation and added a 'mainpage' tag
authored
343
fe6e5e8 @cruppstahl configure switch --enable-internal is now obsolete
authored
344 o fully (!) automize the whole release process for unix; the result (on
345 success) are the following files:
346 o tar-ball
347 o the README
348 o the documentation as a tar
349 o the Changelog
350 o the release notes (a template)
351 o the output (xml) of the monster tests
352
7a2551d @cruppstahl updated TODO items
authored
353 . port to WinCE
354
131c913 @cruppstahl refactoring; removed several TODOs and renamed a couple of functions.…
authored
355 o how can we extend the monster-tests to have reliable tests for transactions?
356
7a2551d @cruppstahl updated TODO items
authored
357 . if memory consumption in the txn-tree is too high: flush records to disk
358 (not sure if this should get high priority)
affbe6f @cruppstahl fixed several regressions
authored
359
e9a9e6d @cruppstahl added a unittest to make sure that the dupecache is extended correctl…
authored
360 o when recovering, give users the choice if active transactions should be
361 aborted (default behavior) or re-created
362
66cc747 @cruppstahl cleanup for the next version - removed all items which were implement…
authored
363 o extkeys: don't use txn_id for the 'age', use lsn instead
3c8400b @cruppstahl updated TODO file
authored
364
f2c0e4f @cruppstahl more fixes and performance for recovery
authored
365 o the DatabaseImplementation subclass is not neccessary; all subclasses
366 can directly derive from Database.
367
66cc747 @cruppstahl cleanup for the next version - removed all items which were implement…
authored
368 . allow use of transactions without a log/journal
4ba05c5 @cruppstahl removed log file switching from physical log file; general refactorin…
authored
369
66cc747 @cruppstahl cleanup for the next version - removed all items which were implement…
authored
370 . allow use of transactions for in-memory databases
4ba05c5 @cruppstahl removed log file switching from physical log file; general refactorin…
authored
371
e4991ac @cruppstahl version 2.0.0
authored
372 XXXXXXXXXXXXXXXXXXXXX release 2.0.0 STABLE XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
f9a1f7b @cruppstahl fixed several regressions; ham_cursor_erase and ham_cursor_find are n…
authored
373
a95e0b7 @cruppstahl updated TODO file
authored
374 o ham_get_count: could be atomically updated with every journal entry
375
83d34cc @cruppstahl fixed more regressions
authored
376 o when flushing a changeset: sort by offset, use writev()
377
a95e0b7 @cruppstahl updated TODO file
authored
378 o add concurrency (on a high level)
379
e9a9e6d @cruppstahl added a unittest to make sure that the dupecache is extended correctl…
authored
380 o flush transactions in background
f9a1f7b @cruppstahl fixed several regressions; ham_cursor_erase and ham_cursor_find are n…
authored
381
e9a9e6d @cruppstahl added a unittest to make sure that the dupecache is extended correctl…
authored
382 . have a flag to disable flushing of logfiles/journal files (or flush them
383 async.)
384
1bcb6af red-black tree will be needed for transaction stuff
Christoph Rupp authored
385 o continue as described in integrate-ham2.txt...
4ba05c5 @cruppstahl removed log file switching from physical log file; general refactorin…
authored
386
Something went wrong with that request. Please try again.