Improve current storage durability #1404

Closed
laa opened this Issue Mar 21, 2013 · 10 comments

Comments

Projects
None yet
2 participants
Member

laa commented Mar 21, 2013

Current storage uses MMAP as disk buffer which means that management of disk pages relies upon OS not DB itself as result we can control cache behavior in limited manner and this also makes storage not safe in case of power failure to improve given situation following is proposed:

  1. Replaced MMAP by our own 2Q cache which was designed for usage in DBs as disk cache + write cache which will contain records are going to be flushed sorted by position which will minimize random IO operations.
  2. Implement paginated storage so we can treat pages as atomic units of data, as side effect we can remove storage global lock and replace 2 disk seeks IO by one disk seek IO for retrieval of record by id
  3. Implement WAL inside of paginated storage.
  4. Use direct memory for page cache to avoid GC overhead (Unsafe + JNA fallback).

laa was assigned Mar 21, 2013

@laa laa added a commit that referenced this issue Apr 1, 2013

@laa laa 1. Enhancement - If I/O exception will be thrown data consistency wil…
…l not be compromised. Issue #1404.

 2. Build fix.
ce2578c
Owner

lvca commented Apr 1, 2013

WDYT about creating a compression factory with Snappy as implementation and a Nothing as default?

Something like
OCompressionFactory{getCompression(String name);register(String name, OCompression algo);}
OCompression{compress();uncompress();}

In this way we could use it in different places even OStorageLocal where compression.compress() do nothing by default.

WDYT?

Owner

lvca commented Apr 1, 2013

I'm going to open a new issue...

Member

laa commented Apr 1, 2013

Yes,
Sure why not.

On Mon, Apr 1, 2013 at 11:51 AM, Luca Garulli notifications@github.comwrote:

I'm going to open a new issue...


Reply to this email directly or view it on GitHubhttps://github.com/nuvolabase/orientdb/issues/1404#issuecomment-15708782
.

With best regards,
Andrey Lomakin

Owner

lvca commented Apr 1, 2013

It already exists: issue #208

@laa laa added a commit that referenced this issue Apr 1, 2013

@laa laa Issue #1404. Magic numbers were removed. f7064aa

@laa laa added a commit that referenced this issue Apr 2, 2013

@laa laa Issue #1404. LRU cache was replaced by 2Q cache. Cache API was refact…
…ored to provide more robust approach for cache management.
8c97d61

@laa laa added a commit that referenced this issue Apr 2, 2013

@laa laa Issue #1404. LRU cache was replaced by 2Q cache. Cache API was refact…
…ored to provide more robust approach for cache management. Thanks to @logart !
d5f17aa

@laa laa added a commit that referenced this issue Apr 30, 2013

@laa laa Issue #1404 WAL truncation tests were added. 960beec

@laa laa added a commit that referenced this issue May 2, 2013

@laa laa Issue #1404 Checkpoint was slitted to fuzzy checkpoint and full check…
…point. New add page record was added.
d008759

@laa laa added a commit that referenced this issue May 5, 2013

@laa laa Issue #1404 FlushTillLSN tests were added. ac0c7bd

@laa laa added a commit that referenced this issue May 6, 2013

@laa laa Issue #1404 1. Issue wih OFileClassic softly closed detection was fix…
…ed (OS caches were not taken in account). 2. Unsafe direct memory implementation was replaced by JNA because of JIT related issues. 3. Few multithreading improvements for OFile implementations.
cc2e4fb

@laa laa added a commit that referenced this issue May 13, 2013

@laa laa Issue #1404. New paginated WAL version. 432fd4b

@laa laa added a commit that referenced this issue May 13, 2013

@laa laa Issue #1404. 2Q cache tests were fixed. a768ef4

@laa laa added a commit that referenced this issue May 22, 2013

@laa laa Issue #1404. Initial version of restore procedure was implemented. MT…
… issues in multifile segment were fixed.
2f4a457

@laa laa added a commit that referenced this issue May 28, 2013

@laa laa Issue #1404 Track changes mode is detected automatically. If it is ne…
…eded rollbacks support is switched on.
6321fef

@laa laa added a commit that referenced this issue Jun 5, 2013

@laa laa Issue #1404 Initial tests to restore Storage from WAL were successful…
…ly passed !
7b4a0eb

@laa laa added a commit that referenced this issue Jun 6, 2013

@laa laa Issue #1404 Few log messages were added. 65a0480

@laa laa added a commit that referenced this issue Jun 13, 2013

@laa laa Issue #1404 All Tx tests were passed. babe3fd

@laa laa added a commit that referenced this issue Jun 14, 2013

@laa laa Issue #1404 WAL performance optimizations. Blueprints tests were succ…
…essfully passed.
c683e83

@laa laa added a commit that referenced this issue Jun 18, 2013

@laa laa Issue #1404 Durability test which checks that after restore DB can be…
… used without any issues was added.
b34f2a2

@laa laa added a commit that referenced this issue Jun 26, 2013

@laa laa Issue #1404 WAL performance improvements. aa0a85c

@laa laa added a commit that referenced this issue Jun 27, 2013

@laa laa Issue #1404 Crash test with creation of big data records with small c…
…ache size was passed.
4341989

@laa laa added a commit that referenced this issue Jun 27, 2013

@laa laa Issue #1404 Java 6 compatibility fix. a0215d4

@laa laa added a commit that referenced this issue Jun 27, 2013

@laa @lvca laa + lvca Issue #1404 Java 6 compatibility fix. 06efef6

@laa laa added a commit that referenced this issue Jul 17, 2013

@laa laa Issue #1404. Support for plocal in remote server was added. plocal is…
… default storage type for remote tests.
f6b969d

@laa laa added a commit that referenced this issue Jul 23, 2013

@laa laa Issue #1404 OCacheEntry was extracted 94b7b1e
Member

laa commented Jul 24, 2013

plocal is stable now to release but we need to do some improvements to be sure that restore procedure is ready on 100% so we move it to 2.0

@laa laa added a commit that referenced this issue Jul 26, 2013

@laa laa Issue #1404 Stale buffer links were dropped. e6ee0af

@laa laa added a commit that referenced this issue Jul 26, 2013

@laa laa Issue #1404 First WOW cache test was added. aca2a1f

@laa laa added a commit that referenced this issue Aug 7, 2013

@laa laa Issue #1404 Write cache speed improvements. 8d695a2

@laa laa added a commit that referenced this issue Aug 7, 2013

@laa laa Issue #1404 Fix of write cache test issues. 718b4f7
Member

laa commented Aug 11, 2013

I added several sub issues when they will be finished this issue can be completed. #1608 #1607 #1606 #1605 #1604 #1603 #1602 #1626

Owner

lvca commented Aug 28, 2014

@laa Is this fixed?

Member

laa commented Aug 29, 2014

No, till we fix issues listed in #1404 (comment)

laa was assigned by lvca Sep 14, 2014

@laa laa modified the milestone: 2.0-M2, 2.0-M1, 2.1 Sep 17, 2014

Owner

lvca commented Mar 14, 2015

@laa What's the status of this?

@lvca lvca modified the milestone: 2.1-rc1, 2.1-rc2 Apr 1, 2015

Member

laa commented Apr 27, 2015

we may close this issue

laa closed this Apr 27, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment