MVSTORE -> Corrupted database #373

ebocher · 2016-10-07T06:30:02Z

We are faced with a serious bug with mv_store. We have a complex sql chain that integrates spatial processing. Since the Version 1.4.192 , we are not able to execute the chain, due to the following error :

org.h2.jdbc.JdbcSQLException: General error: "java.lang.IllegalStateException: Chunk 13931 no longer exists [1.4.192/9]"; SQL statement:
CREATE TABLE DATA_WORK.DIST (PK integer PRIMARY KEY, MIN_DIST double precision, MEAN_DIST double precision, MAX_DIST double precision, STD_DIST double precision, PK_USR integer)AS SELECT a.PK, MIN(ST_DISTANCE(a.THE_GEOM, b.THE_GEOM)) AS MIN_DIST, AVG(ST_DISTANCE(a.THE_GEOM, b.THE_GEOM)) AS MEAN_DIST, MAX(ST_DISTANCE(a.THE_GEOM, b.THE_GEOM)) AS MAX_DIST, STDDEV_POP(ST_DISTANCE(a.THE_GEOM, b.THE_GEOM)) AS STD_DIST,a.PK_USR FROM BUILDINGS_MAPUCE a, BUILDINGS_MAPUCE b WHERE a.PK<>b.PK AND a.PK_USR=b.PK_USR GROUP BY a.PK_USR, a.PK [50000-192]

If MV_STORE = FALSE the chain works. However because of several new features on mv_store we want to use it.

I can share the database if needed.

Thanks a lot.

Tolsi · 2016-10-07T13:20:15Z

+1 we have the same thing using only mvstore, not in h2 bundle

        java.lang.IllegalStateException: Chunk 114703 no longer exists [1.4.192/9]
    at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
    at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:951)
    at org.h2.mvstore.MVStore.getChunk(MVStore.java:933)
    at org.h2.mvstore.MVStore.readPage(MVStore.java:1935)
    at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
    at org.h2.mvstore.Page.getChildPage(Page.java:217)
    at org.h2.mvstore.MVMap.put(MVMap.java:195)
    at org.h2.mvstore.MVMap.put(MVMap.java:121)
....

thomasmueller · 2016-10-07T13:42:14Z

Hi,

I would be very interested in the database files, and the circumstances
when this occurred (power failure, killing a process). Specially if it is
(somewhat) reproducible.

One case I know it could happen is that if the file system re-orders
writes, and writes the latest changes but not older changes. In the trunk,
the MVStore tool has a feature to repair a file, by rolling back to the
latest working version (MVStoreTool -repair).

Regards,
Thomas

On Fri, Oct 7, 2016 at 3:20 PM, Sergey Tolmachev notifications@github.com
wrote:

+1 we have the same thing using only mvstore, not in h2 bundle
    java.lang.IllegalStateException: Chunk 114703 no longer exists [1.4.192/9]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:951)
at org.h2.mvstore.MVStore.getChunk(MVStore.java:933)
at org.h2.mvstore.MVStore.readPage(MVStore.java:1935)
at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
at org.h2.mvstore.Page.getChildPage(Page.java:217)
at org.h2.mvstore.MVMap.put(MVMap.java:195)
at org.h2.mvstore.MVMap.put(MVMap.java:121)
....

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#373 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA46GfD5V50Dr-V50ORcfwxJUu-GIDbnks5qxkcQgaJpZM4KQtuO
.

Tolsi · 2016-10-07T14:17:46Z

@thomasmueller Hi, thanks for quick answer!

I can reproduce it sometimes on ubuntu 14.04 with ext4 (flags defaults,discard) with 3.13.0-74-generic kernel, but we have big db files (~10gb). I can send it for you on the next failure.

Tolsi · 2016-10-09T07:46:57Z

@thomasmueller Here it is. Our source code repos are one and two

java.lang.IllegalStateException: This store is closed [1.4.192/4]
        at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
        at org.h2.mvstore.MVStore.checkOpen(MVStore.java:2401)
        at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:947)
        at org.h2.mvstore.MVStore.getChunk(MVStore.java:933)
        at org.h2.mvstore.MVStore.readPage(MVStore.java:1935)
        at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
        at org.h2.mvstore.Page.getChildPage(Page.java:217)
        at org.h2.mvstore.MVMap.binarySearch(MVMap.java:468)
        at org.h2.mvstore.MVMap.get(MVMap.java:450)
        at scorex.utils.LogMVMapBuilder$$anon$1.get(LogMVMapBuilder.scala:19)
        at scorex.transaction.state.database.blockchain.StoredBlockchain$BlockchainPersistence.score(StoredBlockchain.scala:83)
        at scorex.transaction.state.database.blockchain.StoredBlockchain.score(StoredBlockchain.scala:124)
        at scorex.network.Coordinator$$anonfun$scorex$network$Coordinator$$idle$1.applyOrElse(Coordinator.scala:44)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at scorex.network.Coordinator.aroundReceive(Coordinator.scala:26)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalStateException: Chunk 149176 not found [1.4.192/6]
        at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
        at org.h2.mvstore.MVStore.getChunk(MVStore.java:936)
        at org.h2.mvstore.MVStore.readPageChunkReferences(MVStore.java:1373)
        at org.h2.mvstore.MVStore.collectReferencedChunks(MVStore.java:1327)
        at org.h2.mvstore.MVStore.collectReferencedChunks(MVStore.java:1332)
        at org.h2.mvstore.MVStore.collectReferencedChunks(MVStore.java:1312)
        at org.h2.mvstore.MVStore.freeUnusedChunks(MVStore.java:1271)
        at org.h2.mvstore.MVStore.storeNowTry(MVStore.java:1060)
        at org.h2.mvstore.MVStore.storeNow(MVStore.java:1048)
        at org.h2.mvstore.MVStore.commitAndSave(MVStore.java:1037)
        at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2481)
        at org.h2.mvstore.MVStore$BackgroundWriterThread.run(MVStore.java:2694)

ebocher · 2016-10-09T13:26:00Z

Dear @thomasmueller

Thanks a lot for the answer. Please find the database here https://drive.google.com/file/d/0B0JU3-h-7ZSzU3JiX2I1XzdKQVk/view?usp=sharing
The zip file contains the trace file, set with TRACE_LEVEL_FILE=3;TRACE_LEVEL_SYSTEM_OUT=3

ebocher · 2016-10-17T16:34:14Z

@thomasmueller
Sorry to bother you, but you are the mvstore expert :-).
Did you find the time to look on our database ? Our processing chain is currently stopped.
I would like to thank you in advance for paying attention of this issue.

ebocher · 2016-11-02T14:56:38Z

The last version 1.4.193 fixes this issue for my tests.
Thanks a lot @thomasmueller and @grandinj

ebocher · 2016-11-15T19:24:10Z

@Tolsi Could you please confirm if it works or not ?

anidotnet · 2016-11-18T08:48:52Z

I guess it still has the issue in the latest build. This is what I am getting as stacktrace.

java.lang.IllegalStateException: Chunk 285 no longer exists [1.4.193/9]
    at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
    at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:955)
    at org.h2.mvstore.MVStore.getChunk(MVStore.java:937)
    at org.h2.mvstore.MVStore.readPage(MVStore.java:1945)
    at org.h2.mvstore.MVMap.readPage(MVMap.java:736)
    at org.h2.mvstore.Page.getChildPage(Page.java:217)
    at org.h2.mvstore.Cursor.fetchNext(Cursor.java:150)
    at org.h2.mvstore.Cursor.next(Cursor.java:50)
    at org.h2.mvstore.MVMap$2$1.next(MVMap.java:879)
    at org.h2.mvstore.MVMap$2$1.next(MVMap.java:870)
    at com.mycomp.Application$ScriptRunner.run(Application.java:77)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784)
    at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
    at com.mycomp.Application.main(Application.java:27)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

Tolsi · 2016-11-18T19:11:08Z

@ebocher I'm not sure, but we did not catch it anymore

Tolsi · 2017-03-23T15:27:57Z

It still exists

tjuchniewicz · 2017-04-04T09:53:05Z

I found the same issue in the latest version:

public class MVStore {

   private Chunk getChunkIfFound(long pos) {
        int chunkId = DataUtils.getPageChunkId(pos);
        Chunk c = chunks.get(chunkId);
        if (c == null) {
            checkOpen();
            if (!Thread.holdsLock(this)) {
                // it could also be unsynchronized metadata
                // access (if synchronization on this was forgotten)
                throw DataUtils.newIllegalStateException(
                        DataUtils.ERROR_CHUNK_NOT_FOUND,
                        "Chunk {0} no longer exists",
                        chunkId); // THIS EXCEPTION IS THROWN
            }
...

The problem is that simple test works fine for me. I can't easily extract my issue to separate unit test...

anidotnet · 2017-12-23T06:36:00Z

Any update on this issue? I am still facing it in version 1.4.196 here. Can anyone please explain what is the rationale behind this line

if (!Thread.holdsLock(this)) {
                // it could also be unsynchronized metadata
                // access (if synchronization on this was forgotten)
                throw DataUtils.newIllegalStateException(
                        DataUtils.ERROR_CHUNK_NOT_FOUND,
                        "Chunk {0} no longer exists",
                        chunkId); // THIS EXCEPTION IS THROWN
            }

In what kind of scenario this could happen?

I have a reproducible code for this error - http://pastebin.com/kk9P06sG

anidotnet · 2017-12-27T18:01:41Z

For now, I have bypassed this issue by stopping auto compacting.

builder.autoCompactFillRate(0);

If I enable auto compacting, the issue reappears.

shollander · 2018-06-05T14:13:26Z

I was having a similar issue of receiving a java.lang.IllegalStateException: Chunk ### no longer exists when iterating through the contents of an MVStore map. Upgrading to the latest version (1.4.197) seems to fix the issue.

mpollmeier · 2019-02-04T03:56:25Z

Issue still exists for me in 1.4.197/9 (only from time to time).
builder.autoCompactFillRate(0) didn't make a difference for me.

katzyn · 2019-02-04T04:01:59Z

Do you use 1.4.197 or own build from the current sources?

mpollmeier · 2019-02-04T07:11:04Z

It's with the official 1.4.197 build.
Currently I'm using one MVStore and two MVMap<Long, byte[]> instances - I'll try if it makes a difference with only one MVMap. The error only comes up occasionally though unfortunately, so it's a bit tough to verify and isolate.

see h2database/h2database#373

mpollmeier · 2019-02-05T00:46:11Z

The issue disappeared for me when I changed to using only one MVMap. Maybe this information is useful for other users and/or track down the underlying issue.

katzyn · 2019-02-05T02:19:45Z

MVStore was changed since 1.4.197, could you build it from the latest sources and retest it?
http://h2database.com/html/build.html#building
You need jarMVStore target.

forchid · 2019-03-16T12:38:43Z

Please use h2-1.4.199, and see #1502.

re h2database/h2database#373 (comment)

grandinj · 2020-04-29T06:15:34Z

please re-open if this is still an issue with the latest version

ebocher mentioned this issue Oct 7, 2016

Crash during computation of distance from the center of the commune orbisgis/MApUCE_tools#25

Closed

Tolsi mentioned this issue Oct 10, 2016

MVStore: Chunk not found wavesplatform/Waves#55

Closed

grandinj changed the title ~~MVSTORE -> Corruped database~~ MVSTORE -> Corrupted database Oct 16, 2016

tatsuo98se mentioned this issue Jan 31, 2018

Backup cause Internal Server Error - java.io.IOException: This archives contains unclosed entries. support-project/knowledge#964

Open

mpollmeier added a commit to ShiftLeftSecurity/tinkergraph-gremlin that referenced this issue Feb 4, 2019

try to work around corrupt db issue, not working though

2fb6fbc

see h2database/h2database#373

mpollmeier added a commit to ShiftLeftSecurity/tinkergraph-gremlin that referenced this issue Mar 16, 2019

upgrade h2

38046db

re h2database/h2database#373 (comment)

essiembre mentioned this issue Jul 29, 2019

Norconex is doing a unexpected recrawl from scratch Norconex/crawlers#620

Closed

grandinj closed this as completed Apr 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MVSTORE -> Corrupted database #373

MVSTORE -> Corrupted database #373

ebocher commented Oct 7, 2016

Tolsi commented Oct 7, 2016

thomasmueller commented Oct 7, 2016

Tolsi commented Oct 7, 2016 •

edited

Loading

Tolsi commented Oct 9, 2016 •

edited

Loading

ebocher commented Oct 9, 2016

ebocher commented Oct 17, 2016

ebocher commented Nov 2, 2016

ebocher commented Nov 15, 2016

anidotnet commented Nov 18, 2016

Tolsi commented Nov 18, 2016

Tolsi commented Mar 23, 2017

tjuchniewicz commented Apr 4, 2017 •

edited

Loading

anidotnet commented Dec 23, 2017 •

edited

Loading

anidotnet commented Dec 27, 2017

shollander commented Jun 5, 2018

mpollmeier commented Feb 4, 2019

katzyn commented Feb 4, 2019

mpollmeier commented Feb 4, 2019

mpollmeier commented Feb 5, 2019

katzyn commented Feb 5, 2019

forchid commented Mar 16, 2019

grandinj commented Apr 29, 2020

MVSTORE -> Corrupted database #373

MVSTORE -> Corrupted database #373

Comments

ebocher commented Oct 7, 2016

Tolsi commented Oct 7, 2016

thomasmueller commented Oct 7, 2016

Tolsi commented Oct 7, 2016 • edited Loading

Tolsi commented Oct 9, 2016 • edited Loading

ebocher commented Oct 9, 2016

ebocher commented Oct 17, 2016

ebocher commented Nov 2, 2016

ebocher commented Nov 15, 2016

anidotnet commented Nov 18, 2016

Tolsi commented Nov 18, 2016

Tolsi commented Mar 23, 2017

tjuchniewicz commented Apr 4, 2017 • edited Loading

anidotnet commented Dec 23, 2017 • edited Loading

anidotnet commented Dec 27, 2017

shollander commented Jun 5, 2018

mpollmeier commented Feb 4, 2019

katzyn commented Feb 4, 2019

mpollmeier commented Feb 4, 2019

mpollmeier commented Feb 5, 2019

katzyn commented Feb 5, 2019

forchid commented Mar 16, 2019

grandinj commented Apr 29, 2020

Tolsi commented Oct 7, 2016 •

edited

Loading

Tolsi commented Oct 9, 2016 •

edited

Loading

tjuchniewicz commented Apr 4, 2017 •

edited

Loading

anidotnet commented Dec 23, 2017 •

edited

Loading