Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Repair database runs out of memory when fixing links #9535

Open
toebes opened this issue Feb 5, 2021 · 0 comments
Open

Repair database runs out of memory when fixing links #9535

toebes opened this issue Feb 5, 2021 · 0 comments
Milestone
3.0.x

Comments

@toebes
Copy link

toebes commented Feb 5, 2021

OrientDB Version: OrientDB Server v3.0.31 - Veloce (build 2a84124, branch UNKNOWN)

Java Version: OpenJDK Runtime Environment (build 1.8.0_252-b09)

OS: Linux 1e1b3cda092b 4.14.214-160.339.amzn2.x86_64

Expected behavior

Repair database completes successfully

Actual behavior

Repair runs out of memory

Steps to reproduce

Somehow our database got into a bad state (not sure how) and I had several records which would generate an out of memory when attempting to access them.

orientdb_1 | java.lang.OutOfMemoryError: Java heap space
orientdb_1 | Dumping heap to java_pid1.hprof ...
orientdb_1 | Heap dump file created [77078547 bytes in 0.282 secs]
orientdb_1 | Exception 6C761769 in storage plocal:/orientdb/databases/5204640393085028696: 3.0.31 - Veloce (build 2a84124, branch UNKNOWN)
orientdb_1 | java.lang.OutOfMemoryError: Java heap space
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.getBinaryValue(ODurablePage.java:146)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.OClusterPage.getRecordBinaryValue(OClusterPage.java:398)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readFullEntry(OPaginatedClusterV1.java:1536)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.internalReadRecord(OPaginatedClusterV1.java:560)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readRecord(OPaginatedClusterV1.java:529)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readRecord(OPaginatedClusterV1.java:512)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:5055)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:4768)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1920)
orientdb_1 | at com.orientechnologies.orient.core.db.document.SimpleRecordReader.readRecord(SimpleRecordReader.java:21)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.executeReadRecord(ODatabaseDocumentAbstract.java:1344)
orientdb_1 | at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:90)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:1136)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:86)
orientdb_1 | at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeReadRecord(OConnectionBinaryExecutor.java:294)
orientdb_1 | at com.orientechnologies.orient.client.remote.message.OReadRecordRequest.execute(OReadRecordRequest.java:98)
orientdb_1 | at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317)
orientdb_1 | at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212)
orientdb_1 | at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)
orientdb_1 | JVM error was thrown
orientdb_1 | java.lang.OutOfMemoryError: Java heap space
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.getBinaryValue(ODurablePage.java:146)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.OClusterPage.getRecordBinaryValue(OClusterPage.java:398)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readFullEntry(OPaginatedClusterV1.java:1536)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.internalReadRecord(OPaginatedClusterV1.java:560)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readRecord(OPaginatedClusterV1.java:529)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readRecord(OPaginatedClusterV1.java:512)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:5055)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:4768)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1920)
orientdb_1 | at com.orientechnologies.orient.core.db.document.SimpleRecordReader.readRecord(SimpleRecordReader.java:21)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.executeReadRecord(ODatabaseDocumentAbstract.java:1344)
orientdb_1 | at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:90)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:1136)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:86)
orientdb_1 | at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeReadRecord(OConnectionBinaryExecutor.java:294)
orientdb_1 | at com.orientechnologies.orient.client.remote.message.OReadRecordRequest.execute(OReadRecordRequest.java:98)
orientdb_1 | at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317)
orientdb_1 | at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212)
orientdb_1 | at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)
orientdb_1 | Uncaught exception in thread OrientDB (/172.20.0.7:2424) <- BinaryClient (/172.20.0.8:34468)
orientdb_1 | java.lang.OutOfMemoryError: Java heap space
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.getBinaryValue(ODurablePage.java:146)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.OClusterPage.getRecordBinaryValue(OClusterPage.java:398)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readFullEntry(OPaginatedClusterV1.java:1536)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.internalReadRecord(OPaginatedClusterV1.java:560)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readRecord(OPaginatedClusterV1.java:529)
orientdb_1 | at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.readRecord(OPaginatedClusterV1.java:512)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:5055)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:4768)
orientdb_1 | at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1920)
orientdb_1 | at com.orientechnologies.orient.core.db.document.SimpleRecordReader.readRecord(SimpleRecordReader.java:21)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.executeReadRecord(ODatabaseDocumentAbstract.java:1344)
orientdb_1 | at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:90)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:1136)
orientdb_1 | at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.load(ODatabaseDocumentAbstract.java:86)
orientdb_1 | at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeReadRecord(OConnectionBinaryExecutor.java:294)
orientdb_1 | at com.orientechnologies.orient.client.remote.message.OReadRecordRequest.execute(OReadRecordRequest.java:98)
orientdb_1 | at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:317)
orientdb_1 | at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212)
orientdb_1 | at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)

I pulled the application down and connected to the database in the docker container to run a repair database command with the
repair database --fix-links -v
it runs for a while and then ends up

--- reset link #30:1707 in field 'model' (rid=#104:4529)
--- reset link #29:1711 in field 'model' (rid=#104:4530)
--- reset link #30:1709 in field 'model' (rid=#104:4531)
Error: java.lang.OutOfMemoryError: Java heap space
@tglman tglman added this to the 3.0.x milestone Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
3.0.x
Development

No branches or pull requests
2 participants
@tglman @toebes