Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Geometry memory limitation with H2 2.0.206 #1243

Closed
ebocher opened this issue Jan 7, 2022 · 16 comments
Closed

Geometry memory limitation with H2 2.0.206 #1243

ebocher opened this issue Jan 7, 2022 · 16 comments
Labels
Priority Critical

Comments

@ebocher
Copy link
Member

ebocher commented Jan 7, 2022 •
edited

H2 fails when a large geometry must be stored.

Here the command, I run to build a delaunay triangulation that works on the last H2GIS release 1.5.0.

CALL GEOJSONREAD('/tmp/contourlines.geojson', 'contourlines');
DROP TABLE IF EXISTS contour_tin;
CREATE TABLE contour_tin AS  SELECT ST_DELAUNAY(ST_ACCUM(ST_UpdateZ(ST_FORCE3D(the_geom), Z))) as the_geom from contourlines;

returns

Caused by: java.sql.SQLException: GeneralError
	at org.h2.message.DbException.convert(DbException.java:419)
	at org.h2.mvstore.db.MVTable.addRow(MVTable.java:543)
	at org.h2.command.dml.Insert.addRow(Insert.java:234)
	at org.h2.command.query.Query.finishResult(Query.java:944)
	at org.h2.command.query.Select.queryWithoutCache(Select.java:851)
	at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
	at org.h2.command.query.Query.query(Query.java:494)
	at org.h2.command.dml.Insert.insertRows(Insert.java:197)
	at org.h2.command.dml.Insert.update(Insert.java:135)
	at org.h2.command.dml.DataChangeStatement.update(DataChangeStatement.java:61)
	at org.h2.command.ddl.CreateTable.update(CreateTable.java:172)
	at org.h2.command.CommandContainer.update(CommandContainer.java:173)
	at org.h2.command.CommandList.executeRemaining(CommandList.java:58)
	at org.h2.command.CommandList.update(CommandList.java:66)
	at org.h2.command.Command.executeUpdate(Command.java:252)

A workaround is to explode on the fly the geometry generated by the triangulation.

DROP TABLE IF EXISTS contour_tin;
CREATE TABLE contour_tin AS  SELECT * FROM ST_EXPLODE('(SELECT ST_DELAUNAY(ST_ACCUM(ST_UpdateZ(ST_FORCE3D(the_geom), Z))) as the_geom from contourlines)');

The input file contourlines is attached and the result triangulation too (30282 triangles).

data.tar.gz
@pierromond
Copy link

Hi @ebocher,

Do you need help to bring this problem up to h2? Is there any point if I/we restart or comment on a issue about it?

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

Hi @pierromond ,
We need some comment, advice from @katzyn because it's a blocking issue.

@katzyn
Copy link

katzyn commented Jan 11, 2022

GEOMETRY should use the same storage as LOBs. We planned to do that in H2 2.0, but it requires massive changes in different areas of H2, and, unfortunately, I didn't have time for H2 and I don't have too much time now.

@katzyn
Copy link

katzyn commented Jan 11, 2022

I have no idea how it worked with H2 1.4.200, because there were no significant changes in that area. Do you have a complete stack trace with all causes?

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

Thanks.
Yes I have a complete stack trace. I will post it.

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

Here we have

CREATE TABLE contour_tin AS  SELECT ST_DELAUNAY(ST_ACCUM(ST_UpdateZ(ST_FORCE3D(the_geom), Z))) as the_geom from contourlines; [50000-206]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:573)
	at org.h2.message.DbException.addSQL(DbException.java:181)
	at org.h2.command.Command.executeUpdate(Command.java:268)
	at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:252)
	at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:223)
	at org.h2gis.tests.SQLScriptTests.bug(SQLScriptTests.java:96)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: java.sql.SQLException: GeneralError
	at org.h2.message.DbException.convert(DbException.java:419)
	at org.h2.mvstore.db.MVTable.addRow(MVTable.java:543)
	at org.h2.command.dml.Insert.addRow(Insert.java:234)
	at org.h2.command.query.Query.finishResult(Query.java:944)
	at org.h2.command.query.Select.queryWithoutCache(Select.java:851)
	at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
	at org.h2.command.query.Query.query(Query.java:494)
	at org.h2.command.dml.Insert.insertRows(Insert.java:197)
	at org.h2.command.dml.Insert.update(Insert.java:135)
	at org.h2.command.dml.DataChangeStatement.update(DataChangeStatement.java:61)
	at org.h2.command.ddl.CreateTable.update(CreateTable.java:172)
	at org.h2.command.CommandContainer.update(CommandContainer.java:173)
	at org.h2.command.Command.executeUpdate(Command.java:252)
	... 72 more
Caused by: java.lang.AssertionError
	at org.h2.mvstore.Page.addMemory(Page.java:853)
	at org.h2.mvstore.Page$Leaf.insertLeaf(Page.java:1544)
	at org.h2.mvstore.MVMap.operate(MVMap.java:1830)
	at org.h2.mvstore.tx.TransactionMap.set(TransactionMap.java:346)
	at org.h2.mvstore.tx.TransactionMap.putIfAbsent(TransactionMap.java:281)
	at org.h2.mvstore.db.MVPrimaryIndex.add(MVPrimaryIndex.java:113)
	at org.h2.mvstore.db.MVTable.addRow(MVTable.java:535)
	... 83 more

@katzyn
Copy link

katzyn commented Jan 11, 2022

Do you know how large is you value in (E)WKB representation?

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

Nope. I'll try to mesure it. Is there a method to do that in H2 ?

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

Ok so

System.out.println("Bytes length : "+ ValueGeometry.getFromGeometry(geom).getBytes().length);

returns

Bytes length : 3300751

@katzyn
Copy link

katzyn commented Jan 11, 2022

This is a size of a largest geometry? It isn't that much to cause a failure.

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

Yes this is the final geom.

@katzyn
Copy link

katzyn commented Jan 11, 2022

This issue isn't related with GEOMETRY at all. I filled a new issue about it:
h2database/h2database#3361

@ebocher
Copy link
Member Author

ebocher commented Jan 11, 2022

oupps thanks

@ebocher
Copy link
Member Author

ebocher commented Jan 12, 2022

There is a fix on H2 master. We must wait a new release to integrate it in H2GIS.

@pierromond
Copy link

Nice ! Thank you @ebocher and @katzyn

@ebocher ebocher mentioned this issue Jan 14, 2022
Merged
@ebocher
Copy link
Member Author

ebocher commented Jan 21, 2022

Fixed since 2.1.210

@ebocher ebocher closed this as completed Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Priority Critical
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pierromond @ebocher @katzyn