Use one memory mapped file for plasma #3871

pcmoritz · 2019-01-26T20:38:53Z

AmplabJenkins · 2019-01-26T22:59:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11195/
Test PASSed.

AmplabJenkins · 2019-01-30T03:48:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11288/
Test FAILed.

robertnishihara · 2019-01-30T04:36:52Z

Because we are updating Arrow, this should also fix #3846.

zhijunfu · 2019-01-30T10:34:21Z

Using a single mmap'ed file should help with shared memory usage.

AmplabJenkins · 2019-02-01T19:48:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11407/
Test FAILed.

AmplabJenkins · 2019-02-01T19:50:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11408/
Test FAILed.

pcmoritz · 2019-02-01T21:47:53Z

@raulchen @guoyuhong Can you have a look at the Java failure? Could it be related to apache/arrow#3306, does the Java API need to be changed to incorporate that PR?

AmplabJenkins · 2019-02-01T22:35:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11409/
Test PASSed.

raulchen · 2019-02-02T04:50:32Z

@raulchen @guoyuhong Can you have a look at the Java failure? Could it be related to apache/arrow#3306, does the Java API need to be changed to incorporate that PR?

Doesn't seem to be related with that PR. I'll try to test it locally in a bit.

guoyuhong · 2019-02-02T05:12:03Z

@raulchen I cannot repro it in my mac. I can repro it in my linux system. It may be caused by the PR mentioned by @pcmoritz .

Stack: [0x00007f99de1f7000,0x00007f99de2f7000],  sp=0x00007f99de2f5410,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x6b9835]  jni_ThrowNew+0xb5
C  [libplasma_java.so+0x16975]  Java_org_apache_arrow_plasma_PlasmaClientJNI_create+0x315
j  org.apache.arrow.plasma.PlasmaClientJNI.create(J[BI[B)Ljava/nio/ByteBuffer;+0
j  org.apache.arrow.plasma.PlasmaClient.put([B[B[B)V+11
j  org.ray.runtime.objectstore.ObjectStoreProxy.put(Lorg/ray/api/id/UniqueId;Ljava/lang/Object;Ljava/lang/Object;)V+21
j  org.ray.runtime.AbstractRayRuntime.put(Lorg/ray/api/id/UniqueId;Ljava/lang/Object;)V+27
j  org.ray.runtime.Worker.execute(Lorg/ray/runtime/task/TaskSpec;)V+190
j  org.ray.runtime.Worker.loop()V+31
j  org.ray.runtime.AbstractRayRuntime.loop()V+4
j  org.ray.runtime.runner.worker.DefaultWorker.main([Ljava/lang/String;)V+27
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68855f]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x107f
V  [libjvm.so+0x6b557d]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x32d
V  [libjvm.so+0x6b703f]  jni_CallStaticVoidMethod+0x17f
C  [libjli.so+0x38a4]  JavaMain+0x604
C  [libpthread.so.0+0x7e25]  start_thread+0xc5

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.arrow.plasma.PlasmaClientJNI.create(J[BI[B)Ljava/nio/ByteBuffer;+0
j  org.apache.arrow.plasma.PlasmaClient.put([B[B[B)V+11
j  org.ray.runtime.objectstore.ObjectStoreProxy.put(Lorg/ray/api/id/UniqueId;Ljava/lang/Object;Ljava/lang/Object;)V+21
j  org.ray.runtime.AbstractRayRuntime.put(Lorg/ray/api/id/UniqueId;Ljava/lang/Object;)V+27
j  org.ray.runtime.Worker.execute(Lorg/ray/runtime/task/TaskSpec;)V+190
j  org.ray.runtime.Worker.loop()V+31
j  org.ray.runtime.AbstractRayRuntime.loop()V+4
j  org.ray.runtime.runner.worker.DefaultWorker.main([Ljava/lang/String;)V+27
v  ~StubRoutines::call_stub

Maybe env->FindClass("org/apache/arrow/plasma/exceptions/DuplicateObjectException"); return something wrong?

guoyuhong · 2019-02-02T05:16:18Z

@pcmoritz In my local Mac or Linux I still found there is a java building failure which can be fixed by this PR: apache/arrow#2738 . I will reopen this PR. Could you please help to merge it? This PR will fix the ray-java building error showed in following picture.

pcmoritz · 2019-02-02T06:54:07Z

@guoyuhong Thanks for looking into this! Do you have an idea what the right fix is? This is very high priority since there is a number of bugs that are blocked on being able to upgrade arrow.

guoyuhong · 2019-02-02T07:10:34Z

@pcmoritz It looks like that the Arrow version in pom.xml is too old. So java runtime cannot find the corresponding class. I have updated the pom.xml. @raulchen could help to review this change. @raulchen also said #3687 will make the JAVA test pass.

guoyuhong · 2019-02-02T08:00:36Z

@pcmoritz With @raulchen 's help, the test passed in my local environment. I copied the change in ObjectStoreProxy.java from #3687 . Otherwise, the two PR is dead-locked.

pcmoritz · 2019-02-02T08:33:39Z

Great, thanks a lot! I'll try to get this PR merged tomorrow then :)

AmplabJenkins · 2019-02-02T09:46:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11433/
Test PASSed.

AmplabJenkins · 2019-02-02T12:04:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11440/
Test PASSed.

AmplabJenkins · 2019-02-02T23:11:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11458/
Test PASSed.

AmplabJenkins · 2019-02-03T02:56:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11462/
Test PASSed.

AmplabJenkins · 2019-02-05T21:49:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11586/
Test FAILed.

pcmoritz · 2019-02-05T23:38:43Z

Jenkins retest this please

AmplabJenkins · 2019-02-06T00:49:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11588/
Test FAILed.

AmplabJenkins · 2019-02-06T01:29:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11590/
Test FAILed.

pcmoritz · 2019-02-06T01:33:42Z

Jenkins retest this please

AmplabJenkins · 2019-02-06T02:04:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11594/
Test FAILed.

pcmoritz · 2019-02-06T03:07:34Z

Jenkins retest this please

AmplabJenkins · 2019-02-06T03:32:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11595/
Test FAILed.

pcmoritz · 2019-02-06T03:41:48Z

Jenkins retest this please

AmplabJenkins · 2019-02-06T04:10:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11596/
Test FAILed.

AmplabJenkins · 2019-02-06T04:17:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11592/
Test FAILed.

AmplabJenkins · 2019-02-06T04:43:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11597/
Test FAILed.

AmplabJenkins · 2019-02-06T04:53:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11598/
Test FAILed.

pcmoritz · 2019-02-06T04:53:18Z

Jenkins retest this please

AmplabJenkins · 2019-02-06T05:39:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11601/
Test FAILed.

AmplabJenkins · 2019-02-06T06:23:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11602/
Test FAILed.

AmplabJenkins · 2019-02-06T07:57:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11599/
Test FAILed.

AmplabJenkins · 2019-02-06T09:00:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11605/
Test PASSed.

guoyuhong

All tests passed. Thanks for your effort!

pcmoritz · 2019-02-06T15:26:13Z

test/stress_tests.py

@@ -197,7 +197,7 @@ def g(x):
 def ray_start_reconstruction(request):
    num_nodes = request.param

-    plasma_store_memory = 10**9
+    plasma_store_memory = int(0.5 * 10**9)


This change was needed because the space calculations in plasma are now done slightly differently (there can now be a discrepancy between the plasma store memory and the size of the memory mapped files, in practice I have seen < 10%).

This change is to be on the safe side to prevent flakiness. With 0.8 here I was still seeing test_simple and test_recursive in stress_tests.py failing occasionally.

robertnishihara · 2019-02-06T23:26:07Z

Let's hold off on merging this until after the 0.6.3 release (which should be soon).

pcmoritz changed the title ~~[WIP] Use one memory mapped file for plasma~~ Use one memory mapped file for plasma Jan 30, 2019

pcmoritz force-pushed the ray-plasma-one-file branch from 695c026 to a9f32b1 Compare February 1, 2019 19:44

pcmoritz added 2 commits February 1, 2019 11:47

use one memory mapped file for plasma

394c85d

rebase

c03992f

pcmoritz force-pushed the ray-plasma-one-file branch from a9f32b1 to c03992f Compare February 1, 2019 19:47

Update pom.xml for arrow

5986a49

Fix java test by copy part code of ray-project#3687

05580d7

raulchen mentioned this pull request Feb 2, 2019

[java] when put an object in plasma store, ignore "object alreay exists" exception #3687

Merged

raulchen approved these changes Feb 2, 2019

View reviewed changes

temp commit: try increased timeout for valgrind

97209a4

update

523b3a3

update

e9bc574

update

513ad14

pcmoritz added 2 commits February 5, 2019 20:18

update

ce71019

update

32cdc25

pcmoritz added 2 commits February 5, 2019 21:24

update

a0b6179

update

90fe323

roll back

d3ab3d7

guoyuhong approved these changes Feb 6, 2019

View reviewed changes

pcmoritz commented Feb 6, 2019

View reviewed changes

robertnishihara merged commit 3bb6567 into ray-project:master Feb 7, 2019

robertnishihara deleted the ray-plasma-one-file branch February 7, 2019 07:53

Use one memory mapped file for plasma #3871

Use one memory mapped file for plasma #3871

Conversation

pcmoritz commented Jan 26, 2019

AmplabJenkins commented Jan 26, 2019

AmplabJenkins commented Jan 30, 2019

robertnishihara commented Jan 30, 2019

zhijunfu commented Jan 30, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

pcmoritz commented Feb 1, 2019 • edited Loading

AmplabJenkins commented Feb 1, 2019

raulchen commented Feb 2, 2019

guoyuhong commented Feb 2, 2019 • edited Loading

guoyuhong commented Feb 2, 2019 • edited Loading

pcmoritz commented Feb 2, 2019 • edited Loading

guoyuhong commented Feb 2, 2019

guoyuhong commented Feb 2, 2019

pcmoritz commented Feb 2, 2019

AmplabJenkins commented Feb 2, 2019

AmplabJenkins commented Feb 2, 2019

AmplabJenkins commented Feb 2, 2019

AmplabJenkins commented Feb 3, 2019

AmplabJenkins commented Feb 5, 2019

pcmoritz commented Feb 5, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

pcmoritz commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

pcmoritz commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

pcmoritz commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

pcmoritz commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

AmplabJenkins commented Feb 6, 2019

guoyuhong left a comment

Choose a reason for hiding this comment

pcmoritz Feb 6, 2019

Choose a reason for hiding this comment

robertnishihara commented Feb 6, 2019

pcmoritz commented Feb 1, 2019 •

edited

Loading

guoyuhong commented Feb 2, 2019 •

edited

Loading

guoyuhong commented Feb 2, 2019 •

edited

Loading

pcmoritz commented Feb 2, 2019 •

edited

Loading