Introduce implementation for PlainDatagramSocketImpl.mcast_join_leave #1404

dmlloyd · 2019-06-19T15:08:56Z

Partially fixes #1398

cstancu · 2019-06-19T23:33:16Z

Can you come up with a simple test that exercises this code at least partially that we can add under com.oracle.svm.test?

dmlloyd · 2019-06-20T13:00:48Z

Sure, maybe I can adapt the original test case.

dmlloyd · 2019-06-20T16:30:37Z

Done.

cstancu · 2019-06-20T19:50:49Z

This currently fails on Darwin with:

Error: Error compiling query code (in /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives.c). Compiler command  gcc /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives.c -o /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives output included error: /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives.c:899:110: error: use of undeclared identifier 'IPV6_ADD_MEMBERSHIP'
    method com.oracle.svm.core.posix.headers.Pthread.PTHREAD_CANCEL_DISABLE()
    C file contents around line 899:
    /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives.c:898:     printf("NativeCodeInfo:PosixDirectives:ConstantInfo:PTHREAD_CANCEL_DISABLE:PropertyInfo:value=%lX\n", PTHREAD_CANCEL_DISABLE);
    /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives.c:899:     printf("NativeCodeInfo:PosixDirectives:ConstantInfo:IPV6_ADD_MEMBERSHIP:PropertyInfo:size=%lu\n", sizeof(IPV6_ADD_MEMBERSHIP));
    /var/folders/gw/2lg06ffn6zvb2svk9svvmrym0000gp/T/SVM-2074132810265460097/PosixDirectives.c:900:     printf("NativeCodeInfo:PosixDirectives:ConstantInfo:IPV6_ADD_MEMBERSHIP:PropertyInfo:signedness=$%s$\n", (IS_CONST_UNSIGNED(IPV6_ADD_MEMBERSHIP)) ? "UNSIGNED" : "SIGNED");

dmlloyd · 2019-06-20T19:56:41Z

OK I'll check it out. I think I have an idea of what the problem might be.

dmlloyd · 2019-06-20T20:16:44Z

My local build is still running on my really slow Mac but this should fix the problem. Just out of curiosity - how do you get the tests to run as a native image?

cstancu · 2019-06-20T20:54:22Z

You can use the mx native-unittest in substratevm like so:

~/graal/substratevm$ mx native-unittest GitHub1398
Building junit image for matching: com.oracle.svm.test.GitHub1398
Build on Server(pid: 14943, port: 44121)*
[com.oracle.svm.junit.svmjunitrunner:14943]    classlist:   1,309.45 ms
[com.oracle.svm.junit.svmjunitrunner:14943]        (cap):   1,186.72 ms
[com.oracle.svm.junit.svmjunitrunner:14943]        setup:   2,638.80 ms
[com.oracle.svm.junit.svmjunitrunner:14943]   (typeflow):   4,862.05 ms
[com.oracle.svm.junit.svmjunitrunner:14943]    (objects):   2,942.30 ms
[com.oracle.svm.junit.svmjunitrunner:14943]   (features):     234.29 ms
[com.oracle.svm.junit.svmjunitrunner:14943]     analysis:   8,237.48 ms
[com.oracle.svm.junit.svmjunitrunner:14943]     (clinit):     126.05 ms
[com.oracle.svm.junit.svmjunitrunner:14943]     universe:     437.07 ms
[com.oracle.svm.junit.svmjunitrunner:14943]      (parse):     852.09 ms
[com.oracle.svm.junit.svmjunitrunner:14943]     (inline):   3,245.91 ms
[com.oracle.svm.junit.svmjunitrunner:14943]    (compile):  10,054.97 ms
[com.oracle.svm.junit.svmjunitrunner:14943]      compile:  17,544.09 ms
[com.oracle.svm.junit.svmjunitrunner:14943]        image:     717.03 ms
[com.oracle.svm.junit.svmjunitrunner:14943]        write:     154.42 ms
[com.oracle.svm.junit.svmjunitrunner:14943]      [total]:  31,190.11 ms
Running: /home/cdrtz/projects/graal-ee-ws-0/graal/substratevm/svmbuild/linux-amd64/junit/tmpdWkVy_/com.oracle.svm.junit.svmjunitrunner --verbose
SVMJUnitCore
JUnit version 4.12
com.oracle.svm.test.GitHub1398 started (1 of 1)
  testMulticast: MulticastSocket: local=0.0.0.0/0.0.0.0:6789, remote=null
Passed
com.oracle.svm.test.GitHub1398 finished
Time: 0

OK (1 test)

dmlloyd · 2019-06-20T21:10:00Z

Thanks!

dmlloyd · 2019-06-20T21:13:51Z

Looks like there's one more problem. On it

cstancu · 2019-06-20T21:14:22Z

substratevm/src/com.oracle.svm.test/src/com/oracle/svm/test/GitHub1398.java

I don't really understand this comment. Also, can you please add some asserts in the test?

The success condition is simply that the test completes without error. I can try to think of something else to add though.

Even something as simple as an Assert.fail() on exception, like we do for GitHub1087 test.

...m/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixJavaNetSubstitutions.java

cstancu · 2019-06-20T21:18:09Z

...m/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixJavaNetSubstitutions.java

This leads to style check error: The value of the local variable ipaddress is not used

Hmm I think that might be unused in the upstream code as well. For now I just removed the useless declaration.

cstancu · 2019-06-20T21:21:46Z

...m/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixJavaNetSubstitutions.java

The value of the local variable ipaddress is not used.

...m/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixJavaNetSubstitutions.java

dmlloyd · 2019-06-20T22:14:04Z

It looks as though wrapping with IsDefined.__linux__() is not sufficient to make the code analyzer happy with the usage of LinuxIn, so I'll have to split this into two methods. Is it allowed to create e.g. a mcast_join_leave_Linux with @Platforms(Platform.LINUX.class) and similarly one for Darwin? Can I put both methods in this class, or should I make two more classes in this file for the Linux- and Darwin-specific versions?

cstancu · 2019-06-20T22:36:11Z

I wonder why IsDefined.__linux__() doesn't work as that should be constant folded.

dmlloyd · 2019-06-20T22:43:00Z

Yeah I was surprised too. It didn't give me a line number, it happened during pointsto analysis:

com.oracle.svm.core.util.UserError$UserException: type is not available in this platform: com.oracle.svm.core.posix.headers.linux.LinuxIn
Detailed message:

	at com.oracle.svm.core.util.UserError.abort(UserError.java:75)
	at com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:216)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:732)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:522)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:440)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: type is not available in this platform: com.oracle.svm.core.posix.headers.linux.LinuxIn
Detailed message:

	at com.oracle.graal.pointsto.constraints.UnsupportedFeatures.report(UnsupportedFeatures.java:130)
	at com.oracle.graal.pointsto.BigBang.finish(BigBang.java:565)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:683)
	... 8 more
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: type is not available in this platform: com.oracle.svm.core.posix.headers.linux.LinuxIn
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.createType(AnalysisUniverse.java:212)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:203)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:180)
	at com.oracle.graal.pointsto.meta.AnalysisType.getEnclosingType(AnalysisType.java:947)
	at com.oracle.svm.hosted.analysis.Inflation.checkType(Inflation.java:162)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1083)
	at com.oracle.svm.hosted.analysis.Inflation.checkObjectGraph(Inflation.java:136)
	at com.oracle.graal.pointsto.BigBang.checkObjectGraph(BigBang.java:603)
	at com.oracle.graal.pointsto.BigBang.finish(BigBang.java:558)
	... 9 more
Error: Image build request failed with exit status 1

cstancu · 2019-06-20T23:01:52Z

Can you try cleaning and rebuilding? I just finished running your latest version (from 3 hours ago) through our internal gate and I don't see the error: use of undeclared identifier 'IPV6_ADD_MEMBERSHIP' anymore, so I am not really sure that I understand where IsDefined.__linux__() is failing.

dmlloyd · 2019-06-20T23:21:27Z

OK, I'll give it a shot.

dmlloyd · 2019-06-20T23:45:52Z

I cleaned everything and rebuilt and the error remains. I've pushed up the commits with fixes as previously described, and hopefully you can get the same error I am.

bobmcwhirter · 2019-06-21T00:32:09Z

I have the wrong version of eclipse apparently. What are the correct versions?

…

On Thu, Jun 20, 2019 at 8:21 PM Codrut Stancu ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In substratevm/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixJavaNetSubstitutions.java <#1404 (comment)>: > + // 1891 * kernel releases. In the future it's possible that IP_ADD_MEMBERSHIP + // 1892 * will be updated to return ENOPROTOOPT if uses with an IPv6 socket (Solaris + // 1893 * already does this). Thus to cater for this we first try with the IPv4 + // 1894 * socket options and if they fail we use the IPv6 socket options. This + // 1895 * seems a reasonable failsafe solution. + // 1896 */ + // 1897 static void mcast_join_leave(JNIEnv *env, jobject this, + // 1898 jobject iaObj, jobject niObj, + // 1899 jboolean join) { + static void mcast_join_leave(Target_java_net_PlainDatagramSocketImpl self, InetAddress iaObj, NetworkInterface niObj, boolean join) throws IOException { + // The Linux code path uses a different mreq structure, so for type safety we have to factor it out completely. + if (IsDefined.__linux__()) { + // 1901 jobject fdObj = (*env)->GetObjectField(env, this, pdsi_fdID); + final FileDescriptor fdObj = self.fd; + // 1902 jint fd; + int fd; You should be able to run with mx gate --strict-mode --tags style <https://travis-ci.org/oracle/graal/jobs/548381854#L2783>, but you need to have the right versions of Eclipse JDT, pylint, checkstyle installed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1404?email_source=notifications&email_token=AAAD4T2IK4EPDQF7GIO34ZLP3QNI7A5CNFSM4HZKURH2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB4HEWWQ#discussion_r296058628>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAD4T6JGUCEFURAELPAL2LP3QNI7ANCNFSM4HZKURHQ> .

cstancu · 2019-06-21T01:02:10Z

@bobmcwhirter the one that travis downloads. That should be done automatically, I raised this issue with the right people already.

cstancu · 2019-06-21T02:01:15Z

@dmlloyd I run the latest changes in your PR through our internal gate (after fixing the remaining style issues) and all the gates pass. Do you see the error in one of your tests or in the general build?

dmlloyd · 2019-06-21T10:08:25Z

I'm seeing the error when I run mx native-unittest GitHub1398.

dmlloyd · 2019-06-21T10:36:19Z

Maybe you can sanity-check me. Here are the steps I'm running:

$ cd substratevm
$ mx clean
$ cd ../vm
$ mx clean
$ mx --dynamicimports /substratevm --disable-polyglot --disable-libpolyglot build
$ cd ../substratevm
$ mx native-unittest GitHub1398

Am I forgetting something obvious?

dmlloyd · 2019-06-21T10:41:17Z

I'm running through the steps one more time just to be certain...

dmlloyd · 2019-06-21T10:52:01Z

Same error. I'm going to update my JDK just to see if that matters.

Partially fixes oracle#1398

dmlloyd · 2019-06-21T10:59:11Z

One other thing to note is that I'm testing with Java 11, not Java 8.

dmlloyd · 2019-06-21T11:19:08Z

I wonder - the exception occurs in BigBang.finish; isn't this after constant folding would have been done?

dmlloyd · 2019-06-21T15:18:06Z

Pushed up another style fix.

cstancu · 2019-06-21T18:34:26Z

@dmlloyd the steps that you are running look good to me. I don't have access to a Mac to try to debug the issue; maybe @vjovanov can help.

BigBang.finish() is an intermediate step in the fixpoint analysis, but there should be constant folding happening before that. Plus, IsDefined.__linux__() is annotated with @Fold.

dmlloyd · 2019-06-24T13:33:42Z

I don't know what that last CI failure is all about. It looks like mx itself has crashed?

vjovanov · 2019-06-24T13:34:59Z

Ignore it, we have an issue for that.

dmlloyd · 2019-07-03T11:17:11Z

I factored out the method into two, one for Linux and one non-Linux, and that seems to have fixed the constant folding issue. I'll add that as a separate commit so that it can be easily individually reverted if/when the constant folding mystery is solved.

dmlloyd · 2019-07-03T11:17:51Z

Pushed.

cstancu · 2019-07-03T17:38:49Z

Thank you! I'll pull in the changes.

dmlloyd mentioned this pull request Jun 19, 2019

SubstrateVM: MulticastSocket.joinGroup() fails #1398

Closed

dmlloyd force-pushed the fix-1398 branch from e6720a1 to 585c2cf Compare June 19, 2019 15:11

graalvmbot added the oca-signed label Jun 19, 2019

cstancu self-assigned this Jun 19, 2019

dmlloyd force-pushed the fix-1398 branch from e6178eb to c48b58c Compare June 20, 2019 20:15

cstancu requested changes Jun 20, 2019

View reviewed changes

dmlloyd force-pushed the fix-1398 branch from c48b58c to a3f63a7 Compare June 20, 2019 23:43

Introduce implementation for PlainDatagramSocketImpl.mcast_join_leave

07f5e1a

Partially fixes oracle#1398

dmlloyd force-pushed the fix-1398 branch from a3f63a7 to 7246220 Compare June 21, 2019 10:58

Add test for MulticastSocket portion of oracle#1398

ed8ea80

dmlloyd force-pushed the fix-1398 branch from 7246220 to ed8ea80 Compare June 21, 2019 15:17

Split Linux and non-Linux into separate methods

e7edb18

cstancu added the native-image label Jul 3, 2019

graalvmbot merged commit e7edb18 into oracle:master Jul 3, 2019

dmlloyd deleted the fix-1398 branch July 4, 2019 00:21

Introduce implementation for PlainDatagramSocketImpl.mcast_join_leave #1404

Introduce implementation for PlainDatagramSocketImpl.mcast_join_leave #1404

Uh oh!

Conversation

dmlloyd commented Jun 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cstancu commented Jun 19, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

cstancu commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

cstancu commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

cstancu Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

dmlloyd Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

cstancu Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cstancu Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

dmlloyd Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

cstancu Jun 20, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

cstancu commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

cstancu commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

dmlloyd commented Jun 20, 2019

Uh oh!

bobmcwhirter commented Jun 21, 2019 via email

Uh oh!

cstancu commented Jun 21, 2019

Uh oh!

cstancu commented Jun 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 21, 2019

Uh oh!

cstancu commented Jun 21, 2019

Uh oh!

dmlloyd commented Jun 19, 2019 •

edited

Loading

cstancu commented Jun 21, 2019 •

edited

Loading