Changing CXXFLAGS and LDFLAGS is ignored due to Gradle daemon

[avsdev-cw]

Apologies for the length, I've tried to provide as much detail as possible. I'm trying to natively compile this on an nVidia Jetson TX2 board (aarch64)

What version of gRPC are you using?

nvidia@tegra-ubuntu:~/grpc-java$ git show --oneline -s

903197b core: StreamTracer (#2863)

What JVM are you using (java -version)?

nvidia@tegra-ubuntu:~/grpc-java$ java -version

openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

What did you do?

nvidia@tegra-ubuntu:~$ git clone https://github.com/google/protobuf.git && cd protobuf
nvidia@tegra-ubuntu:~/protobuf$ git checkout v3.1.0
nvidia@tegra-ubuntu:~/protobuf$ ./autogen.sh
nvidia@tegra-ubuntu:~/protobuf$ ./configure --prefix=/opt/protobuf
nvidia@tegra-ubuntu:~/protobuf$ make && make install
nvidia@tegra-ubuntu:~/protobuf$ export PATH=/opt/protobuf/bin/:$PATH
nvidia@tegra-ubuntu:~/protobuf$ export LD_LIBRARY_PATH=/opt/protobuf/lib/:$LD_LIBRARY_PATH
nvidia@tegra-ubuntu:~/protobuf$ export CXXFLAGS=-I/opt/protobuf/include
nvidia@tegra-ubuntu:~/protobuf$ export LDFLAGS=-L/opt/protobuf/lib
nvidia@tegra-ubuntu:~/protobuf$ ls -l /opt/protobuf/{lib,bin,include/google/protobuf/io}

/opt/protobuf/bin:
total 268
-rwxr-xr-x 1 root root 272752 Apr  7 13:47 protoc

/opt/protobuf/include/google/protobuf/io:
total 160
-rw-r--r-- 1 root root 59255 Apr  7 13:47 coded_stream.h
-rw-r--r-- 1 root root  6598 Apr  7 13:47 gzip_stream.h
-rw-r--r-- 1 root root 16666 Apr  7 13:47 printer.h
-rw-r--r-- 1 root root  2431 Apr  7 13:47 strtod.h
-rw-r--r-- 1 root root 16598 Apr  7 13:47 tokenizer.h
-rw-r--r-- 1 root root 10185 Apr  7 13:47 zero_copy_stream.h
-rw-r--r-- 1 root root 13655 Apr  7 13:47 zero_copy_stream_impl.h
-rw-r--r-- 1 root root 16753 Apr  7 13:47 zero_copy_stream_impl_lite.h

/opt/protobuf/lib:
total 209472
-rw-r--r-- 1 root root 60953590 Apr  7 13:47 libprotobuf.a
-rwxr-xr-x 1 root root      991 Apr  7 13:47 libprotobuf.la
-rw-r--r-- 1 root root  5724374 Apr  7 13:47 libprotobuf-lite.a
-rwxr-xr-x 1 root root     1026 Apr  7 13:47 libprotobuf-lite.la
lrwxrwxrwx 1 root root       26 Apr  7 13:47 libprotobuf-lite.so -> libprotobuf-lite.so.12.0.0
lrwxrwxrwx 1 root root       26 Apr  7 13:47 libprotobuf-lite.so.12 -> libprotobuf-lite.so.12.0.0
-rwxr-xr-x 1 root root  2390392 Apr  7 13:47 libprotobuf-lite.so.12.0.0
lrwxrwxrwx 1 root root       21 Apr  7 13:47 libprotobuf.so -> libprotobuf.so.12.0.0
lrwxrwxrwx 1 root root       21 Apr  7 13:47 libprotobuf.so.12 -> libprotobuf.so.12.0.0
-rwxr-xr-x 1 root root 24405752 Apr  7 13:47 libprotobuf.so.12.0.0
-rw-r--r-- 1 root root 91333170 Apr  7 13:47 libprotoc.a
-rwxr-xr-x 1 root root     1010 Apr  7 13:47 libprotoc.la
lrwxrwxrwx 1 root root       19 Apr  7 13:47 libprotoc.so -> libprotoc.so.12.0.0
lrwxrwxrwx 1 root root       19 Apr  7 13:47 libprotoc.so.12 -> libprotoc.so.12.0.0
-rwxr-xr-x 1 root root 29662544 Apr  7 13:47 libprotoc.so.12.0.0
drwxr-xr-x 2 root root     4096 Apr  7 13:47 pkgconfig

nvidia@tegra-ubuntu:~/protobuf$ cd ~/
nvidia@tegra-ubuntu:~$ git clone https://github.com/grpc/grpc-java.git && cd grpc-java
nvidia@tegra-ubuntu:~/grpc-java$ ./gradlew build

*** Building codegen requires Protobuf version 3.2.0
*** Please refer to https://github.com/grpc/grpc-java/blob/master/COMPILING.md#how-to-build-code-generation-plugin

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':grpc-compiler'.
> Exception thrown while executing model rule: NativeComponentModelPlugin.Rules#createBinaries(TargetedNativeComponentInternal, PlatformResolvers, BuildTypeContainer, FlavorContainer, ServiceRegistry)
   > Invalid NativePlatform: linux_aarch64

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 2.646 secs

Googled for a bit and found this discussion amongst others so made some mods...

nvidia@tegra-ubuntu:~/grpc-java$ git diff

diff --git a/compiler/build.gradle b/compiler/build.gradle
index 10d9c31..cfd1850 100644
--- a/compiler/build.gradle
+++ b/compiler/build.gradle
@@ -48,6 +48,9 @@ model {
     }
     gcc(Gcc) {
       target("ppcle_64")
+      target("linux_aarch64"){
+        cppCompiler.executable = "/usr/bin/gcc"
+      }
     }
     clang(Clang) {
     }
@@ -60,6 +63,9 @@ model {
     x86_64 {
       architecture "x86_64"
     }
+    linux_aarch64 {
+      architecture "arm64"
+    }
     ppcle_64 {
       architecture "ppcle_64"
     }
@@ -67,7 +73,7 @@ model {
 
   components {
     java_plugin(NativeExecutableSpec) {
-      if (arch in ['x86_32', 'x86_64', 'ppcle_64']) {
+      if (arch in ['x86_32', 'x86_64', 'arm64', 'ppcle_64']) {
         // If arch is not within the defined platforms, we do not specify the
         // targetPlatform so that Gradle will choose what is appropriate.
         targetPlatform arch

nvidia@tegra-ubuntu:~/grpc-java$ ./gradlew build

*** Building codegen requires Protobuf version 3.2.0
*** Please refer to https://github.com/grpc/grpc-java/blob/master/COMPILING.md#how-to-build-code-generation-plugin
:grpc-context:compileJava
:grpc-context:processResources NO-SOURCE
:grpc-context:classes
:grpc-context:jar
:grpc-core:compileJava
:grpc-core:processResources
:grpc-core:classes
:grpc-core:jar
:grpc-auth:compileJava
:grpc-auth:processResources NO-SOURCE
:grpc-auth:classes
:grpc-auth:jar
:grpc-netty:compileJava
:grpc-netty:processResources
:grpc-netty:classes
:grpc-netty:jar
:grpc-okhttp:compileJava
:grpc-okhttp:processResources
:grpc-okhttp:classes
:grpc-okhttp:jar
:grpc-compiler:compileJava_pluginExecutableJava_pluginCppIn file included from /home/nvidia/grpc-java/compiler/src/java_plugin/cpp/java_generator.cpp:1:0:
/home/nvidia/grpc-java/compiler/src/java_plugin/cpp/java_generator.h:8:49: fatal error: google/protobuf/io/zero_copy_stream.h: No such file or directory
compilation terminated.

In file included from /home/nvidia/grpc-java/compiler/src/java_plugin/cpp/java_plugin.cpp:8:0:
/home/nvidia/grpc-java/compiler/src/java_plugin/cpp/java_generator.h:8:49: fatal error: google/protobuf/io/zero_copy_stream.h: No such file or directory
compilation terminated.

/home/nvidia/grpc-java/compiler/src/java_plugin/cpp/java_generator.h:8:49: fatal error: google/protobuf/io/zero_copy_stream.h: No such file or directory
compilation terminated.

 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':grpc-compiler:compileJava_pluginExecutableJava_pluginCpp'.
> Multiple build operations failed.
      C++ compiler failed while compiling java_generator.cpp.
      C++ compiler failed while compiling java_plugin.cpp.
      C++ compiler failed while compiling java_generator.h.
  See the complete log at: file:///home/nvidia/grpc-java/compiler/build/tmp/compileJava_pluginExecutableJava_pluginCpp/output.txt

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 43.251 secs

What did you expect to see?

Successful build

What did you see instead?

In short:

/home/nvidia/grpc-java/compiler/src/java_plugin/cpp/java_generator.h:8:49: fatal error: google/protobuf/io/zero_copy_stream.h: No such file or directory
compilation terminated.

despite protobuf being compiled & installed and CXXFLAGS and LDFLAGS being set.

Also tried adding the 4 exports above at the end of my .bashrc file as it seemed to have helped in [this issue] (#1346)

[avsdev-cw]

After further experimenting this seems to be a fundamental problem with gradle's running environment:

nvidia@tegra-ubuntu:~/grpc-java$ cat compiler/build.gradle <<EOF

task foobar {
    def value = System.getenv("LDFLAGS")
    print "LDFLAGS: "
    print value
    if (value != null) {
      println " (not null)"
    } else {
      println " (null)"
    }

    println System.getenv()
}
EOF
nvidia@tegra-ubuntu:~/grpc-java$ LDFLAGS=-L/opt/protobuf/lib CXXFLAGS=-I/opt/protobuf/include ./gradlew -q foobar
*** Building codegen requires Protobuf version 3.2.0
*** Please refer to https://github.com/grpc/grpc-java/blob/master/COMPILING.md#how-to-build-code-generation-plugin
LDFLAGS:  (not null)
[PATH:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, TERM:xterm, LANG:en_US.UTF-8, LDFLAGS:, XDG_SESSION_ID:2, MAIL:/var/mail/nvidia, LD_LIBRARY_PATH:, LOGNAME:nvidia, PWD:/home/nvidia/grpc-java, _:./gradlew, SHELL:/bin/bash, OLDPWD:/home/nvidia/grpc-java, USER:nvidia, HOME:/home/nvidia, CXXFLAGS:]

[avsdev-cw]

Colour me stunned... on a whim I have just tried the --no-daemon switch:

nvidia@tegra-ubuntu:~/grpc-java$ LDFLAGS=-L/opt/protobuf/lib CXXFLAGS=-I/opt/protobuf/include ./gradlew --no-daemon -q foobar
*** Building codegen requires Protobuf version 3.2.0
*** Please refer to https://github.com/grpc/grpc-java/blob/master/COMPILING.md#how-to-build-code-generation-plugin
LDFLAGS: -L/opt/protobuf/lib (not null)
[PATH:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, TERM:xterm, LANG:en_US.UTF-8, LDFLAGS:-L/opt/protobuf/lib, XDG_SESSION_ID:2, MAIL:/var/mail/nvidia, LD_LIBRARY_PATH:, LOGNAME:nvidia, PWD:/home/nvidia/grpc-java, _:./gradlew, SHELL:/bin/bash, OLDPWD:/home/nvidia/grpc-java, USER:nvidia, HOME:/home/nvidia, CXXFLAGS:-I/opt/protobuf/include]

Proceeding with full build...

[ejona86]

@avsdev-cw, oooh. Good sleuthing. That's interesting.

That actually makes some sense. When the code was added the gradle daemon didn't exist yet (IIRC). And I'd expect if your LDFLAGS/CXXFLAGS existed on the first invocation that the daemon will have them. We don't notice this because we don't need to modify our LDFLAGS/CXXFLAGS much.

So I'd assume the workarounds would be to use --no-daemon (as you mentioned), or to --stop the daemon when you change the environment variables.

This looks like it should be reproducable on x86.

[avsdev-cw]

It is reproducible on x86_64 so I assume the same is true for x86.

I should probably open another issue for this but now I am stuck here:

:grpc-interop-testing:test[jetty-alpn-agent] Using: alpn-boot-8.1.11.v20170118.jar


io.grpc.testing.integration.Http2NettyTest > classMethod FAILED
    java.lang.UnsatisfiedLinkError: failed to load the required native library
        at io.netty.handler.ssl.OpenSsl.ensureAvailability(OpenSsl.java:324)
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:216)
        at io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:42)
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:344)
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:334)
        at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:414)
        at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:402)
        at io.grpc.testing.integration.Http2NettyTest.startServer(Http2NettyTest.java:68)

        Caused by:
        java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-aarch_64, netty-tcnative-linux-aarch_64-fedora, netty-tcnative]

250 tests completed, 1 failed, 4 skipped
:grpc-interop-testing:test FAILED

I've tried compiling multiple versions of netty-tcnative (1.1.33-Fork26, 1.1.33.Fork16, -parent-2.0.0.Final) and each compiles and tests fine internally. Not sure why gradle tests fail though...

[ejona86]

@avsdev-cw, yeah, another issue or we can converse on the grpc-io mailing list.

I've tried compiling multiple versions of netty-tcnative

You may try specifying a log.properties. You can add jvmArgs to gradle's test {}, and then could look at the output in the junit report.

[zpencer]

I'm poking around this bug but I can't seem to reproduce this issue. Is there something wrong with the steps I took?

First, try ./gradlew :grpc-compiler:build and verified everything is starts off working:

$ git checkout 903197b
HEAD is now at 903197b2... core: StreamTracer (#2863)
$ ./gradlew --stop
Stopping Daemon(s)
1 Daemon stopped
$ ./gradlew --status
No Gradle daemons are running.
   PID STATUS   INFO
 14488 STOPPED  (stop command received)
 28929 STOPPED  (stop command received)

Only Daemons for the current Gradle version are displayed. See https://docs.gradle.org/3.4.1/userguide/gradle_daemon.html#sec:status
$ echo $LDFLAGS
-L/home/spencerfang/opt/protobuf/3.3.2/lib
$ echo $CXXFLAGS
-I/home/spencerfang/opt/protobuf/3.3.2/include
$ echo $LD_LIBRARY_PATH
/home/spencerfang/opt/protobuf/3.3.2/lib/

$ ./gradlew :grpc-compiler:build --debug 2>&1 > /tmp/working ; echo $?
0
$ ./gradlew --status
   PID STATUS   INFO
  8169 IDLE     3.4.1
 14488 STOPPED  (stop command received)
 28929 STOPPED  (stop command received)

Only Daemons for the current Gradle version are displayed. See https://docs.gradle.org/3.4.1/userguide/gradle_daemon.html#sec:status

Then I changed the three flags to something that I expect to break the build, and confirmed the build broke. This fails to reproduce the problem where the current env variables are not transmitted to the gradle daemon.

$ ./gradlew :grpc-compiler:build --debug 2>&1 > /tmp/bad_flags ; echo $?
09:33:23.075 [ERROR] [org.gradle.internal.operations.logging.DefaultBuildOperationLoggerFactory] /home/spencerfang/git/grpc-java/compiler/src/java_plugin/cpp/java_plugin.cpp:9:53: fatal error: google/protobuf/compiler/code_generator.h: No such file or directory
 #include <google/protobuf/compiler/code_generator.h>
                                                     ^
compilation terminated.

09:33:23.335 [ERROR] [org.gradle.internal.operations.logging.DefaultBuildOperationLoggerFactory] /home/spencerfang/git/grpc-java/compiler/src/java_plugin/cpp/java_generator.cpp:7:54: fatal error: google/protobuf/compiler/java/java_names.h: No such file or directory
 #include <google/protobuf/compiler/java/java_names.h>
                                                      ^
compilation terminated.

09:33:23.350 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
09:33:23.351 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
09:33:23.351 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
09:33:23.352 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
09:33:23.352 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':grpc-compiler:compileJava_pluginExecutableJava_pluginCpp'.
09:33:23.352 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Multiple build operations failed.
09:33:23.353 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       C++ compiler failed while compiling java_plugin.cpp.
09:33:23.353 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       C++ compiler failed while compiling java_generator.cpp.
09:33:23.353 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   See the complete log at: file:///home/spencerfang/git/grpc-java/compiler/build/tmp/compileJava_pluginExecutableJava_pluginCpp/output.txt
09:33:23.354 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
09:33:23.354 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:
09:33:23.356 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace. 
1

$ grep CXXFLAGS /tmp/bad_flags 

# Modified the output here for clarity
09:33:22.158 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Environment for process 'command '/usr/bin/gcc'': {LDFLAGS=-L/break/it/, ..., CXXFLAGS=-I/break/it/, ..., LD_PRELOAD_PATH=/break/it}

$ ./gradlew --status
   PID STATUS   INFO
  8169 IDLE     3.4.1
 14488 STOPPED  (stop command received)
 28929 STOPPED  (stop command received)

[ejona86]

Yes, if you stop the daemon then the next daemon will start correctly (I mentioned it was a workaround). But the fact you need to stop the old daemon is the problem, because that's non-obvious and when trying to get things working initially you're very likely to be changing the environment variables.

[zpencer]

I killed the old daemon to show I started off with a fresh environment. But between the two runs, I kept the same daemon (PID is 8169). Shouldn't we expect that the flags remain stuck in this case?