Cannot install agent in AWS environment

[Sam-Kruglov]

Comming from mockk/mockk#254
I have a Maven project called tmp with only two files:

tmp/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com</groupId>
    <artifactId>tmp</artifactId>
    <version>0.0.0</version>

    <properties>
        <maven.compiler.release>11</maven.compiler.release>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <dependencies>

        <dependency>
            <groupId>net.bytebuddy</groupId>
            <artifactId>byte-buddy-agent</artifactId>
            <version>1.9.10</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.4.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
            </plugin>
        </plugins>
    </build>

</project>

tmp/src/test/java/MyTest.java

import net.bytebuddy.agent.ByteBuddyAgent;
import org.junit.jupiter.api.Test;

public class MyTest {

    @Test
    void test() {
        ByteBuddyAgent.install();
    }
}

---

I run this with mvn clean test.
We have a Jenkins master running on AWS. It calls a slave node via SSH (also on AWS) to run the build.
Locally and in Jenkins I use the exact same Docker image but it fails only in Jenkins (not sure if it will fail also without Docker I could check that later).

---

On Jenkins it throws java.lang.IllegalStateException: Could not self-attach to current VM using external process

Here is full debug log of this command: build.log

Here is the generated streamdump (had to append the extention .log for GitHub): 2019-02-14T12-25-31_589-jvmRun1.dumpstream.log

[Sam-Kruglov]

As I said, I use the exact same setup locally and in Jenkins and they use Open JDK 11.0.1. I also tried setting <maven.compiler.release>1.8</maven.compiler.release> but that didn't help.

[raphw]

I assume that it might not be possible to start a process on AWS? It is unfortunately required to attach the agent on JVMs post 8: https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/src/main/java/net/bytebuddy/agent/ByteBuddyAgent.java#L476

You can set -Djdk.attach.allowAttachSelf=true during your build to avoid this requirement.

[Sam-Kruglov]

My root problem is that I can't run my tests that are written using mockk library. So whenever a mock is created they call this ByteBuddyAgent.install(). Mockito however works fine because they use a different technique. see mockk/mockk#254 (comment)

[Sam-Kruglov]

@raphw thanks, I will try this right now!

[Sam-Kruglov]

@raphw now prints another error:

java.lang.IllegalStateException: Error during attachment using: net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@5aebe890

	at MyTest.test(MyTest.java:8)

Caused by: java.lang.reflect.InvocationTargetException

	at MyTest.test(MyTest.java:8)

Caused by: java.io.IOException: Can not attach to current VM

	at MyTest.test(MyTest.java:8)

What do you mean by creating a process?

[raphw]

Byte Buddy needs to start a helper process as the JVM forbids to attach to your own JVM unless this property is set. However, it seems like it is generally not possible to attach in your enviornment, maybe the JDK being used does not include the attachment API? If its a linux machone, you can include this as a test dependency where Byte Buddy attempts emulation: https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/pom.xml#L32

[Sam-Kruglov]

@raphw it is Amazon Linux 2. I tried including this dependency but it didn't change anything.
I use the same JDK locally and it works. Also, note this happens inside Docker, maybe that has something to do with it. Although again, locally works in Docker too.

[raphw]

I assume that it is caused by some environmental restriction enforced by Amazon to using the attach API. You could also try to find out if java.home is set to a strange property on AWS as it is required to resolve a location.

[Sam-Kruglov]

Also figured that using -Djdk.attach.allowAttachSelf=true build fails even locally

[raphw]

That is interesting. Could you set some breakpoints throughout ByteBuddyAgent and check which attachment provider is used?

[Sam-Kruglov]

@raphw ForModularizedVm

[Sam-Kruglov]

@raphw JAVA_HOME is surely valid since I have the jdk installed inside docker

[Sam-Kruglov]

@raphw I have removed byte buddy from the project and placed new ProcessBuilder("echo", "hey").start(); in the test. It passed on AWS. So the problem must be with the exact command we are trying to execute?

processId is the current PID? I figured there is a one-liner since java 9 ProcessHandle.current().pid()

[raphw]

Did you include the jdk.attach module in the build?

[Sam-Kruglov]

@raphw perhaps not, how do I do it?

[raphw]

It's only part of a JDK. The module is named jdk.attach. The parameter name is add-module, it is documented as a JVM option.

[flaredragon]

Hey I found a work through for this, for some reason mvn clean install -Djdk.attach.allowAttachSelf=true fails locally and on docker.

Putting this argument in pom.xml of the module using byte buddy solves this issue for both local and docker
https://github.com/assylias/jBloomberg/blob/1a23c3ea4064335a145654867cf31ec68d2e975f/pom.xml#L100
Now running mvn clean install is successful.

[raphw]

Then it seems like you cannot start up a new process in this environment which is how Byte Buddy normally works around this issue.

Glad you got it figured out!

[Sam-Kruglov]

Yes! Putting it in surefire makes the difference because it is responsible for starting the process.
But I had to add =true like this:

<argLine>
    -Djdk.attach.allowAttachSelf=true <!-- required to run test with jmockit on JDK 9 -->
</argLine>

Unfortunately, I just get another error:

java.lang.IllegalStateException: Error during attachment using: net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@6a03bcb1

	at MyTest.test(MyTest.java:8)

Caused by: java.lang.reflect.InvocationTargetException

	at MyTest.test(MyTest.java:8)

Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/54/root/tmp/.java_pid54: target process 54 doesn't respond within 10500ms or HotSpot VM not loaded

	at MyTest.test(MyTest.java:8)

[raphw]

That is a very particular error issued by HotSpot. The attach API communicates by socket files. Might it be that those files cannot be written to the file system or are deleted by a routine job?

[Sam-Kruglov]

I have resolved the problem by also adding -XX:+StartAttachListener parameter!

see https://stackoverflow.com/a/25439003/6166627