File descriptors of datadog temp jars are not released

[athom]

Hi there,

I was encountering an issue, it looks like the datadog jar are loaded and deleted, but still left fd when you using lsof, you can see it there.

It is very similar to
#938
#1285
Any one know what is the issue?

It can be reproduced by a helloworld file.

main.java

package main;

import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;

public class Main {
    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress(8000), 0);
        server.createContext("/test", new MyHandler());
        server.setExecutor(null); // creates a default executor
        server.start();
    }

    static class MyHandler implements HttpHandler {
        @Override
        public void handle(HttpExchange t) throws IOException {
            String response = "This is the response";
            t.sendResponseHeaders(200, response.length());
            OutputStream os = t.getResponseBody();
            os.write(response.getBytes());
            os.close();
        }
    }

}

Together with a Dockerfile

FROM maven:3.6-jdk-11-slim as application

# for debugging
RUN java -version

WORKDIR /app

# Install runtime dependencies
RUN apt-get -qq update && \
    apt-get -qq install -y --no-install-recommends \
      curl \
      locales \
      telnet \
      wget \
      unzip \
    && rm -rf /var/lib/apt/lists/*

# Set locale: https://daten-und-bass.io/blog/fixing-missing-locale-setting-in-ubuntu-docker-image/
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend noninteractive locales && \
    update-locale LANG=en_US.UTF-8
ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

# Update timezone
ENV TZ=Asia/Singapore

EXPOSE 8080

# Download datadog javaagent: https://docs.datadoghq.com/tracing/setup/java/#installation-and-getting-started
RUN curl 'https://repository.sonatype.org/service/local/artifact/maven/redirect?r=central-proxy&g=com.datadoghq&a=dd-java-agent&v=0.57.0' \
    -L -s -o ./dd-java-agent.jar

COPY . .

#CMD java main.java
CMD java \
  -javaagent:/app/dd-java-agent.jar \
  -Ddd.trace.global.tags="env:test" \
  -Ddd.service.name="curiosity-test" \
  -Ddd.agent.host='172.17.0.1' \
  -Ddd.trace.agent.port=8126 \
  -Ddd.http.server.tag.query-string=TRUE \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.local.only=false \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.port=7091 \
  -Djava.rmi.server.useLocalHostname=true \
  main.java

Then I run it on AWS ECS with datadog daemon installed and running.
Then I check in the Host, it indicate the jar file deleted.

[root@ip-10-230-1-27 ~]# lsof -p 122085
COMMAND    PID USER   FD      TYPE DEVICE  SIZE/OFF     NODE NAME
java    122085 root  cwd       DIR  0,101      4096  1705724 /app
java    122085 root  rtd       DIR  0,101      4096  1705733 /
java    122085 root  txt       REG  0,101     13349  1449141 /usr/local/openjdk-11/bin/java
java    122085 root  mem       REG  202,1            1449141 /usr/local/openjdk-11/bin/java (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578933 /usr/local/openjdk-11/lib/libmanagement_ext.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578931 /usr/local/openjdk-11/lib/libmanagement.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578916 /usr/local/openjdk-11/lib/libextnet.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1705717 /app/dd-java-agent.jar (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578935 /usr/local/openjdk-11/lib/libnet.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578936 /usr/local/openjdk-11/lib/libnio.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578946 /usr/local/openjdk-11/lib/modules (stat: No such file or directory)
java    122085 root  DEL       REG  202,1            1705800 /tmp/datadog-temp-jars3502052644790155051/jarJO8BSUZw.jar
java    122085 root  DEL       REG  202,1            1705798 /tmp/datadog-temp-jars6674992106479432760/jar4IqRKG6G.jar
java    122085 root  DEL       REG  202,1            1705796 /tmp/datadog-temp-jars1465922631243842544/jarCDrPpsMA.jar
java    122085 root  DEL       REG  202,1            1705794 /tmp/datadog-temp-jars1342511038984961876/jaruk2ZZ1oR.jar
java    122085 root  DEL       REG  202,1            1705792 /tmp/datadog-temp-jars12276611993146532493/jary4YPcxym.jar
java    122085 root  DEL       REG  202,1            1705790 /tmp/datadog-temp-jars4490040634254223383/jarPWDIm9SV.jar
java    122085 root  DEL       REG  202,1            1705788 /tmp/datadog-temp-jars7220689836028653051/jar39Z1VKk5.jar
java    122085 root  DEL       REG  202,1            1705786 /tmp/datadog-temp-jars7032575659553622168/jarCASvKBnW.jar
java    122085 root  DEL       REG  202,1            1705784 /tmp/datadog-temp-jars12412375479428152682/jarjZNnBewK.jar
java    122085 root  DEL       REG  202,1            1705782 /tmp/datadog-temp-jars5143451830426337371/jarVhV4FRUJ.jar
java    122085 root  DEL       REG  202,1            1705780 /tmp/datadog-temp-jars4056429630377778383/jar7UlBkI0i.jar
java    122085 root  DEL       REG  202,1            1705778 /tmp/datadog-temp-jars6663744072196803405/jaradSSLG9G.jar
java    122085 root  DEL       REG  202,1            1705776 /tmp/datadog-temp-jars4835615375040176392/jarTt5LIh3L.jar
java    122085 root  DEL       REG  202,1            1705774 /tmp/datadog-temp-jars2301966173302611815/jarUGDVODEk.jar
java    122085 root  mem       REG  202,1            1705701 /usr/lib/locale/locale-archive (path inode=1187736)
java    122085 root  mem       REG  202,1            1578927 /usr/local/openjdk-11/lib/libjimage.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578945 /usr/local/openjdk-11/lib/libzip.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1705764 /tmp/hsperfdata_root/14 (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050595 /lib/x86_64-linux-gnu/libnss_files-2.28.so (stat: No such file or directory)
java    122085 root  DEL       REG  202,1            1705772 /tmp/datadog-temp-jars16829642410582105734/jarhQB4ONaO.jar
java    122085 root  DEL       REG  202,1            1705770 /tmp/datadog-temp-jars1093428737908816706/jarFtpfL6MF.jar
java    122085 root  DEL       REG  202,1            1705768 /tmp/datadog-temp-jars11357735538402901526/jarz6gCOHLH.jar
java    122085 root  mem       REG  202,1            1578918 /usr/local/openjdk-11/lib/libinstrument.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578923 /usr/local/openjdk-11/lib/libjava.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578944 /usr/local/openjdk-11/lib/libverify.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050616 /lib/x86_64-linux-gnu/librt-2.28.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050573 /lib/x86_64-linux-gnu/libgcc_s.so.1 (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050580 /lib/x86_64-linux-gnu/libm-2.28.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1051323 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578957 /usr/local/openjdk-11/lib/server/libjvm.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050557 /lib/x86_64-linux-gnu/libc-2.28.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050565 /lib/x86_64-linux-gnu/libdl-2.28.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1578907 /usr/local/openjdk-11/lib/jli/libjli.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050612 /lib/x86_64-linux-gnu/libpthread-2.28.so (stat: No such file or directory)
java    122085 root  mem       REG  202,1            1050637 /lib/x86_64-linux-gnu/libz.so.1.2.11 (stat: No such file or directory)
java    122085 root  DEL       REG  202,1            1705766 /tmp/datadog-temp-jars13401894934582242722/jarUWcZeWf7.jar
java    122085 root  mem       REG  202,1            1050543 /lib/x86_64-linux-gnu/ld-2.28.so (stat: No such file or directory)
java    122085 root    0u      CHR    1,3       0t0 76071711 /dev/null
java    122085 root    1w     FIFO   0,11       0t0 76072460 pipe
java    122085 root    2w     FIFO   0,11       0t0 76072461 pipe
java    122085 root    3r      REG  0,101 141501679  1578946 /usr/local/openjdk-11/lib/modules
java    122085 root    4r      REG  0,101  16390779  1705717 /app/dd-java-agent.jar
java    122085 root    5r      REG  0,101  16390779  1705717 /app/dd-java-agent.jar
java    122085 root    6r      CHR    1,8       0t0 76071712 /dev/random
java    122085 root    7r      CHR    1,9       0t0 76071716 /dev/urandom
java    122085 root    8u     sock    0,8       0t0 76078397 protocol: UDP
java    122085 root    9r      REG  0,101       721  1705766 /tmp/datadog-temp-jars13401894934582242722/jarUWcZeWf7.jar (deleted)
java    122085 root   10r      REG  0,101      2011  1705768 /tmp/datadog-temp-jars11357735538402901526/jarz6gCOHLH.jar (deleted)
java    122085 root   11r      REG  0,101      2104  1705770 /tmp/datadog-temp-jars1093428737908816706/jarFtpfL6MF.jar (deleted)
java    122085 root   12r      REG  0,101      5972  1705772 /tmp/datadog-temp-jars16829642410582105734/jarhQB4ONaO.jar (deleted)
java    122085 root   13r      REG  0,101       700  1705774 /tmp/datadog-temp-jars2301966173302611815/jarUGDVODEk.jar (deleted)
java    122085 root   14r      REG  0,101      1990  1705776 /tmp/datadog-temp-jars4835615375040176392/jarTt5LIh3L.jar (deleted)
java    122085 root   15r      REG  0,101       726  1705778 /tmp/datadog-temp-jars6663744072196803405/jaradSSLG9G.jar (deleted)
java    122085 root   16r      REG  0,101      2015  1705780 /tmp/datadog-temp-jars4056429630377778383/jar7UlBkI0i.jar (deleted)
java    122085 root   17r      REG  0,101      2795  1705782 /tmp/datadog-temp-jars5143451830426337371/jarVhV4FRUJ.jar (deleted)
java    122085 root   18r      REG  0,101      7953  1705784 /tmp/datadog-temp-jars12412375479428152682/jarjZNnBewK.jar (deleted)
java    122085 root   19r      REG  0,101       700  1705786 /tmp/datadog-temp-jars7032575659553622168/jarCASvKBnW.jar (deleted)
java    122085 root   20r      REG  0,101      1990  1705788 /tmp/datadog-temp-jars7220689836028653051/jar39Z1VKk5.jar (deleted)
java    122085 root   21r      REG  0,101      4538  1705790 /tmp/datadog-temp-jars4490040634254223383/jarPWDIm9SV.jar (deleted)
java    122085 root   22r      REG  0,101       801  1705792 /tmp/datadog-temp-jars12276611993146532493/jary4YPcxym.jar (deleted)
java    122085 root   23r      REG  0,101      2093  1705794 /tmp/datadog-temp-jars1342511038984961876/jaruk2ZZ1oR.jar (deleted)
java    122085 root   24u     sock    0,8       0t0 76075663 protocol: UNIX
java    122085 root   25r      REG  0,101       712  1705796 /tmp/datadog-temp-jars1465922631243842544/jarCDrPpsMA.jar (deleted)
java    122085 root   26r      REG  0,101      7853  1705798 /tmp/datadog-temp-jars6674992106479432760/jar4IqRKG6G.jar (deleted)
java    122085 root   27u     sock    0,8       0t0 76075665 protocol: TCP
java    122085 root   28u     sock    0,8       0t0 76067482 protocol: UNIX
java    122085 root   29r      REG  0,101       694  1705800 /tmp/datadog-temp-jars3502052644790155051/jarJO8BSUZw.jar (deleted)
java    122085 root   30u     sock    0,8       0t0 76075668 protocol: TCP
java    122085 root   31u     sock    0,8       0t0 76075672 protocol: TCP
java    122085 root   32u  a_inode   0,12         0    10725 [eventpoll]
java    122085 root   33r     FIFO   0,11       0t0 76075673 pipe
java    122085 root   34w     FIFO   0,11       0t0 76075673 pipe

[lpriima]

we still can have lingering FDs in ByteBuddy:

https://github.com/raphw/byte-buddy/blob/d9a5b3af63d12730c39d15ea6830933383c648d4/byte-buddy-dep/src/main/java/net/bytebuddy/dynamic/loading/ClassInjector.java#L2320

[randomanderson]

This is behaving as expected. The only way to add classes to the bootstrap classpath is by creating a jar. We use this mechanism, through ByteBuddy, and delete the jars when we are done. Because the JVM still references the jars, they are not released until it shutdown.

In the future, we are looking for ways of not having to inject the bootstrap classpath. That effort is being tracked in #429