Diagnostic tool failed to get any data, no diagnostics.log, need troubleshooting guidance

[hubbleview]

This environment is hardened.

When running diagnostic utility by root account, it didn't collect any data and failed to create archive.

[root@elasticsearch-1 diagnostics-8.2.2]# ./diagnostics.sh --host localhost -u elastic -p --port 9200 --ssl --type api --noVerify
Using /bin/java as Java Runtime
Using -Xms256m -Xmx2000m for options.
2021-10-13 22:11:02,429 main ERROR Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...
Elasticsearch user password: ********************
Closing loggers.
Archiving diagnostic results.
Archive: -20211013-221210.zip was created
Couldn't create zip archive. Trying tar.gz
Archive: -20211013-221210.tar.gz was created
Couldn't create tar.gz archive.
Fatal error occurred: Couldn't create zip and tar.gz archives.. Check diagnostics.log in the archive file for more detail.

No diagnostics.log can be found in the directory.

[root@elasticsearch-1 diagnostics-8.2.2]# ls -altr
total 132
-rwxr-xr-x. 1 root root 90 Feb 11 2021 docker-build.sh
-rwxr-xr-x. 1 root root 960 Jun 28 16:31 scrub.sh
-rw-r--r--. 1 root root 875 Jun 28 16:31 scrub.bat
-rwxr-xr-x. 1 root root 992 Jun 28 16:31 import-monitoring.sh
-rw-r--r--. 1 root root 891 Jun 28 16:31 import-monitoring.bat
-rwxr-xr-x. 1 root root 992 Jun 28 16:31 export-monitoring.sh
-rw-r--r--. 1 root root 891 Jun 28 16:31 export-monitoring.bat
-rwxr-xr-x. 1 root root 982 Jun 28 16:31 diagnostics.sh
-rw-r--r--. 1 root root 883 Jun 28 16:31 diagnostics.bat
-rw-r--r--. 1 root root 53171 Sep 13 15:42 README.md
-rw-r--r--. 1 root root 3859 Sep 13 15:42 LICENSE.txt
-rw-r--r--. 1 root root 5744 Sep 16 12:18 NOTICE.txt
drwxr-xr-x. 2 root root 4096 Sep 16 12:41 lib
drwxr-xr-x. 2 root root 60 Sep 16 12:41 docker
drwxr-xr-x. 3 root root 183 Sep 16 12:41 config
dr-xr-x---. 9 root root 4096 Oct 13 19:54 ..
-rw-------. 1 root root 47 Oct 13 19:54 -20211013-195414.zip
-rw-------. 1 root root 101 Oct 13 19:54 -20211013-195414.tar.gz
-rw-------. 1 root root 0 Oct 13 19:58 startup.log
-rw-------. 1 root root 47 Oct 13 20:00 -20211013-200006.zip
-rw-------. 1 root root 100 Oct 13 20:00 -20211013-200006.tar.gz
drwxr-xr-x. 5 root root 4096 Oct 13 21:16 .

We could see zip and tar.gz file, but they are all empty.
We also changed SELinux to permissive and tried running but still had the same problem.

Running with shell debug flag, got the following:

[root@elasticsearch-1 diagnostics-8.2.2]# ./diagnostics.sh --host localhost -u elastic -p --port 9200 --ssl --noVerify
+ scriptDir=./diagnostics.sh
+ scriptDir=.
+ libDir=./lib
+ '[' -d libDir ']'
+ '[' -x /bin/java ']'
+ JAVA=/bin/java
+ echo 'Using /bin/java as Java Runtime'
Using /bin/java as Java Runtime
+ '[' '!' -x /bin/java ']'
+ [[ '' != '' ]]
+ [[ '' == '' ]]
+ export 'DIAG_JAVA_OPTS=-Xms256m -Xmx2000m'
+ DIAG_JAVA_OPTS='-Xms256m -Xmx2000m'
+ echo 'Using -Xms256m -Xmx2000m for options.'
Using -Xms256m -Xmx2000m for options.
+ /bin/java -Xms256m -Xmx2000m -cp './config:./lib/*' co.elastic.support.diagnostics.DiagnosticApp --host localhost -u elastic -p --port 9200 --ssl --noVerify
2021-10-14 13:38:34,463 main ERROR Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...
Elasticsearch user password: ********************
Closing loggers.
Archiving diagnostic results.
Archive: -20211014-133840.zip was created
Couldn't create zip archive. Trying tar.gz
Archive: -20211014-133840.tar.gz was created
Couldn't create tar.gz archive.
Fatal error occurred: Couldn't create zip and tar.gz archives.. Check diagnostics.log in the archive file for more detail.

Tried running prior version 8.1.4, it reported issue with creating temp directory:

[root@elasticsearch-1 support-diagnostics-8.1.4]# ./diagnostics.sh --host localhost -u elastic -p --port 9200 --ssl --type api --noVerify
+ scriptDir=./diagnostics.sh
+ scriptDir=.
+ libDir=./lib
+ '[' -d libDir ']'
+ '[' -x /bin/java ']'
+ JAVA=/bin/java
+ echo 'Using /bin/java as Java Runtime'
Using /bin/java as Java Runtime
+ '[' '!' -x /bin/java ']'
+ [[ '' != '' ]]
+ [[ '' == '' ]]
+ export 'DIAG_JAVA_OPTS=-Xms256m -Xmx2000m'
+ DIAG_JAVA_OPTS='-Xms256m -Xmx2000m'
+ echo 'Using -Xms256m -Xmx2000m for options.'
Using -Xms256m -Xmx2000m for options.
+ /bin/java -Xms256m -Xmx2000m -cp './config:./lib/*' com.elastic.support.diagnostics.DiagnosticApp --host localhost -u elastic -p --port 9200 --ssl --type api --noVerify
2021-10-14 13:45:40,125 main ERROR Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...
Elasticsearch user password: ********************
Issue with creating temp directory. Check diagnostics.log in the archive file for more detail.
Closing loggers.
Archiving diagnostic results.
Couldn't create zip archive. Trying tar.gz
Couldn't create tar.gz archive.
Deleted directory: .

But creating output dir manually can be done successfully, even tried changing output dir permission to 777, still the same. We also tried changing umask to 011, still the same.

I was wondering what error code was returned when creating dir or creating zip. But we don't have any diagnostics.log to view.

Can I have guidance how to move forward to make the tool working?

[eeinworks8]

With the latest version of the diag tool, I'm seeing the exact same issue running on windows. I'm running as an admin with an admin command prompt and it will not successfully create the zip file. It's always empty. Tried running on 3 different machines with same issue. Only way I could get it to work was to hit Ctrl-C right before it tried to create the zip file. Then I just manually created the zip file.

It would be nice if the zip/tar file was NOT created successfully, it would leave the files in the directory rather than deleting them.

[CharafouddineAhmed]

Any solutions to this issue?