Bug on wazuh-indexer startup while installing Wazuh 4.3 on Ubuntu 20

[jimmynarula]

Wazuh version	Component	Install type	Install method	Platform
4.3	Wazuh Indexer	Manage	assistant & step-by-step	Ubuntu 20

I am continuously facing a bug while installing the Wazuh 4.3 release. I have tried both Wazuh assistant & step by step installation methods. In both cases, the Wazuh-Indexer service was unable to get started. Later, when I checked the log I found the following error in the logs:

May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:47)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.systemd.Libsystemd.<clinit>(Libsystemd.java:46)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:137)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:148)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.node.Node.start(Node.java:1153)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:421)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.cli.Command.main(Command.java:101)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
May 06 09:47:28 ip-172-31-17-8 systemd-entrypoint[9774]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)

After a bit of research, I found a temporary fix by updating the following value in the file /etc/wazuh-indexer/jvm.options

-Djava.io.tmpdir=/var/log/wazuh-indexer

[rauldpm]

Hello @jimmynarula

I have migrated the issue to the wazuh-packages repository, since it is in this repository where the Wazuh indexer package is created and configured, the url of the issue you opened in the wazuh repository will automatically redirect to this issue.

Regarding the error, thanks for the contribution of the fix, we will investigate it and consider if we modify the default path of the java temporary directory. About the cause, can you tell me something about your environment? Do you have Elasticsearch or Cassandra installed on the system? Was it a clean environment?

From what I have been able to investigate, the error Could not initialize class com.sun.jna.Native may be due to the fact that the temporary directory used by the java embedded package does not have execution permissions (openjdk 15.0.1 2020-10-20).

I will continue to investigate this issue while I wait for your response.

Regards, Raúl.

[jimmynarula]

Hello @rauldpm

Thanks for looking into this bug, I would like to confirm that I was using a clean install of Wazuh 4.3, there was no Elasticsearch or Cassandra was already installed in the server because it was a fresh new EC2.

[rauldpm]

Hello @jimmynarula

Could you tell me which ami you are using (ami id) to see if I can reproduce the problem on the same system? Also, i could you run this command and show me the full output?

• grep -i -R -E "Djava.io.tmpdir" /var/log/wazuh-indexer/ > tmpdir.log

An example of output within the file would be the following:

/var/log/wazuh-indexer/wazuh-cluster.log:[2022-05-09T12:17:10,763][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms8000m, -Xmx8000m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8603777270765886168, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=4194304000, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]

By default, the temporary directory that java assigns during execution is /tmp/ as you can see in the output that I have shared: -Djava.io.tmpdir=/tmp/opensearch-8603777270765886168, therefore, I would need to know what temporary directory it was trying to assign before applying your fix and what permissions that directory has, in the case of /tmp, it could be observed as follows

# ls -ld /tmp/
drwxrwxrwt 15 root root 4096 May  9 12:19 /tmp/

Once we know the directory where it was trying to mount the directory, we need to check if it has the noexec flag set, this can be checked with the command

• findmnt -l | grep noexec

I wait your answer.
Regards, Raúl.

[jimmynarula]

Hello @rauldpm

We are using the following API for our WAZUH server:
https://aws.amazon.com/marketplace/pp/prodview-acrp2dhekgpd4?sr=0-6&ref_=beagle&applicationId=AWSMPContessa#pdp-pricing

I tried to find the log you requested but it fetched no results back. Do you want me to look anywhere else?

I also check the permissions on /tmp/ folder and it's same as you have mentioned in the last message and after that I checked for noexec flag set, and I found that /tmp/ folder has been set as one.

/tmp tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,inode64

Regards,
Jaswinder

[rauldpm]

Hello @jimmynarula

Using the AMI that you have indicated, I have been able to reproduce your problem, effectively, the partition mounted in /tmp has the noexec flag, which prevents a correct installation since certain files are downloaded in that directory.

Removing the noexec flag from fstab, leaving it as follows:

LABEL=cloudimg-rootfs / ext4 defaults,discard 0 1
tmpfs /tmp tmpfs defaults,rw,nosuid,nodev,relatime 0 0
tmpfs /dev/shm tmpfs defaults,nodev,nosuid,noexec 0 0

The installation carried out with the installer assistant has been satisfactory and functional. It is true that for security it is possible that the /tmp directory has this flag to avoid unwanted execution, having said that, I have opened this issue: #1539, to carry out a check in that directory and change it to avoid this problem without affecting this security measure that the user may have done if necessary.

That said, the solution to your problem would be to remove the noexec flag from the /tmp directory (by editing the fstab file) or by changing the directory that java uses as you indicated at the beginning.

Regards, Raúl.

[jimmynarula]

Hello @rauldpm

Thanks for your help.

Regards,
Jaswinder

[alberpilot]

I proceed to close this issue, please don't hesitate to re-open it.