unable to restart fess

[wumpz]

I need some kind of starter. I am using the latest docker image with elasticsearch included. At the moment there is no configuration change except a web crawler.

Fess indexed successfully. Something about 60000 Docs, Index size 600MB.

Now I am not able to restart fess due to

ElasticSearch does not start anymore. Fess is restarting itself again and again to find a running elasticsearch.

I can't see any exception warning or stuff. Where should I look for hints?

[marevol]

Did you check the console log printed by docker?
I think elasticsearch failed to start.

[wumpz]

Do you mean this:

OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
fessserver_1 | Starting Elasticsearch Server:.
fessserver_1 | Checking Elasticsearch status #1
fessserver_1 | [2019-05-28T13:36:34,704][INFO ][o.e.e.NodeEnvironment ] [3642823c5c4e] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/sdb1)]], net usable_space [48.4gb], net total_space [98.3gb], types [ext4]
fessserver_1 | [2019-05-28T13:36:34,712][INFO ][o.e.e.NodeEnvironment ] [3642823c5c4e] heap size [1gb], compressed ordinary object pointers [true]
fessserver_1 | [2019-05-28T13:36:34,934][INFO ][o.e.n.Node ] [3642823c5c4e] node name [3642823c5c4e], node ID [sSFniH4pQa639J7Fr-Qyjw]
fessserver_1 | [2019-05-28T13:36:34,935][INFO ][o.e.n.Node ] [3642823c5c4e] version[7.0.1], pid[84], build[oss/deb/e4efcb5/2019-04-29T12:56:03.145736Z], OS[Linux/4.4.0-145-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.3/11.0.3+1-Debian-1bpo91]
fessserver_1 | [2019-05-28T13:36:34,936][INFO ][o.e.n.Node ] [3642823c5c4e] JVM home [/usr/lib/jvm/java-11-openjdk-amd64]
fessserver_1 | [2019-05-28T13:36:34,937][INFO ][o.e.n.Node ] [3642823c5c4e] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:InitiatingHeapOccupancyPercent=75, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=unpooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
fessserver_1 | Checking Elasticsearch status #2
fessserver_1 | Checking Elasticsearch status #3
fessserver_1 | [2019-05-28T13:36:36,744][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [aggs-matrix-stats]
fessserver_1 | [2019-05-28T13:36:36,746][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [analysis-common]
fessserver_1 | [2019-05-28T13:36:36,747][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [ingest-common]
fessserver_1 | [2019-05-28T13:36:36,748][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [ingest-geoip]
fessserver_1 | [2019-05-28T13:36:36,750][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [ingest-user-agent]
fessserver_1 | [2019-05-28T13:36:36,751][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [lang-expression]
fessserver_1 | [2019-05-28T13:36:36,752][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [lang-mustache]
fessserver_1 | [2019-05-28T13:36:36,753][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [lang-painless]
fessserver_1 | [2019-05-28T13:36:36,754][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [mapper-extras]
fessserver_1 | [2019-05-28T13:36:36,755][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [parent-join]
fessserver_1 | [2019-05-28T13:36:36,756][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [percolator]
fessserver_1 | [2019-05-28T13:36:36,757][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [rank-eval]
fessserver_1 | [2019-05-28T13:36:36,757][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [reindex]
fessserver_1 | [2019-05-28T13:36:36,758][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [repository-url]
fessserver_1 | [2019-05-28T13:36:36,758][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [transport-netty4]
fessserver_1 | [2019-05-28T13:36:36,760][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [analysis-extension]
fessserver_1 | [2019-05-28T13:36:36,760][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [analysis-fess]
fessserver_1 | [2019-05-28T13:36:36,761][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [configsync]
fessserver_1 | [2019-05-28T13:36:36,761][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [dataformat]
fessserver_1 | [2019-05-28T13:36:36,762][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [minhash]
fessserver_1 | Checking Elasticsearch status #4
fessserver_1 | Checking Elasticsearch status #5
fessserver_1 | Checking Elasticsearch status #6
fessserver_1 | Checking Elasticsearch status #7
fessserver_1 | Checking Elasticsearch status #8
fessserver_1 | Checking Elasticsearch status #9
fessserver_1 | Checking Elasticsearch status #10
fessserver_1 | Checking Elasticsearch status #11
fessserver_1 | Checking Elasticsearch status #12
fessserver_1 | Checking Elasticsearch status #13
fessserver_1 | Checking Elasticsearch status #14
fessserver_1 | Checking Elasticsearch status #15
fessserver_1 | Checking Elasticsearch status #16
fessserver_1 | Checking Elasticsearch status #17
fessserver_1 | Checking Elasticsearch status #18

[wumpz]

And after checking for elasticsearch til #60, it restarts fess again.

[marevol]

Does your server have enough memory?
OOM Killer might kill the process.

[wumpz]

I started the docker container without any memory limitations. Therefore the standard configuration should be in place. By the way, this server runs with 20GB ram.

However, shouldn't there be some kind of error message if the process is killed?

[marevol]

What command did you run?

[wumpz]

It is part of a docker compose file. I did not apply a command here. Therefore the one in your image applies. Right?

[wumpz]

This is the part of my docker-compose file, that starts fess.

fessserver:
image: codelibs/fess:latest
restart: always
volumes:
- fess_config:/opt/fess
- elasticsearch_config:/etc/elasticsearch
- elasticsearch_data:/var/lib/elasticsearch
ports:
- "8083:8080"

[marevol]

• elasticsearch_config:/etc/elasticsearch

It's better to check configuration files in the above.

[wumpz]

I attached the three files within this directory. I skipped the keystore file.

fess_es_config.zip

Since I did not change anything, except over FESS web gui, I assume this is the standard configuration.

[wumpz]

So where can I look next for some problems?

[marevol]

Did you check /var/log/messages or syslog in the host?

[wumpz]

Found a hs_err - file.

hs_err_pid84.zip

Could this something Java 11 problematic using elasticsearch be:

Event: 17.265 Thread 0x00007f2364013800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x00000000ff7134b0}: Found class java.lang.Object, but interface was expected> (0x00000000ff7134b0) thrown at [/build/openjdk-11-1uPVZa/openjdk-11-11.0.3+1/src/hotspot/share/interpreter/linkResolver.cp
Event: 17.346 Thread 0x00007f2364013800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x00000000ff443d60}: Found class java.lang.Object, but interface was expected> (0x00000000ff443d60) thrown at [/build/openjdk-11-1uPVZa/openjdk-11-11.0.3+1/src/hotspot/share/interpreter/linkResolver.cp
Event: 17.366 Thread 0x00007f2364013800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x00000000ff4f8fb8}: Found class java.lang.Object, but interface was expected> (0x00000000ff4f8fb8) thrown at [/build/openjdk-11-1uPVZa/openjdk-11-11.0.3+1/src/hotspot/share/interpreter/linkResolver.cp
Event: 17.578 Thread 0x00007f2364013800 Exception <a 'java/lang/NoSuchMethodError'{0x00000000fee8bff0}: java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V> (0x00000000fee8bff0) thrown at [/build/openjdk-11-1uPVZa/op

[wumpz]

Reverted back to version 12.6.1, rebuilt the complete index.

Now fess restarts without any problems.

Curiously at the same directory fess finds nearly 5000 documents less.

[wumpz]

So the older version runs without any problems. I stick with it. Thx for your help.