Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kibana plugin install stalls at "Optimizing and caching browser bundles..." #19678

Open
bradvido opened this Issue Jun 5, 2018 · 22 comments

Comments

Projects
None yet
@bradvido
Copy link

commented Jun 5, 2018

Kibana version:
6.2.4
Elasticsearch version:
6.2.4
Server OS version:
Windows Server 2016

Original install method (e.g. download page, yum, from source, etc.):
zip download from elastic.co

Describe the bug:
I executed the install command here about 16 hours ago, and it still hasn't gotten past this step. As you can see node is using ~8 GB of memory which seems excessive.
image
Note that per recommendations in other issues, I did set the --max-old-space-size to 8000
image

Steps to reproduce:

  1. Install ES/Kibana on Server 2016
  2. Execute command seen in screenshot (happens for any plugin install, not just the one pictured)
  3. Wait

Expected behavior:
Plugin install should finish quickly

@tylersmalley

This comment has been minimized.

@tylersmalley

This comment has been minimized.

Copy link
Member

commented Jun 5, 2018

Related: #19633

@bradvido

This comment has been minimized.

Copy link
Author

commented Jun 23, 2018

Still an issue for me in 6.3.0

@akhilvatts

This comment has been minimized.

Copy link

commented Sep 11, 2018

+1 in 6.4.0

@derentis

This comment has been minimized.

Copy link

commented Oct 4, 2018

+1 in 6.3.1 on Ubuntu 18.0.4.1 LTS

just sits there with processes open but never actually goes anywhere. That's been running for 24hrs now and it isn't doing anything.
image
image
image
image

@zukeru

This comment has been minimized.

Copy link

commented Oct 10, 2018

+1 here I have the same issue with sentinl plugin. It will get caught in the optimizing part. If I kill it and then run it again it says its installed and configured. This doesn't happen with searchguard however.

@Mrfly-ca

This comment has been minimized.

Copy link

commented Oct 17, 2018

I am having the same issue running the install for wazuhapp-3.6.1_6.4.2.zip

Everything else is the same. (Server, OS, mem etc)

@fl4nker

This comment has been minimized.

Copy link

commented Oct 19, 2018

+1 , same issue occured on ELK Stack6.4.1,Ubuntu16.04 LTS

@akhilvatts

This comment has been minimized.

Copy link

commented Nov 16, 2018

+1 ELK 6.5.0

@memelet

This comment has been minimized.

Copy link

commented Dec 8, 2018

I get a hang with wazuh 3.7.1 on kibana 6.5.1

@ppoetz

This comment has been minimized.

Copy link

commented Jan 8, 2019

Same here with 6.5.1 and the logtrail plugin of the same version.

@q2dg

This comment has been minimized.

Copy link

commented Jan 14, 2019

What a mess...The same with Kibana 6.5.4 trying to install Sentinl plugin...

@anottrott

This comment has been minimized.

Copy link

commented Jan 14, 2019

Same issue with Logtrail 6.5.1 on Ubuntu 18.04, CPU and memory are maxed out. It seems like this is a common issue with the kibana build when installing plugins

After some time the installation crashed with the following error message... Seems like NodeJS running out of memory?

Attempting to transfer from https://github.com/sivasamyk/logtrail/releases/download/v0.1.30/logtrail-6.5.2-0.1.30.zip
Transferring 2096312 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

Plugin installation was unsuccessful due to error "Command failed: /elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node /elasticsearch/kibana-6.5.2-linux-x86_64/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
(node:658) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
2: 0x8cce9c [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
6: v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString, v8::internal::PretenureFlag) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
7: v8::internal::String::Flatten(v8::internal::Handlev8::internal::String, v8::internal::PretenureFlag) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
10: 0x235c5098697d

{"type":"log","@timestamp":"2019-01-14T18:30:37Z","tags":["info","optimize"],"pid":658,"message":"Optimizing and caching bundles for ml, logtrail, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra and kibana. This may take a few minutes"}

<--- Last few GCs --->

[658:0x38a9170] 995416 ms: Mark-sweep 1310.6 (1457.1) -> 1310.5 (1458.6) MB, 1777.2 / 0.0 ms allocation failure GC in old space requested
[658:0x38a9170] 997310 ms: Mark-sweep 1310.5 (1458.6) -> 1310.5 (1427.6) MB, 1893.5 / 0.0 ms last resort GC in old space requested
[658:0x38a9170] 999434 ms: Mark-sweep 1310.5 (1427.6) -> 1310.5 (1427.6) MB, 2124.7 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1c668eca58b9
0: builtin exit frame: lastIndexOf(this=0x3deac35b02c9 <Very long string[1106657]>,0x289b9bd120e9 <String[1]: \n>)

1: has_nlb(aka has_nlb) [0x13dff58022d1 <undefined>:5970] [bytecode=0x37933e0febd9 offset=15](this=0x13dff58022d1 <undefined>)
2: /* anonymous */(aka /* anonymous */) [0x13dff58022d1 <undefined>:6070] [bytecode=0x132e668035b1 offset=60](this=0x13dff58022d1 <undefined>,...

"

@CannibalKush

This comment has been minimized.

Copy link

commented Jan 15, 2019

Exact same problem with kibana 6.4.2 and searchguard 6.4.2-17, any fix on the horizon?

@minhle2994

This comment has been minimized.

Copy link

commented Jan 15, 2019

+1 in Kibana 6.5.4

@fabrei

This comment has been minimized.

Copy link

commented Jan 15, 2019

same with 6.5.1 and the plugin network_vis 5.5.0-1
there are already some issues #25849 and #21794 which are dealing with this topic. #21809 should resolve this, but it seems it doesn't.

my workaround is:

  1. stop kibana (maybe elastic too)
  2. install the plugin until Optimizing and caching browser bundles... is shown
  3. stop installation
  4. run rm -rf /usr/share/kibana/optimize/bundles
  5. in /usr/share/kibana/bin/kibana add --max-old-space-size=3072 to $NODE_OPTIONS
  6. start kibana normally
  7. kibana starts the optimization and after 3 to 5 minutes kibana is up..

seems that the optimization at startup is quite different to the optimization after a plugin installation!?

@mistic

This comment has been minimized.

Copy link
Member

commented Jan 15, 2019

We have made several improvements on this matter that will come along by 6.6.x and 6.7.x.
Meanwhile I believe the best workaround to deal with this situation would be:

  1. Stop kibana.
  2. Set the current directory to the Kibana installation dir. (For example cd /usr/share/kibana)
  3. rm -rf optimize/bundles
  4. NODE_OPTIONS="--max-old-space-size=4096" ./bin/kibana
  5. Kibana should optimize and start normally in some minutes
@minhle2994

This comment has been minimized.

Copy link

commented Jan 16, 2019

Sometime installing process hang at Optimizing and caching browser bundles but plugin still install successfully

@mistic

This comment has been minimized.

Copy link
Member

commented Jan 16, 2019

@minhle2994 can u try what I suggest on my previous comment please?

floragunncom added a commit to floragunncom/search-guard-docs that referenced this issue Mar 20, 2019

@Beginner77

This comment has been minimized.

Copy link

commented Apr 16, 2019

+1 in Kibana 6.7.1. It just stalls at the optimizing and caching browser bundle points. when checked inside the container, this command - " /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false" was running for a very long time.Killed the process and tried to run this manually and it just gets stuck there do nothing before I eventually kill it.

@g3rhard

This comment has been minimized.

Copy link

commented Apr 16, 2019

+1 same thing in Kibana 6.7.1
Debian 9

NODE_OPTIONS="--max-old-space-size=8192" /usr/share/kibana/bin/kibana-plugin install https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.3/elastalert-kibana-plugin-1.0.3-6.7.1.zip
Attempting to transfer from https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.3/elastalert-kibana-plugin-1.0.3-6.7.1.zip
Transferring 22930973 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

In kibana.yml:

...
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false
@mistic

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

@Beginner77 @g3rhard that is strange, can you try to run the steps I mention above on #19678 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.