add command line option to execute the optimize task standalone

[bernieke]

We change the basePath setting before packaging, and want to avoid having the optimize step run on each node that package is deployed to. (Also, we don't want to give the user running kibana write permissions to the kibana sources.)

I've now written a small script to run kibana, and scrape the output to detect the end of the optimize step, upon which it kills the process.

Better would be if this could be done directly through a command line option.

[rashidkpc]

Seems reasonable, though we don't really need it immediately. If you want to submit a pull we would happily accept it, otherwise it'll probably be awhile.

[serbaut]

I'm using this as a workaround in my Dockerfile:

# give it two minues to optimize bundles
RUN timeout -t 120 node /kibana/src/cli || true

[smousa]

Here is the hack I used in my Dockerfile (kibana-4.5.2):

RUN {PATH TO KIBANA BIN} 2>&1 | grep -m 1 "Optimization of .* complete in .* seconds" 

That way you don't have to rely on an arbitrary sleep value

[Hronom]

Hack from @smousa helps me, but I'm still waiting for an official command that I can execute while I'm building docker image.
Something like:

./kibana/bin/kibana optimize

[alainchiasson]

This would also help in faster deploys on Kubernetes.

[EDIT: The above hack works it would still be good to have the command line option.]

[xycloud]

mark, also need this feature

[CpuID]

+1 to getting this added, would be much better to be able to compile AOT rather than JIT on ephemeral container based platforms

[rocketraman]

As I commented on #10724 , this can also be caused by setting environment variable XPACK_SECURITY_ENABLED=false with docker container 5.4.3.

docker run -e XPACK_SECURITY_ENABLED=false docker.elastic.co/kibana/kibana:5.4.3
{"type":"log","@timestamp":"2017-06-28T20:34:34Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for graph, monitoring, ml, kibana, timelion and status_page. This may take a few minutes"} 

[Tenzer]

My workaround for making a Kibana Docker image with the bundles pre-optimised is to just install a plugin of any kind during the image build stage, as that causes the bundles to be built.

[ejsmith]

Seems as though the optimize doesn't happen now if it can't connect to elasticsearch. So starting the kibana command doesn't trigger optimize. Really sucks waiting so damn long for this image to come up every single time.

[shaharmor]

Is there any plan to implement this?

[ejsmith]

Makes for some pretty crappy docker images when you have to wait like 5 minutes for it to respond.

[ElijahGlover]

I'd like this feature, any ideas for release timing?

[gentunian]

it should have a way to build the image optimized. It makes no sense to optimize on launch, even more in our containers world...

The optimization step is making impossible to deploy in servers where you have the exact hardware for runtime. Now you need the requirements for runtime plus the requirements for optimizing (let's call it rebuilding).

No sense at all...

[trondhindenes]

WIth a little bit of cpu capping, our Kibana pods in Kubernetes takes close to 30 minutes to start. We desperately need a robust way of pre-compiling Kibana, hopefully without requiring an active Elasticsearch cluster while compiling (we're just setting some x-pack flags)

[FireBurn]

I've been patching Kibana myself, I can supply the backported patch if you'd like

…

On Fri, 2 Feb 2018 at 10:47 Trond Hindenes ***@***.***> wrote: WIth a little bit of cpu capping, our Kibana pods in Kubernetes takes close to 30 minutes to start. We desperately need a robust way of pre-compiling Kibana, hopefully without requiring an active Elasticsearch cluster while compiling (we're just setting some x-pack flags) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6057 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFR810lD9nJP8Pariv55QDAAYMKalM1ks5tQuepgaJpZM4HQViq> .

[gentunian]

@FireBurn that would be nice. The bundling and optimizing thing takes about 8 minutes in a 16 core environment after installing a plugin. We should have a way to do this just once.

[anitaIncaendo]

seriously , me too checking, how can i run my kibana plugin code, without caching and optimizing code in dev environment.

I am trying to find the way, no need to start kibana server again and again after my change in dev environment

[vinuales]

True... fixing a deprecated plugin is a mess... An environment parameter could be useful.

[tsullivan]

I'm not sure how long the hacks will keep working, but I found something that works for me. I wanted a custom image of Kibana with a few plugins installed, and a few x-pack settings for disabling some features (causes a reoptimization). I ended up waiting for optimization to finish in the container, then did a docker commit to create a new image for optimized Kibana.

See https://discuss.elastic.co/t/how-to-do-custom-docker-image-that-has-certain-features-disabled/132007/6?u=tsullivan

[d8888]

FROM docker.elastic.co/kibana/kibana:6.2.4

# ...
ENV XPACK_SECURITY_ENABLED false
# ...

# modified from: https://github.com/elastic/kibana-docker/issues/27
# Jarpy's answer
# A "hack" to force kibana to optimize

RUN /usr/local/bin/kibana-docker 2>&1 | grep -m 1 "Optimization of .* complete" # [1]

This particular snippet of Dockerfile allows me to trigger optimization when building my customized Kibana image with Search Guard plugin (not shown) installed. In production, the docker image loads quickly without having to wait for "optimization" during start-up.

Using "/usr/local/bin/kibana-docker" instead of "kibana" as seen in Jarpy's original answer, allows some official helper scripts to translate environment variables to appropriate Kibana settings, such as capturing "XPACK_SECURITY_ENABLED false" to "xpack.security.enabled=false" in this particular example.

[bdashrad]

unfortunately i'm running into this on 6.4.0, even when using the trick above to optimize before uploading our customized image.

Dockerfile:

ARG ELASTICSEARCH_VERSION=6.4.0
FROM docker.elastic.co/kibana/kibana:${ELASTICSEARCH_VERSION}

ARG ELASTICSEARCH_VERSION
ENV SEARCHGUARD_VERSION=${ELASTICSEARCH_VERSION}-14
ENV XPACK_GRAPH_ENABLED false
ENV XPACK_ML_ENABLED false
ENV XPACK_MONITORING_ENABLED true
ENV XPACK_MONITORING_UI_ENABLED true
ENV XPACK_REPORTING_ENABLED false
ENV XPACK_SECURITY_ENABLED false
ENV XPACK_WATCHER_ENABLED false

RUN kibana-plugin install http://repo1.maven.org/maven2/com/floragunn/search-guard-kibana-plugin/${SEARCHGUARD_VERSION}/search-guard-kibana-plugin-${SEARCHGUARD_VERSION}.zip

# optimize kibana before building final image
# https://github.com/elastic/kibana/issues/6057
RUN /usr/local/bin/kibana-docker 2>&1 | grep -m 1 "Optimization of .* complete"

EXPOSE 5601

CMD ["kibana"]

during build:

Step 13/15 : RUN /usr/local/bin/kibana-docker 2>&1 | grep -m 1 "Optimization of .* complete"
 ---> Running in 2abeb42b3085
{"type":"log","@timestamp":"2018-09-19T02:24:55Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for stateSessionStorageRedirect, status_page, timelion, monitoring, dashboardViewer, apm, searchguard-login, searchguard-customerror, searchguard-multitenancy, searchguard-configuration and kibana complete in 245.81 seconds"}

on start:

$ kubectl logs -f kibana-665c769944-sp5p6 kibana
{"type":"log","@timestamp":"2018-09-19T11:23:33Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for stateSessionStorageRedirect, status_page, timelion, monitoring, dashboardViewer, apm, searchguard-login, searchguard-customerror, searchguard-multitenancy, searchguard-configuration and kibana. This may take a few minutes"}

[Tenzer]

@bdashrad If I try the Dockerfile you have posted and just remove the RUN command which is mentioned as a workaround, the resulting Docker image won't have to optimise anything when it starts up. This is because the installation of a plugin causes the optimisation to take place, so you shouldn't need to have to do anything else after that.

[bdashrad]

@Tenzer i tried a few ways, including the docker file i put above and your suggestion, and still seem to get the optimizing message. I guess we'll skip using the xpack monitoring plugin for now and go back to datadog.

[Battleroid]

@d8888's fix works for us currently. We build an image with our preferred modules disabled, works like a charm, startup is instant like it should be.

[horacimacias]

I'm having similar issues. In my case I believe they are aggravated by the fact that I'm using environment properties on my kibana.yml file, e.g.

elasticsearch.username: ${KIBANA_ES_USERNAME}
elasticsearch.password: ${KIBANA_ES_PASSWORD}

Even if I install a plugin during docker build, which takes forever Optimizing and caching browser bundles when I run my docker image I still get Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, login, logout, dashboardViewer, apm and kibana. This may take a few minutes

Would using a different environment variable value force the optimization to be ran again? is there any way to not have this behaviour?
I'd imagine using environment variables with changing values is a common scenario for users running kibana as docker images.

[horacimacias]

also, would there be a way to keep the result of optimization so that at least optimization is only done once, and not every single time the docker image is restarted?
perhaps using a volume to store the result of optimization?

still not ideal, but at least it's waiting for 10min once, not every single time.

[Pitasi]

I'm using 6.5.1 and having the same behaviour as bdashrad here: #6057 (comment). In the dockerfile I install search guard, run kibana and grep for "complete". When I start up the container it re-runs the optimize step, not sure if it's triggered by different environment variables and how to avoid that.

[dragonsmith]

Hello!
I'm experiencing the same behavior as @Pitasi .

I have a custom image with Logtrail plugin installed. I've tried to add /usr/local/bin/kibana-docker 2>&1 | grep -m 1 "Optimization of .* complete" but to no good. It just does not start any optimizations and it hangs forever. I've also discovered an --optimize option in kibana_docker. It doesn't help either.

When I start Kibana on production from my image it runs optimizations immediately:

{"type":"log","@timestamp":"2018-12-10T14:44:04Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for logtrail, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra and kibana. This may take a few minutes"}

[tylersmalley]

We added a command line option to run the optimize in #16302

A lot of folks here mentioned issues with the duration of the optimize task - this has gotten better over the past couple of years and we hope to soon get rid of it entirely. If there are follow-up issues, please create a new issue or visit us over on the Discuss forums.

[elpaisa]

For those who need a pre-cache in docker build stage, this worked for me: RUN bin/kibana | grep -q -m 1 "Status changed"

[tylersmalley]

@bernieke just pass --optimize to bin/kibana per the PR in my last comment.

[elpaisa]

Does that work for version 6.2.3 OSS?

[orinciog]

@elpaisa No, this PR is not included in 6.2.3 OSS. You have to merge it yourself in the distribution. Attention: docker files are different from those in github. If you want to patch it and you are using docker version, see https://discuss.elastic.co/t/difference-between-kibana-github-and-kibana-docker-oss/200001.

Kibana triggers the optimization process if at least one loaded file changed from the last optimization process. This applies also to configuration files.

[tylersmalley]

@orinciog, that is correct for all builds. The Javascript files in the builds are transpiled using Babel. If you want to make modifications to the underlying source code you will need to run a build. You can find more information on that with yarn build --help.