Kibana paths configuration

[ghost]

Hello @mistic some questions about directories modified/added to at runtime by kibana. Please confirm, that based on [1]

• config and plugings are considered user-serviceable, meaning user can modify/add/remove content, correct?
• Is there a setting that I can change to point Kibana to read config file from somewhere else?
• data directory is written to by Kibana at runtime as well
• [2] lists path.data as a setting that can be modified to move data directory somewhere, yet this setting is not in the reference kibana.yml file.
• Is optimize also modified by Kibana at run-time?
• If above it true, is the a setting to move optimize directory somewhere else?

[1] https://www.elastic.co/guide/en/kibana/current/windows.html
[2] https://www.elastic.co/guide/en/kibana/current/settings.html

[mistic]

config and plugings are considered user-serviceable, meaning user can modify/add/remove content, correct?

Correct

Is there a setting that I can change to point Kibana to read config file from somewhere else?

--config

data directory is written to by Kibana at runtime as well

data folder could have its content changed during the runtime

[2] lists path.data as a setting that can be modified to move data directory somewhere, yet this setting is not in the reference kibana.yml file.

I think we don't list all the settings in the kibana.yml file but we could add this one if you think it is important

Is optimize also modified by Kibana at run-time?

optimize is now located inside data and it can change at runtime

If above it true, is the a setting to move optimize directory somewhere else?

you can just move the data folder

@ygel do you think that answer your questions?

[ghost]

Hello @mistic, I found some issues during testing that make MSI life harder than it needs to be:

kibana.bat

• would be really useful to be able to specify data directory path from command line, like config and plugin. Now I have to drop an extra .yml file just to set path.data and specify both config files when running.

kibana-keystore.bat create

• unable to specify neither path to config file, nor data directory from command line. Always tries to create keystore in data dir relative to package.json.

kibana-plugin.bat

• lists --plugin-dir as DEPRECATED, while main kibana.bat does not. Which should I believe?

[mistic]

@tylersmalley can you provide some help here? I think we might be able to work with @ygel in some of those items but I'm not sure about --plugin-dir .

@ygel are those blockers to you?

[ghost]

On Windows MSI installs "mutable" (user-serviceable) files to a separate location outside of main install path. Hence need ability to customize all paths that app and user can write to.

• I have a workaround for not being able to specify data dir, it's not pretty though.
• Not sure if plugin dir will break on me
• keystore will not work

[tylersmalley]

@ygel, those settings can also be specified as command line arguments. For example: bin\kibana.bat --path.data=C:\kibana-data

[mistic]

@ygel while talking with @tylersmalley he also mentioned that for the kibana-keystore we have this PR opened elastic/kibana#57856 that should solve your problem (however it looks like it is only targeting 8.0.0). @jbudz are we good to go with this?

So I think the only thing still open is the plugin dir for the kibana-plugin cli. @ygel what is your target release version for the MSI Installer?

[ghost]

From the looks of it, elastic/kibana#57856 bases the key-store off of the config file. Want to make sure that when I move the config file with --config, it affects key-store path as well.

This however doesn't solve the problem of src/cli_keystore not understanding the --config arg. Am I missing something?

Maybe have --path.data and --config universally understood by all CLI tools?

[ghost]

So I think the only thing still open is the plugin dir for the kibana-plugin cli. @ygel what is your target release version for the MSI Installer?

Original target was 7.8.0.

@mistic @tylersmalley At this point I believe it is too dangerous to rush Kibana MSI into 7.8.0. I am aware of these UX issues (below) that in my opinion warrant a delay to the next release. Since we don't generally add new artifacts in minor releases, it will be 7.9 for Kibana MSI.

• as noted in the comment above, Kibana CLI tools should universally understand --config, --path.data, --plugins or a combination thereof that would allow running kibana with user-serviceable directories moved to a user-writable location. (ref: Move kibana-keystore from data/ to config/ kibana#57856)
• A problem was reported by support about inability to print when running Kibana as a service (Unable to print visualizations when running as a service #65)

cc: @mgreau

[jbudz]

@ygel agreed. I would hold off on the msi. These both have a large surface area at the fs level and if we can keep them separate for a stability period we'll have a better idea of new issues vs existing issues.

Some of the issues we're seeing above^ (not kibana/static-fs related) are lower hanging fruit on the kibana side so it'll give us time to fix them at the source vs workarounds.

[tylersmalley]

@ygel, I also agree we shouldn't push this for 7.8.0. I would also like to let the static-fs sit for at least a minor version to ensure it's stable.

[mistic]

I've opened an issue within respect for the problem with kibana-keystore elastic/kibana#66064

@tylersmalley after talking with @ygel I think I got it why he needs kibana-keystore to both support --config and --path.data. I've tried to explain it in the issue I've linked above.

For the issue you were facing with kibana.bat it self I think we are over it now with --path.data as @tylersmalley suggested right @ygel?

Regarding the last issue with kibana-plugin.bat the discussion was longer. From what I've understood from it @joshdover thinks it might be possible to do it and in that case we would be able to replace the warning when using --plugin-dir with the kibana-plugin cli but we'll need to hard testing this. @tylersmalley am I missing something?

[ghost]

For the issue you were facing with kibana.bat it self I think we are over it now with --path.data

Running Kibana itself looks ok with Tyler's suggestion --path.data

[joshdover]

Regarding the last issue with kibana-plugin.bat the discussion was longer. From what I've understood from it @joshdover thinks it might be possible to do it and in that case we would be able to replace the warning when using --plugin-dir with the kibana-plugin cli but we'll need to hard testing this

We did originally intend to deprecate the --plugin-dir option, but at this point I'm not sure that's feasible. There are just too many scenarios where this is useful (we use it in our own functional tests for example).

We will probably want to retain that warning message though, regarding relative paths. Until we have either a bundling solution for server-side plugin code or an babel plugin that allows importing from the root of the repo (elastic/kibana#52995), this warning is still relevant.

[mistic]

@joshdover that means, even though we can keep using the flag and considering the fact we would want to solve this for the future instead of deprecating it, if we use it now we could have plugins that won't work. Is that correct?

[joshdover]

if we use it now we could have plugins that won't work. Is that correct?

Right so the problem is that the plugin will need to be developed in the same relative location to where it is on the filesystem when deployed because of relative imports. In that way, plugins can't really be easily moved around on the filesystem without changing their source code.

Maybe this should be a dev-only warning message?

[ghost]

For reference, taking a page from Beats book, we could define --path.base=foo for example and derive config, data, plugins, etc as ${path.base}/config, ${path.base}/data, ${path.base}/plugins.

[mistic]

@joshdover I think the warning is still useful. Maybe we could just no call it a deprecation warning 😄

I think our current point of situation here is: we will have in some close future the ability to have plugins being installed in an arbitrary directory, we cannot just doing it now for the things mentioned above. As we cannot do it now I think having an MSI installer with the ability to install custom plugins will not be possible because C:/Program Files is not writable after the installation. @tylersmalley @ygel @joshdover do you think we can agree into point to a first version without supporting custom plugin installations while we move on in the general effort of having that ability by 8.0?

[joshdover]

do you think we can agree into point to a first version without supporting custom plugin installations while we move on in the general effort of having that ability by 8.0?

Absolutely, this was going to be my suggestion. Plugins are generally "not officially supported" so we can iterate on this and add support there later. Given the limitations of Windows, I don't think we can support them outside the main install location until we solve one of the two problems I listed above. It makes sense to decouple that from adding the MSI installer.