Merge branch 'master' into changing-all-calls-to-ml-endpoints-to-use-…

…internal-user

.gitignore

@@ -31,6 +31,7 @@ disabledPlugins
 webpackstats.json
 /config/*
 !/config/kibana.yml
+!/config/node.options
 coverage
 selenium
 .babel_register_cache.json

config/node.options

@@ -0,0 +1,6 @@
+## Node command line options
+## See `node --help` and `node --v8-options` for available options
+## Please note you should specify one option per line
+
+## max size of old space in megabytes
+#--max-old-space-size=4096

docs/developer/advanced/development-basepath.asciidoc

@@ -0,0 +1,18 @@
+[[development-basepath]]
+=== Considerations for basepath
+
+In dev mode, {kib} by default runs behind a proxy which adds a random path component to its URL.
+
+You can set this explicitly using `server.basePath` in <<settings>>. 
+
+Use the server.rewriteBasePath setting to tell {kib} if it should remove the basePath from requests it receives, and to prevent a deprecation warning at startup. This setting cannot end in a slash (/).
+
+If you want to turn off the basepath when in development mode, start {kib} with the `--no-basepath` flag
+
+[source,bash]
+----
+yarn start --no-basepath
+----
+
+
+

docs/developer/core/development-es-snapshots.asciidoc → docs/developer/advanced/development-es-snapshots.asciidoc

@@ -1,21 +1,21 @@
 [[development-es-snapshots]]
 === Daily Elasticsearch Snapshots
 
-For local development and CI, Kibana, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).
+For local development and CI, {kib}, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).
 
 A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of Elasticsearch, and a JSON-based manifest file containing metadata about the distributions.
 
 https://ci.kibana.dev/es-snapshots[A dashboard] is available that shows the current status and compatibility of the latest Elasticsearch snapshots.
 
 ==== Process Overview
 
-1. Elasticsearch snapshots are built for each current tracked branch of Kibana.
+1. Elasticsearch snapshots are built for each current tracked branch of {kib}.
 2. Each snapshot is uploaded to a public Google Cloud Storage bucket, `kibana-ci-es-snapshots-daily`.
 ** At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first.
-3. Each snapshot is tested with the latest commit of the corresponding Kibana branch, using the full CI suite.
+3. Each snapshot is tested with the latest commit of the corresponding {kib} branch, using the full CI suite.
 4. After CI
 ** If the snapshot passes, it is promoted and automatically used in CI and local development.
-** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and Kibana.
+** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and {kib}.
 
 ==== Using the latest snapshot
 
@@ -39,7 +39,7 @@ KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server
 
 Currently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change.
 
-1. Edit `Jenkinsfile` in the root of the Kibana repo
+1. Edit `Jenkinsfile` in the root of the {kib} repo
 2. Add `env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'` at the top of the file.
 3. Commit the change
 
@@ -75,13 +75,13 @@ The file structure for this bucket looks like this:
 
 ==== How snapshots are built, tested, and promoted
 
-Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of Kibana.
+Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of {kib}.
 
 ===== Build
 
 https://kibana-ci.elastic.co/job/elasticsearch+snapshots+build/[This Jenkins job] builds the Elasticsearch snapshots and uploads them to GCS.
 
-The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the kibana repo].
+The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the {kib} repo].
 
 1. Checkout Elasticsearch repo for the given branch/version.
 2. Run `./gradlew -p distribution/archives assemble --parallel` to create all of the Elasticsearch distributions.
@@ -91,15 +91,15 @@ The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/ma
 ** e.g. `<version>/archives/<unique id>`
 6. Replace `<version>/manifest-latest.json` in GCS with this newest manifest.
 ** This allows the `KBN_ES_SNAPSHOT_USE_UNVERIFIED` flag to work.
-7. Trigger the verification job, to run the full Kibana CI test suite with this snapshot.
+7. Trigger the verification job, to run the full {kib} CI test suite with this snapshot.
 
 ===== Verification and Promotion
 
-https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full Kibana CI pipeline, and promotes if it there are no test failures.
+https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full {kib} CI pipeline, and promotes if it there are no test failures.
 
-The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the kibana repo].
+The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the {kib} repo].
 
-1. Checkout Kibana and set up CI environment as normal.
+1. Checkout {kib} and set up CI environment as normal.
 2. Set the `ES_SNAPSHOT_MANIFEST` env var to point to the latest snapshot manifest.
 3. Run CI (functional tests, integration tests, etc).
 4. After CI

docs/developer/advanced/index.asciidoc

@@ -0,0 +1,12 @@
+[[advanced]]
+== Advanced
+
+* <<running-elasticsearch>>
+* <<development-es-snapshots>>
+* <<development-basepath>>
+
+include::development-es-snapshots.asciidoc[]
+
+include::running-elasticsearch.asciidoc[]
+
+include::development-basepath.asciidoc[]

docs/developer/advanced/running-elasticsearch.asciidoc

@@ -0,0 +1,118 @@
+[[running-elasticsearch]]
+=== Running elasticsearch during development
+
+There are many ways to run Elasticsearch while you are developing.
+
+[float]
+
+==== By snapshot
+
+This will run a snapshot of elasticsearch that is usually built nightly. Read more about <<development-es-snapshots>>.
+
+[source,bash]
+----
+yarn es snapshot
+----
+
+See all available options, like how to specify a specific license, with the `--help` flag.
+
+[source,bash]
+----
+yarn es snapshot --help
+----
+
+`trial` will give you access to all capabilities.
+
+**Keeping data between snapshots**
+
+If you want to keep the data inside your Elasticsearch between usages of this command, you should use the following command, to keep your data folder outside the downloaded snapshot folder:
+
+[source,bash]
+----
+yarn es snapshot -E path.data=../data
+----
+
+==== By source
+
+If you have the Elasticsearch repo checked out locally and wish to run against that, use `source`. By default, it will reference an elasticsearch checkout which is a sibling to the {kib} directory named elasticsearch. If you wish to use a checkout in another location you can provide that by supplying --source-path 
+
+[source,bash]
+----
+yarn es source
+----
+
+==== From an archive
+
+Use this if you already have a distributable. For released versions, one can be obtained on the Elasticsearch downloads page.
+
+[source,bash]
+----
+yarn es archive <full_path_to_archive>
+----
+
+Each of these will run Elasticsearch with a basic license. Additional options are available, pass --help for more information.
+
+==== From a remote host
+
+You can save some system resources, and the effort of generating sample data, if you have a remote Elasticsearch cluster to connect to. (Elasticians: you do! Check with your team about where to find credentials)
+
+You'll need to create a kibana.dev.yml (<<customize-kibana-yml>>) and add the following to it:
+
+[source,bash]
+----
+elasticsearch.hosts:
+  - {{ url }}
+elasticsearch.username: {{ username }}
+elasticsearch.password: {{ password }}
+elasticsearch.ssl.verificationMode: none
+----
+
+If many other users will be interacting with your remote cluster, you'll want to add the following to avoid causing conflicts:
+
+[source,bash]
+----
+kibana.index: '.{YourGitHubHandle}-kibana'
+xpack.task_manager.index: '.{YourGitHubHandle}-task-manager-kibana'
+----
+
+===== Running remote clusters
+
+Setup remote clusters for cross cluster search (CCS) and cross cluster replication (CCR).
+
+Start your primary cluster by running:
+
+[source,bash]
+----
+yarn es snapshot -E path.data=../data_prod1
+----
+
+Start your remote cluster by running:
+
+[source,bash]
+----
+yarn es snapshot -E transport.port=9500 -E http.port=9201 -E path.data=../data_prod2
+----
+
+Once both clusters are running, start {kib}. {kib} will connect to the primary cluster.
+
+Setup the remote cluster in {kib} from either Management -> Elasticsearch -> Remote Clusters UI or by running the following script in Console.
+
+[source,bash]
+----
+PUT _cluster/settings
+{
+  "persistent": {
+    "cluster": {
+      "remote": {
+        "cluster_one": {
+          "seeds": [
+            "localhost:9500"
+          ]
+        }
+      }
+    }
+  }
+}
+----
+
+Follow the cross-cluster search instructions for setting up index patterns to search across clusters (<<management-cross-cluster-search>>).

docs/developer/architecture/add-data-tutorials.asciidoc

@@ -0,0 +1,38 @@
+[[add-data-tutorials]]
+=== Add data tutorials
+
+`Add Data` in the {kib} Home application contains tutorials for setting up data flows in the Elastic stack.
+
+Each tutorial contains three sets of instructions:
+
+* `On Premise.` Set up a data flow when both {kib} and Elasticsearch are running on premise.
+* `On Premise Elastic Cloud.` Set up a data flow when {kib} is running on premise and Elasticsearch is running on Elastic Cloud.
+* `Elastic Cloud.` Set up a data flow when both {kib} and Elasticsearch are running on Elastic Cloud.
+
+[float]
+==== Creating a new tutorial
+1. Create a new directory in the link:https://github.com/elastic/kibana/tree/master/src/plugins/home/server/tutorials[tutorials directory].
+2. In the new directory, create a file called `index.ts` that exports a function.
+The function must return a function object that conforms to the `TutorialSchema` interface link:{kib-repo}tree/{branch}/src/plugins/home/server/services/tutorials/lib/tutorial_schema.ts[tutorial schema].
+3. Register the tutorial in link:{kib-repo}tree/{branch}/src/plugins/home/server/tutorials/register.ts[register.ts] by adding it to the `builtInTutorials`.
+// TODO update path once assets are migrated
+4. Add image assets to the link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/tutorial_resources[tutorial_resources directory].
+5. Run {kib} locally to preview the tutorial.
+6. Create a PR and go through the review process to get the changes approved.
+
+If you are creating a new plugin and the tutorial is only related to that plugin, you can also place the `TutorialSchema` object into your plugin folder. Add `home` to the `requiredPlugins` list in your `kibana.json` file.
+Then register the tutorial object by calling `home.tutorials.registerTutorial(tutorialObject)` in the `setup` lifecycle of your server plugin.
+
+[float]
+===== Variables
+String values can contain variables that are substituted when rendered. Variables are specified by `{}`.
+For example: `{config.docs.version}` is rendered as `6.2` when running the tutorial in {kib} 6.2.
+
+link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/np_ready/components/tutorial/replace_template_strings.js#L23[Provided variables]
+
+[float]
+===== Markdown
+String values can contain limited Markdown syntax.
+
+link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/components/tutorial/content.js#L8[Enabled Markdown grammars]
+

docs/developer/visualize/development-visualize-index.asciidoc → docs/developer/architecture/development-visualize-index.asciidoc

@@ -1,13 +1,13 @@
 [[development-visualize-index]]
-== Developing Visualizations
+=== Developing Visualizations
 
 [IMPORTANT]
 ==============================================
-These pages document internal APIs and are not guaranteed to be supported across future versions of Kibana.
+These pages document internal APIs and are not guaranteed to be supported across future versions of {kib}.
 ==============================================
 
 The internal APIs for creating custom visualizations are in a state of heavy churn as
-they are being migrated to the new Kibana platform, and large refactorings have been
+they are being migrated to the new {kib} platform, and large refactorings have been
 happening across minor releases in the `7.x` series. In particular, in `7.5` and later
 we have made significant changes to the legacy APIs as we work to gradually replace them.
 
@@ -20,7 +20,7 @@ If you would like to keep up with progress on the visualizations plugin in the m
 here are a few resources:
 
 * The <<breaking-changes,breaking changes>> documentation, where we try to capture any changes to the APIs as they occur across minors.
-* link:https://github.com/elastic/kibana/issues/44121[Meta issue] which is tracking the move of the plugin to the new Kibana platform
+* link:https://github.com/elastic/kibana/issues/44121[Meta issue] which is tracking the move of the plugin to the new {kib} platform
 * Our link:https://www.elastic.co/blog/join-our-elastic-stack-workspace-on-slack[Elastic Stack workspace on Slack].
 * The {kib-repo}blob/{branch}/src/plugins/visualizations[source code], which will continue to be
 the most accurate source of information.

docs/developer/architecture/index.asciidoc

@@ -0,0 +1,25 @@
+[[kibana-architecture]]
+== Architecture
+
+[IMPORTANT]
+==============================================
+{kib} developer services and apis are in a state of constant development. We cannot provide backwards compatibility at this time due to the high rate of change.
+==============================================
+
+Our developer services are changing all the time. One of the best ways to discover and learn about them is to read the available
+READMEs from all the plugins inside our {kib-repo}tree/{branch}/src/plugins[open source plugins folder] and our
+{kib-repo}/tree/{branch}/x-pack/plugins[commercial plugins folder].
+
+A few services also automatically generate api documentation which can be browsed inside the {kib-repo}tree/{branch}/docs/development[docs/development section of our repo]
+
+A few notable services are called out below.
+
+* <<development-security>>
+* <<add-data-tutorials>>
+* <<development-visualize-index>>
+
+include::add-data-tutorials.asciidoc[]
+
+include::development-visualize-index.asciidoc[]
+
+include::security/index.asciidoc[]