Merge branch 'master' of github.com:elastic/kibana into apm-fleet-tut…

…orial

.buildkite/scripts/post_build_kibana.sh

@@ -6,7 +6,7 @@ if [[ ! "${DISABLE_CI_STATS_SHIPPING:-}" ]]; then
   echo "--- Ship Kibana Distribution Metrics to CI Stats"
   node scripts/ship_ci_stats \
     --metrics target/optimizer_bundle_metrics.json \
-    --metrics packages/kbn-ui-shared-deps/target/metrics.json
+    --metrics node_modules/@kbn/ui-shared-deps/shared_built_assets/metrics.json
 fi
 
 echo "--- Upload Build Artifacts"

.eslintignore

@@ -37,7 +37,7 @@ snapshots.js
 /packages/kbn-test/src/functional_test_runner/__tests__/fixtures/
 /packages/kbn-test/src/functional_test_runner/lib/config/__tests__/fixtures/
 /packages/kbn-ui-framework/dist
-/packages/kbn-ui-shared-deps/flot_charts
+/packages/kbn-ui-shared-deps/src/flot_charts
 /packages/kbn-monaco/src/painless/antlr
 
 # Bazel

.eslintrc.js

@@ -875,6 +875,16 @@ module.exports = {
       },
     },
 
+    /**
+     * Cases overrides
+     */
+    {
+      files: ['x-pack/plugins/cases/**/*.{js,mjs,ts,tsx}'],
+      rules: {
+        'no-duplicate-imports': 'error',
+      },
+    },
+
     /**
      * Security Solution overrides
      */
@@ -1461,7 +1471,7 @@ module.exports = {
       },
     },
     {
-      files: ['packages/kbn-ui-shared-deps/flot_charts/**/*.js'],
+      files: ['packages/kbn-ui-shared-deps/src/flot_charts/**/*.js'],
       env: {
         jquery: true,
       },

.github/CODEOWNERS

@@ -76,6 +76,7 @@
 /src/plugins/apm_oss/  @elastic/apm-ui
 /src/apm.js @elastic/kibana-core @vigneshshanmugam
 /packages/kbn-apm-config-loader/ @elastic/kibana-core @vigneshshanmugam
+/src/core/types/elasticsearch @elastic/apm-ui
 #CC# /src/plugins/apm_oss/ @elastic/apm-ui
 #CC# /x-pack/plugins/observability/ @elastic/apm-ui
 

.i18nrc.json

@@ -11,7 +11,7 @@
     "uiActionsExamples": "examples/ui_action_examples",
     "share": "src/plugins/share",
     "home": "src/plugins/home",
-    "flot": "packages/kbn-ui-shared-deps/flot_charts",
+    "flot": "packages/kbn-ui-shared-deps/src/flot_charts",
     "charts": "src/plugins/charts",
     "esUi": "src/plugins/es_ui_shared",
     "devTools": "src/plugins/dev_tools",

dev_docs/best_practices.mdx

@@ -241,35 +241,136 @@ There are some exceptions where a separate repo makes sense. However, they are e
 
 It may be tempting to get caught up in the dream of writing the next package which is published to npm and downloaded millions of times a week. Knowing the quality of developers that are working on Kibana, this is a real possibility. However, knowing which packages will see mass adoption is impossible to predict. Instead of jumping directly to writing code in a separate repo and accepting all of the complications that come along with it, prefer keeping code inside the Kibana repo. A [Kibana package](https://github.com/elastic/kibana/tree/master/packages) can be used to publish a package to npm, while still keeping the code inside the Kibana repo. Move code to an external repo only when there is a good reason, for example to enable external contributions.
 
-## Hardening
-
-Review the following items related to vulnerability and security risks.
-
-- XSS
-  - Check for usages of `dangerouslySetInnerHtml`, `Element.innerHTML`, `Element.outerHTML`
-  - Ensure all user input is properly escaped.
-  - Ensure any input in `$.html`, `$.append`, `$.appendTo`, $.prepend`, `$.prependTo`is escaped. Instead use`$.text`, or don't use jQuery at all.
-- CSRF
-  - Ensure all APIs are running inside the Kibana HTTP service.
-- RCE
-  - Ensure no usages of `eval`
-  - Ensure no usages of dynamic requires
-  - Check for template injection
-  - Check for usages of templating libraries, including `_.template`, and ensure that user provided input isn't influencing the template and is only used as data for rendering the template.
-  - Check for possible prototype pollution.
-- Prototype Pollution - more info [here](https://docs.google.com/document/d/19V-d9sb6IF-fbzF4iyiPpAropQNydCnoJApzSX5FdcI/edit?usp=sharing)
-  - Check for instances of `anObject[a][b] = c` where a, b, and c are user defined. This includes code paths where the following logical code steps could be performed in separate files by completely different operations, or recursively using dynamic operations.
-  - Validate any user input, including API url-parameters/query-parameters/payloads, preferable against a schema which only allows specific keys/values. At a very minimum, black-list `__proto__` and `prototype.constructor` for use within keys
-  - When calling APIs which spawn new processes or potentially perform code generation from strings, defensively protect against Prototype Pollution by checking `Object.hasOwnProperty` if the arguments to the APIs originate from an Object. An example is the Code app's [spawnProcess](https://github.com/elastic/kibana/blob/b49192626a8528af5d888545fb14cd1ce66a72e7/x-pack/legacy/plugins/code/server/lsp/workspace_command.ts#L40-L44).
-    - Common Node.js offenders: `child_process.spawn`, `child_process.exec`, `eval`, `Function('some string')`, `vm.runIn*Context(x)`
-    - Common Client-side offenders: `eval`, `Function('some string')`, `setTimeout('some string', num)`, `setInterval('some string', num)`
-- Check for accidental reveal of sensitive information
-  - The biggest culprit is errors which contain stack traces or other sensitive information which end up in the HTTP Response
-- Checked for Mishandled API requests
-  - Ensure no sensitive cookies are forwarded to external resources.
-  - Ensure that all user controllable variables that are used in constructing a URL are escaped properly. This is relevant when using `transport.request` with the Elasticsearch client as no automatic escaping is performed.
-- Reverse tabnabbing - https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/HTML5_Security_Cheat_Sheet.md#tabnabbing
-  - When there are user controllable links or hard-coded links to third-party domains that specify target="\_blank" or target="\_window", the `a` tag should have the rel="noreferrer noopener" attribute specified.
-  - Allowing users to input markdown is a common culprit, a custom link renderer should be used
-- SSRF - https://www.owasp.org/index.php/Server_Side_Request_Forgery
-  - All network requests made from the Kibana server should use an explicit configuration or white-list specified in the `kibana.yml`
+## Security best practices
+
+When writing code for Kibana, be sure to follow these best practices to avoid common vulnerabilities. Refer to the included Open Web
+Application Security Project (OWASP) references to learn more about these types of attacks.
+
+### Cross-site Scripting (XSS)
+
+[_OWASP reference for XSS_](https://owasp.org/www-community/attacks/xss)
+
+XSS is a class of attacks where malicious scripts are injected into vulnerable websites. Kibana defends against this by using the React
+framework to safely encode data that is rendered in pages, the EUI framework to [automatically sanitize
+links](https://elastic.github.io/eui/#/navigation/link#link-validation), and a restrictive `Content-Security-Policy` header.
+
+**Best practices**
+
+* Check for dangerous functions or assignments that can result in unescaped user input in the browser DOM. Avoid using:
+  * **React:** [`dangerouslySetInnerHtml`](https://reactjs.org/docs/dom-elements.html#dangerouslysetinnerhtml).
+  * **Browser DOM:** `Element.innerHTML` and `Element.outerHTML`.
+* If using the aforementioned unsafe functions or assignments is absolutely necessary, follow [these XSS prevention
+rules](https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html#xss-prevention-rules) to ensure that
+user input is not inserted into unsafe locations and that it is escaped properly.
+* Use EUI components to build your UI, particularly when rendering `href` links. Otherwise, sanitize user input before rendering links to
+ensure that they do not use the `javascript:` protocol.
+* Don't use the `eval`, `Function`, and `_.template` functions -- these are restricted by ESLint rules.
+* Be careful when using `setTimeout` and `setInterval` in client-side code. If an attacker can manipulate the arguments and pass a string to
+one of these, it is evaluated dynamically, which is equivalent to the dangerous `eval` function.
+
+### Cross-Site Request Forgery (CSRF/XSRF)
+
+[_OWASP reference for CSRF_](https://owasp.org/www-community/attacks/csrf)
+
+CSRF is a class of attacks where a user is forced to execute an action on a vulnerable website that they're logged into, usually without
+their knowledge. Kibana defends against this by requiring [custom request
+headers](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#use-of-custom-request-headers)
+for API endpoints. For more information, see [API Request
+Headers](https://www.elastic.co/guide/en/kibana/master/api.html#api-request-headers).
+
+**Best practices**
+
+* Ensure all HTTP routes are registered with the [Kibana HTTP service](https://www.elastic.co/guide/en/kibana/master/http-service.html) to
+take advantage of the custom request header security control.
+  * Note that HTTP GET requests do **not** require the custom request header; any routes that change data should [adhere to the HTTP
+specification and use a different method (PUT, POST, etc.)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods)
+
+### Remote Code Execution (RCE)
+
+[_OWASP reference for Command Injection_](https://owasp.org/www-community/attacks/Command_Injection),
+[_OWASP reference for Code Injection_](https://owasp.org/www-community/attacks/Code_Injection)
+
+RCE is a class of attacks where an attacker executes malicious code or commands on a vulnerable server. Kibana defends against this by using
+ESLint rules to restrict vulnerable functions, and by hooking into or hardening usage of these in third-party dependencies.
+
+**Best practices**
+
+* Don't use the `eval`, `Function`, and `_.template` functions -- these are restricted by ESLint rules.
+* Don't use dynamic `require`.
+* Check for usages of templating libraries. Ensure that user-provided input doesn't influence the template and is used only as data for
+rendering the template.
+* Take extra caution when spawning child processes with any user input or parameters that are user-controlled.
+
+### Prototype Pollution
+
+Prototype Pollution is an attack that is unique to JavaScript environments. Attackers can abuse JavaScript's prototype inheritance to
+"pollute" objects in the application, which is often used as a vector for XSS or RCE vulnerabilities. Kibana defends against this by
+hardening sensitive functions (such as those exposed by `child_process`), and by requiring validation on all HTTP routes by default.
+
+**Best practices**
+
+* Check for instances of `anObject[a][b] = c` where `a`, `b`, and `c` are controlled by user input. This includes code paths where the
+following logical code steps could be performed in separate files by completely different operations, or by recursively using dynamic
+operations.
+* Validate all user input, including API URL parameters, query parameters, and payloads. Preferably, use a schema that only allows specific
+keys and values. At a minimum, implement a deny-list that prevents `__proto__` and `prototype.constructor` from being used within object
+keys.
+* When calling APIs that spawn new processes or perform code generation from strings, protect against Prototype Pollution by checking if
+`Object.hasOwnProperty` has arguments to the APIs that originate from an Object. An example is the defunct Code app's
+[`spawnProcess`](https://github.com/elastic/kibana/blob/b49192626a8528af5d888545fb14cd1ce66a72e7/x-pack/legacy/plugins/code/server/lsp/workspace_command.ts#L40-L44)
+function.
+  * Common Node.js offenders: `child_process.spawn`, `child_process.exec`, `eval`, `Function('some string')`, `vm.runInContext(x)`,
+`vm.runInNewContext(x)`, `vm.runInThisContext()`
+  * Common client-side offenders: `eval`, `Function('some string')`, `setTimeout('some string', num)`, `setInterval('some string', num)`
+
+See also:
+
+* [Prototype pollution: The dangerous and underrated vulnerability impacting JavaScript applications |
+portswigger.net](https://portswigger.net/daily-swig/prototype-pollution-the-dangerous-and-underrated-vulnerability-impacting-javascript-applications)
+* [Prototype pollution attack in NodeJS application | Olivier
+Arteau](https://github.com/HoLyVieR/prototype-pollution-nsec18/blob/master/paper/JavaScript_prototype_pollution_attack_in_NodeJS.pdf)
+
+### Server-Side Request Forgery (SSRF)
+
+[_OWASP reference for SSRF_](https://owasp.org/www-community/attacks/Server_Side_Request_Forgery)
+
+SSRF is a class of attacks where a vulnerable server is forced to make an unintended request, usually to an HTTP API. This is often used as
+a vector for information disclosure or injection attacks.
+
+**Best practices**
+
+* Ensure that all outbound requests from the Kibana server use hard-coded URLs.
+* If user input is used to construct a URL for an outbound request, ensure that an allow-list is used to validate the endpoints and that
+user input is escaped properly. Ideally, the allow-list should be set in `kibana.yml`, so only server administrators can change it.
+  * This is particularly relevant when using `transport.request` with the Elasticsearch client, as no automatic escaping is performed.
+  * Note that URLs are very hard to validate properly; exact match validation for user input is most preferable, while URL parsing or RegEx
+validation should only be used if absolutely necessary.
+
+### Reverse tabnabbing
+
+[_OWASP reference for Reverse Tabnabbing_](https://owasp.org/www-community/attacks/Reverse_Tabnabbing)
+
+Reverse tabnabbing is an attack where a link to a malicious page is used to rewrite a vulnerable parent page. This is often used as a vector
+for phishing attacks. Kibana defends against this by using the EUI framework, which automatically adds the `rel` attribute to anchor tags,
+buttons, and other vulnerable DOM elements.
+
+**Best practices**
+
+* Use EUI components to build your UI whenever possible. Otherwise, ensure that any DOM elements that have an `href` attribute also have the
+`rel="noreferrer noopener"` attribute specified. For more information, refer to the [OWASP HTML5 Security Cheat
+Sheet](https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/HTML5_Security_Cheat_Sheet.md#tabnabbing).
+* If using a non-EUI markdown renderer, use a custom link renderer for rendered links.
+
+### Information disclosure
+
+Information disclosure is not an attack, but it describes whenever sensitive information is accidentally revealed. This can be configuration
+info, stack traces, or other data that the user is not authorized to access. This concern cannot be addressed with a single security
+control, but at a high level, Kibana relies on the hapi framework to automatically redact stack traces and detailed error messages in HTTP
+5xx response payloads.
+
+**Best practices**
+
+* Look for instances where sensitive information might accidentally be revealed, particularly in error messages, in the UI, and URL
+parameters that are exposed to users.
+* Make sure that sensitive request data is not forwarded to external resources. For example, copying client request headers and using them
+to make an another request could accidentally expose the user's credentials.

docs/canvas/canvas-function-reference.asciidoc

@@ -71,7 +71,7 @@ Alias: `condition`
 [[alterColumn_fn]]
 === `alterColumn`
 
-Converts between core types, including `string`, `number`, `null`, `boolean`, and `date`, and renames columns. See also <<mapColumn_fn>> and <<staticColumn_fn>>.
+Converts between core types, including `string`, `number`, `null`, `boolean`, and `date`, and renames columns. See also <<mapColumn_fn>>, <<mathColumn_fn>>, and <<staticColumn_fn>>.
 
 *Expression syntax*
 [source,js]
@@ -1717,23 +1717,23 @@ Adds a column calculated as the result of other columns. Changes are made only w
 |===
 |Argument |Type |Description
 
+|`id`
+
+|`string`, `null`
+|An optional id of the resulting column. When no id is provided, the id will be looked up from the existing column by the provided name argument. If no column with this name exists yet, a new column with this name and an identical id will be added to the table.
+
 |_Unnamed_ ***
 
 Aliases: `column`, `name`
 |`string`
-|The name of the resulting column.
+|The name of the resulting column. Names are not required to be unique.
 
 |`expression` ***
 
 Aliases: `exp`, `fn`, `function`
 |`boolean`, `number`, `string`, `null`
 |A Canvas expression that is passed to each row as a single row `datatable`.
 
-|`id`
-
-|`string`, `null`
-|An optional id of the resulting column. When not specified or `null` the name argument is used as id.
-
 |`copyMetaFrom`
 
 |`string`, `null`
@@ -1808,6 +1808,47 @@ Default: `"throw"`
 *Returns:* `number` | `boolean` | `null`
 
 
+[float]
+[[mathColumn_fn]]
+=== `mathColumn`
+
+Adds a column by evaluating `TinyMath` on each row. This function is optimized for math, so it performs better than the <<mapColumn_fn>> with a <<math_fn>>.
+*Accepts:* `datatable`
+
+[cols="3*^<"]
+|===
+|Argument |Type |Description
+
+|id ***
+|`string`
+|id of the resulting column. Must be unique.
+
+|name ***
+|`string`
+|The name of the resulting column. Names are not required to be unique.
+
+|_Unnamed_
+
+Alias: `expression`
+|`string`
+|A `TinyMath` expression evaluated on each row. See https://www.elastic.co/guide/en/kibana/current/canvas-tinymath-functions.html.
+
+|`onError`
+
+|`string`
+|In case the `TinyMath` evaluation fails or returns NaN, the return value is specified by onError. For example, `"null"`, `"zero"`, `"false"`, `"throw"`. When `"throw"`, it will throw an exception, terminating expression execution.
+
+Default: `"throw"`
+
+|`copyMetaFrom`
+
+|`string`, `null`
+|If set, the meta object from the specified column id is copied over to the specified target column. Throws an exception if the column doesn't exist
+|===
+
+*Returns:* `datatable`
+
+
 [float]
 [[metric_fn]]
 === `metric`
@@ -2581,7 +2622,7 @@ Default: `false`
 [[staticColumn_fn]]
 === `staticColumn`
 
-Adds a column with the same static value in every row. See also <<alterColumn_fn>> and <<mapColumn_fn>>.
+Adds a column with the same static value in every row. See also <<alterColumn_fn>>, <<mapColumn_fn>>, and <<mathColumn_fn>>.
 
 *Accepts:* `datatable`