[8.7] [packages] migrate all plugins to packages (elastic#148130) (el…

…astic#157306)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[packages] migrate all plugins to packages
(elastic#148130)](elastic#148130)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT
[{"author":{"name":"Spencer","email":"spencer@elastic.co"},"sourceCommit":{"committedDate":"2023-02-09T03:06:50Z","message":"[packages]
migrate all plugins to packages (elastic#148130)\n\nFixes
elastic#149344 PR migrates
all plugins to packages automatically. It does this\r\nusing `node
scripts/lint_packages` to automatically migrate\r\n`kibana.json` files
to `kibana.jsonc` files. By doing this automatically\r\nwe can simplify
many build and testing procedures to only support\r\npackages, and not
both \"packages\" and \"synthetic packages\" (basically\r\npointers to
plugins).\r\n\r\nThe majority of changes are in operations related code,
so we'll be\r\nhaving operations review this before marking it ready for
review. The\r\nvast majority of the code owners are simply pinged
because we deleted\r\nall `kibana.json` files and replaced them with
`kibana.jsonc` files, so\r\nwe plan on leaving the PR ready-for-review
for about 24 hours before\r\nmerging (after feature freeze), assuming we
don't have any blockers\r\n(especially from @elastic/kibana-core since
there are a few core\r\nspecific changes, though the majority were
handled in elastic#149370).\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"1b8581540295fde746dae6b4a09d74fb5821bfef","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","backport:skip","v8.8.0"],"number":148130,"url":"elastic#148130
migrate all plugins to packages (elastic#148130)\n\nFixes
elastic#149344 PR migrates
all plugins to packages automatically. It does this\r\nusing `node
scripts/lint_packages` to automatically migrate\r\n`kibana.json` files
to `kibana.jsonc` files. By doing this automatically\r\nwe can simplify
many build and testing procedures to only support\r\npackages, and not
both \"packages\" and \"synthetic packages\" (basically\r\npointers to
plugins).\r\n\r\nThe majority of changes are in operations related code,
so we'll be\r\nhaving operations review this before marking it ready for
review. The\r\nvast majority of the code owners are simply pinged
because we deleted\r\nall `kibana.json` files and replaced them with
`kibana.jsonc` files, so\r\nwe plan on leaving the PR ready-for-review
for about 24 hours before\r\nmerging (after feature freeze), assuming we
don't have any blockers\r\n(especially from @elastic/kibana-core since
there are a few core\r\nspecific changes, though the majority were
handled in elastic#149370).\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"1b8581540295fde746dae6b4a09d74fb5821bfef"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"elastic#148130
migrate all plugins to packages (elastic#148130)\n\nFixes
elastic#149344 PR migrates
all plugins to packages automatically. It does this\r\nusing `node
scripts/lint_packages` to automatically migrate\r\n`kibana.json` files
to `kibana.jsonc` files. By doing this automatically\r\nwe can simplify
many build and testing procedures to only support\r\npackages, and not
both \"packages\" and \"synthetic packages\" (basically\r\npointers to
plugins).\r\n\r\nThe majority of changes are in operations related code,
so we'll be\r\nhaving operations review this before marking it ready for
review. The\r\nvast majority of the code owners are simply pinged
because we deleted\r\nall `kibana.json` files and replaced them with
`kibana.jsonc` files, so\r\nwe plan on leaving the PR ready-for-review
for about 24 hours before\r\nmerging (after feature freeze), assuming we
don't have any blockers\r\n(especially from @elastic/kibana-core since
there are a few core\r\nspecific changes, though the majority were
handled in elastic#149370).\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"1b8581540295fde746dae6b4a09d74fb5821bfef"}}]}]
BACKPORT-->

---------

Co-authored-by: Spencer <spencer@elastic.co>

.buildkite/scripts/build_kibana.sh

@@ -8,14 +8,15 @@ export KBN_NP_PLUGINS_BUILT=true
 
 echo "--- Build Kibana Distribution"
 
-BUILD_ARGS=""
-is_pr_with_label "ci:build-all-platforms" && BUILD_ARGS="--all-platforms"
-is_pr_with_label "ci:build-example-plugins" && BUILD_ARGS="$BUILD_ARGS --example-plugins"
-is_pr_with_label "ci:build-docker-cross-compile" && BUILD_ARG="$BUILD_ARGS --docker-cross-compile"
-is_pr_with_label "ci:build-os-packages" || BUILD_ARGS="$BUILD_ARGS --skip-os-packages"
-is_pr_with_label "ci:build-canvas-shareable-runtime" || BUILD_ARGS="$BUILD_ARGS --skip-canvas-shareable-runtime"
-is_pr_with_label "ci:build-docker-contexts" || BUILD_ARGS="$BUILD_ARGS --skip-docker-contexts"
-node scripts/build $BUILD_ARGS
+BUILD_ARGS=("--with-test-plugins" "--with-example-plugins")
+is_pr_with_label "ci:build-all-platforms" && BUILD_ARGS+=("--all-platforms")
+is_pr_with_label "ci:build-docker-cross-compile" && BUILD_ARGS+=("--docker-cross-compile")
+is_pr_with_label "ci:build-os-packages" || BUILD_ARGS+=("--skip-os-packages")
+is_pr_with_label "ci:build-canvas-shareable-runtime" || BUILD_ARGS+=("--skip-canvas-shareable-runtime")
+is_pr_with_label "ci:build-docker-contexts" || BUILD_ARGS+=("--skip-docker-contexts")
+
+echo "> node scripts/build" "${BUILD_ARGS[@]}"
+node scripts/build "${BUILD_ARGS[@]}"
 
 if is_pr_with_label "ci:build-cloud-image"; then
   echo "$KIBANA_DOCKER_PASSWORD" | docker login -u "$KIBANA_DOCKER_USERNAME" --password-stdin docker.elastic.co

.buildkite/scripts/steps/artifacts/build.sh

@@ -7,7 +7,7 @@ set -euo pipefail
 source .buildkite/scripts/steps/artifacts/env.sh
 
 echo "--- Build Kibana artifacts"
-node scripts/build --all-platforms --debug --docker-cross-compile $(echo "$BUILD_ARGS")
+node scripts/build --all-platforms --debug --docker-cross-compile "${BUILD_ARGS[@]}"
 
 echo "--- Extract default i18n messages"
 mkdir -p target/i18n

.buildkite/scripts/steps/artifacts/docker_context.sh

@@ -11,7 +11,7 @@ KIBANA_DOCKER_CONTEXT="${KIBANA_DOCKER_CONTEXT:="default"}"
 
 echo "--- Create contexts"
 mkdir -p target
-node scripts/build --skip-initialize --skip-generic-folders --skip-platform-folders --skip-archives --docker-context-use-local-artifact $(echo "$BUILD_ARGS")
+node scripts/build --skip-initialize --skip-generic-folders --skip-platform-folders --skip-archives --docker-context-use-local-artifact "${BUILD_ARGS[@]}"
 
 echo "--- Setup context"
 DOCKER_BUILD_FOLDER=$(mktemp -d)

.buildkite/scripts/steps/artifacts/env.sh

@@ -15,11 +15,11 @@ fi
 
 if [[ "$RELEASE_BUILD" == "true" ]]; then
   FULL_VERSION="$QUALIFIER_VERSION"
-  BUILD_ARGS="--release --version-qualifier=$VERSION_QUALIFIER"
+  BUILD_ARGS=("--release" "--version-qualifier=$VERSION_QUALIFIER")
   WORKFLOW="staging"
 else
   FULL_VERSION="$QUALIFIER_VERSION-SNAPSHOT"
-  BUILD_ARGS="--version-qualifier=$VERSION_QUALIFIER"
+  BUILD_ARGS=("--version-qualifier=$VERSION_QUALIFIER")
   WORKFLOW="snapshot"
 fi
 

.buildkite/scripts/steps/checks.sh

@@ -6,6 +6,7 @@ export DISABLE_BOOTSTRAP_VALIDATION=false
 .buildkite/scripts/bootstrap.sh
 
 .buildkite/scripts/steps/checks/precommit_hook.sh
+.buildkite/scripts/steps/checks/packages.sh
 .buildkite/scripts/steps/checks/ts_projects.sh
 .buildkite/scripts/steps/checks/packages.sh
 .buildkite/scripts/steps/checks/bazel_packages.sh
@@ -18,7 +19,6 @@ export DISABLE_BOOTSTRAP_VALIDATION=false
 .buildkite/scripts/steps/checks/i18n.sh
 .buildkite/scripts/steps/checks/file_casing.sh
 .buildkite/scripts/steps/checks/licenses.sh
-.buildkite/scripts/steps/checks/plugins_with_circular_deps.sh
 .buildkite/scripts/steps/checks/test_projects.sh
 .buildkite/scripts/steps/checks/test_hardening.sh
 .buildkite/scripts/steps/checks/ftr_configs.sh

.buildkite/scripts/steps/checks/packages.sh

@@ -4,7 +4,7 @@ set -euo pipefail
 
 source .buildkite/scripts/common/util.sh
 
-echo --- Lint packages
+echo --- Lint Packages
 cmd="node scripts/lint_packages"
 if is_pr && ! is_auto_commit_disabled; then
   cmd="$cmd --fix"

.buildkite/scripts/steps/checks/ts_projects.sh

@@ -4,7 +4,7 @@ set -euo pipefail
 
 source .buildkite/scripts/common/util.sh
 
-echo --- Lint TS projects
+echo --- Lint TS Projects
 cmd="node scripts/lint_ts_projects"
 if is_pr && ! is_auto_commit_disabled; then
   cmd="$cmd --fix"

.eslintrc.js

@@ -121,7 +121,7 @@ const VENN_DIAGRAM_HEADER = `
 
 /** Packages which should not be included within production code. */
 const DEV_PACKAGE_DIRS = getPackages(REPO_ROOT).flatMap((pkg) =>
-  pkg.isDevOnly ? pkg.normalizedRepoRelativeDir : []
+  pkg.isDevOnly() ? pkg.normalizedRepoRelativeDir : []
 );
 
 /** Directories (at any depth) which include dev-only code. */
@@ -1736,7 +1736,10 @@ module.exports = {
      * Code inside .buildkite runs separately from everything else in CI, before bootstrap, with ts-node. It needs a few tweaks because of this.
      */
     {
-      files: 'packages/kbn-{package-*,repo-*,dep-*}/**/*',
+      files: [
+        'packages/kbn-{package-*,repo-*,dep-*}/**/*',
+        'packages/kbn-find-used-node-modules/**/*',
+      ],
       rules: {
         'max-classes-per-file': 'off',
       },

.gitignore

@@ -57,6 +57,7 @@ webpackstats.json
 !/config/serverless.security.yml
 !/config/node.options
 coverage
+!/test/common/fixtures/plugins/coverage
 selenium
 .babel_register_cache.json
 .webpack.babelcache

dev_docs/api_welcome.mdx

@@ -67,7 +67,7 @@ If that isn't the case, please file an issue, it could be a bug with the system.
 We are [aware of some performance issues](https://github.com/elastic/elastic-docs/issues/274) with deeply nested, or large APIs.
 
 In the short term, the best thing you can do is avoid deeply nested API items. Use interfaces rather than inlined objects. Also consider
-adding `serviceFolders` in your kibana.json. This will automatically split your docs up based on which APIs are defined within the service folders.
+adding `serviceFolders` in your kibana.jsonc. This will automatically split your docs up based on which APIs are defined within the service folders.
 They will get built into a doc with an id of
 `kib${PluginName}${ServiceName}PluginApi`. The data plugin does this, so you
  can [check that out as an example](https://github.com/elastic/kibana/blob/main/src/plugins/data/kibana.json#L13).

dev_docs/operations/operations_landing.mdx

@@ -36,7 +36,6 @@ layout: landing
     { pageId: "kibDevDocsOpsKbnPm" },
     { pageId: "kibDevDocsOpsOptimizer" },
     { pageId: "kibDevDocsOpsBabelPreset" },
-    { pageId: "kibDevDocsOpsBabelPluginPackageImports" },
     { pageId: "kibDevDocsOpsUiSharedDepsNpm" },
     { pageId: "kibDevDocsOpsUiSharedDepsSrc" },
     { pageId: "kibDevDocsOpsPluginDiscovery" },

examples/bfetch_explorer/kibana.jsonc

@@ -0,0 +1,17 @@
+{
+  "type": "plugin",
+  "id": "@kbn/bfetch-explorer-plugin",
+  "owner": "@elastic/appex-sharedux",
+  "plugin": {
+    "id": "bfetchExplorer",
+    "server": true,
+    "browser": true,
+    "requiredPlugins": [
+      "bfetch",
+      "developerExamples"
+    ],
+    "requiredBundles": [
+      "kibanaReact"
+    ]
+  }
+}

examples/controls_example/kibana.jsonc

@@ -0,0 +1,18 @@
+{
+  "type": "plugin",
+  "id": "@kbn/controls-example-plugin",
+  "owner": "@elastic/kibana-presentation",
+  "plugin": {
+    "id": "controlsExample",
+    "server": false,
+    "browser": true,
+    "requiredPlugins": [
+      "controls",
+      "data",
+      "developerExamples",
+      "embeddable",
+      "navigation",
+      "presentationUtil"
+    ]
+  }
+}

examples/dashboard_embeddable_examples/kibana.jsonc

@@ -0,0 +1,18 @@
+{
+  "type": "plugin",
+  "id": "@kbn/dashboard-embeddable-examples-plugin",
+  "owner": "@elastic/kibana-presentation",
+  "description": "Example app that shows how to embed a dashboard in an application",
+  "plugin": {
+    "id": "dashboardEmbeddableExamples",
+    "server": false,
+    "browser": true,
+    "requiredPlugins": [
+      "embeddable",
+      "embeddableExamples",
+      "dashboard",
+      "developerExamples",
+      "kibanaReact"
+    ]
+  }
+}

examples/data_view_field_editor_example/kibana.jsonc

@@ -0,0 +1,16 @@
+{
+  "type": "plugin",
+  "id": "@kbn/data-view-field-editor-example-plugin",
+  "owner": "@elastic/kibana-app-services",
+  "description": "Data view field editor example app",
+  "plugin": {
+    "id": "dataViewFieldEditorExample",
+    "server": false,
+    "browser": true,
+    "requiredPlugins": [
+      "data",
+      "dataViewFieldEditor",
+      "developerExamples"
+    ]
+  }
+}

examples/developer_examples/kibana.jsonc

@@ -0,0 +1,10 @@
+{
+  "type": "plugin",
+  "id": "@kbn/developer-examples-plugin",
+  "owner": "@elastic/appex-sharedux",
+  "plugin": {
+    "id": "developerExamples",
+    "server": false,
+    "browser": true
+  }
+}

examples/embeddable_examples/kibana.jsonc

@@ -0,0 +1,26 @@
+{
+  "type": "plugin",
+  "id": "@kbn/embeddable-examples-plugin",
+  "owner": "@elastic/kibana-presentation",
+  "description": "Example app that shows how to register custom embeddables",
+  "plugin": {
+    "id": "embeddableExamples",
+    "server": true,
+    "browser": true,
+    "requiredPlugins": [
+      "embeddable",
+      "uiActions",
+      "savedObjects",
+      "dashboard",
+      "kibanaUtils"
+    ],
+    "requiredBundles": [
+      "kibanaReact"
+    ],
+    "extraPublicDirs": [
+      "public/todo",
+      "public/hello_world",
+      "public/todo/todo_ref_embeddable"
+    ]
+  }
+}