[Fleet] Move integrations to a separate app #99848

kpollich · 2021-05-11T20:51:23Z

This is the initial work required to move the Integrations UI out into its own Fleet app. The goal here is to allow us to make room for substantially expanding and rethinking what Integrations are and what they do.

A large chunk of this work came from referencing @jen-huang's POC branch from a few months ago: https://github.com/jen-huang/kibana/tree/poc/integrations-ui.

This commit moves the sections/epm directory from the Fleet application to a new Integrations application, still within the Fleet codebase. This should be considered the first step, and it's likely that we'll want to consider moving Integrations to its own plugin entirely at a later date.

This PR addresses the following tasks in the parent tracking issue:

Move Integrations section of Fleet app to new Integrations app, linked under Management in Kibana nav
Adjust routing, breadcrumbs, and related url/routing code
Adjust Integrations layout to have 2 top-level tabs: Browse, Manage
Adjust Fleet layout to have 4 top-level tabs: Agents, Agent policies, Enrollment tokens, Data streams (Assuming this one is outdated - the new tabs are Overview, Policies, Agents, Data Streams)

Screenshot Comparison

Screen	Main Branch	PR	Notes
Fleet Home			Integrations removed from navbar, "View integrations" links to Integrations app
Integrations Browse			Moved to Integrations App
Integrations Installed			Moved to Integrations App
Integration Details - Nginx			Moved to Integrations App, breadcrumbs use package name
Add Integrations - Nginx			Remains in Fleet app. Breadcrumbs weren't easily preserved. May need a follow up to address
Kibana Sidebar			Added Integrations link above Fleet per designs

kpollich · 2021-05-14T12:54:03Z

Edit: Fixed the tests mentioned here

~~These tests are now fixed~~

Blocked on getting some tests to pass here now that they've moved:

x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/index.test.tsx

Tests for the "policies tab" are failing, I think because of a routing issue where the <PackagePoliciesPage /> component isn't being rendered.

when on integration detail › and on the Policies Tab › should display policies list

    TestingLibraryElementError: Unable to find an element by: [data-test-subj="integrationPolicyTable"]

    <body>
      <div />
    </body>

      205 |     it('should display policies list', async () => {
      206 |       await mockedApi.waitForApi();
    > 207 |       const table = renderResult.getByTestId('integrationPolicyTable');
          |                                  ^
      208 |       expect(table).not.toBeNull();
      209 |     });
      210 |

      at Object.getElementError (node_modules/@testing-library/dom/dist/config.js:37:19)
      at node_modules/@testing-library/dom/dist/query-helpers.js:90:38
      at node_modules/@testing-library/dom/dist/query-helpers.js:62:17
      at getByTestId (node_modules/@testing-library/dom/dist/query-helpers.js:111:19)
      at Object.<anonymous> (x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/index.test.tsx:207:34)

  ● when on integration detail › and on the Policies Tab › should link to integration policy detail when an integration policy is clicked

    TestingLibraryElementError: Unable to find an element by: [data-test-subj="integrationNameLink"]

    <body>
      <div />
    </body>

      211 |     it('should link to integration policy detail when an integration policy is clicked', async () => {
      212 |       await mockedApi.waitForApi();
    > 213 |       const firstPolicy = renderResult.getAllByTestId(
          |                                        ^
      214 |         'integrationNameLink'
      215 |       )[0] as HTMLAnchorElement;
      216 |       expect(firstPolicy.href).toEqual(

      at Object.getElementError (node_modules/@testing-library/dom/dist/config.js:37:19)
      at node_modules/@testing-library/dom/dist/query-helpers.js:90:38
      at getAllByTestId (node_modules/@testing-library/dom/dist/query-helpers.js:130:15)
      at Object.<anonymous> (x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/index.test.tsx:213:40)

  ● when on integration detail › and on the Policies Tab › should NOT show link for agent count if it is zero

    TestingLibraryElementError: Unable to find an element by: [data-test-subj="rowAgentCount"]

    <body>
      <div />
    </body>

      221 |     it('should NOT show link for agent count if it is zero', async () => {
      222 |       await mockedApi.waitForApi();
    > 223 |       const firstRowAgentCount = renderResult.getAllByTestId('rowAgentCount')[0];
          |                                               ^
      224 |       expect(firstRowAgentCount.textContent).toEqual('0');
      225 |       expect(firstRowAgentCount.tagName).not.toEqual('A');
      226 |     });

      at Object.getElementError (node_modules/@testing-library/dom/dist/config.js:37:19)
      at node_modules/@testing-library/dom/dist/query-helpers.js:90:38
      at getAllByTestId (node_modules/@testing-library/dom/dist/query-helpers.js:130:15)
      at Object.<anonymous> (x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/index.test.tsx:223:47)

  ● when on integration detail › and on the Policies Tab › should show link for agent count if greater than zero

    TestingLibraryElementError: Unable to find an element by: [data-test-subj="rowAgentCount"]

    <body>
      <div />
    </body>

      228 |     it('should show link for agent count if greater than zero', async () => {
      229 |       await mockedApi.waitForApi();
    > 230 |       const secondRowAgentCount = renderResult.getAllByTestId('rowAgentCount')[1];
          |                                                ^
      231 |       expect(secondRowAgentCount.textContent).toEqual('100');
      232 |       expect(secondRowAgentCount.tagName).toEqual('A');
      233 |     });

      at Object.getElementError (node_modules/@testing-library/dom/dist/config.js:37:19)
      at node_modules/@testing-library/dom/dist/query-helpers.js:90:38
      at getAllByTestId (node_modules/@testing-library/dom/dist/query-helpers.js:130:15)
      at Object.<anonymous> (x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/index.test.tsx:230:48)

There are also a ton of React Testing Library warnings around act calls for state updates that I'm not sure how to resolve. The errors aren't super clear, but they're repeated for almost every test:

console.error
    Warning: An update to FleetStatusProvider inside a test was not wrapped in act(...).

    When testing, code that causes React state updates should be wrapped into act(...):

    act(() => {
      /* fire events that update state */
    });
    /* assert on the output */

    This ensures that you're testing the behavior the user would see in the browser. Learn more at https://fb.me/react-wrap-tests-with-act
        in FleetStatusProvider
        in ThemeProvider (created by EuiThemeProvider)
        in EuiThemeProvider
        in EuiErrorBoundary
        in Provider
        in PseudoLocaleWrapper (created by I18nProvider)
        in IntlProvider (created by I18nProvider)
        in I18nProvider (created by mockConstructor)
        in mockConstructor
        in Unknown
        in Unknown

      42 |       }
      43 |
    > 44 |       setState((s) => ({
         |       ^
      45 |         ...s,
      46 |         isLoading: false,
      47 |         isReady: res.data?.isReady ?? false,

      at warningWithoutStack (node_modules/react-dom/cjs/react-dom.development.js:530:32)
      at warnIfNotCurrentlyActingUpdatesInDEV (node_modules/react-dom/cjs/react-dom.development.js:25848:7)
      at setState (node_modules/react-dom/cjs/react-dom.development.js:17072:9)
      at sendGetStatus (x-pack/plugins/fleet/public/hooks/use_fleet_status.tsx:44:7)

elasticmachine · 2021-06-01T15:13:08Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2021-06-01T15:36:12Z

Edit: This was resolved in a310e4a

Getting an odd test failure that I could use a second set of eyes on in the latest here (appears in a few tests) - seems like some kind of issue with the context components. For some reason this UIExtensionsContext object is coming back undefined in tests. Other similar providers like KibanaVersionContext or ConfigContext work as expected. Issue seems unique to UIExtensionsContext.

when on the package policy create page › and Route state is provided via Fleet HashRouter › and the cancel Link or Button is clicked › should redirect via Fleet HashRouter when cancel Button (button bar) is clicked

    TypeError: Cannot read property 'Provider' of undefined

      213 |               <KibanaVersionContext.Provider value={kibanaVersion}>
      214 |                 <EuiThemeProvider darkMode={isDarkMode}>
    > 215 |                   <UIExtensionsContext.Provider value={extensions}>
          |                                        ^
      216 |                     <FleetStatusProvider>
      217 |                       <IntraAppStateProvider kibanaScopedHistory={history}>
      218 |                         <Router history={routerHistoryInstance}>

nchaulet · 2021-06-01T17:19:45Z

x-pack/plugins/fleet/common/constants/plugin.ts

@@ -6,3 +6,4 @@
 */

 export const PLUGIN_ID = 'fleet';
+export const INTEGRATIONS_PLUGIN_ID = 'integrations';


Do we plan to move integrations to a different plugin? or it will always be an app inside the fleet plugin?

I think the plan would be to eventually move it to a separate plugin entirely.

later on it may become separate, but that's not firmly decided at this point

jloleysens

Great work so far @kpollich !

I left a few comments in the code, not any blockers for now.

I tested locally and saw the following behaviour when view an integration package:

It looks like this is one of the markdown renderers (https://github.com/kpollich/kibana/blob/b174e639dab53190c1393f1f7519cd9b259ac46d/x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/overview/markdown_renderers.tsx#L62) receiving an unknown language $json. Not sure whether this is a known problem?

x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/components/index.ts

x-pack/plugins/fleet/public/applications/integrations/app.tsx

jloleysens · 2021-06-02T10:55:23Z

Getting an odd test failure that I could use a second set of eyes on in the latest here (appears in a few tests) - seems like some kind of issue with the context components. For some reason this UIExtensionsContext object is coming back undefined

This is peculiar, I tried running this test and was not able to reproduce it locally.

kpollich · 2021-06-02T11:50:09Z

This is peculiar, I tried running this test and was not able to reproduce it locally.

My mistake - this was resolved late in the day yesterday over Slack in a310e4a and I didn't update the initial comment.

kpollich · 2021-06-02T14:49:00Z

Great work so far @kpollich !

I left a few comments in the code, not any blockers for now.

I tested locally and saw the following behaviour when view an integration package:

It looks like this is one of the markdown renderers (https://github.com/kpollich/kibana/blob/b174e639dab53190c1393f1f7519cd9b259ac46d/x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/detail/overview/markdown_renderers.tsx#L62) receiving an unknown language $json. Not sure whether this is a known problem?

@jloleysens looks like this a known issue as of a few minutes ago. Something upstream. #101168

kpollich · 2021-06-04T13:05:27Z

@jen-huang I've added screenshots + notes for each major UI change to the PR description as requested.

jbudz

limits.yml LGTM

kpollich · 2021-06-04T14:15:04Z

@elasticmachine merge upstream

hbharding · 2021-06-04T14:34:01Z

Thanks @kpollich this is a great start and I appreciate the before/after screenshots. I haven't tested this locally, but I noticed a few layout + minor details in looking at the screenshots. I'll summarize here, and i've included annotated screenshots below for more detail. Also, just in case, here's the link to the Figma designs

Breadcrumbs on Integrations home page should match the tabs, "Browse" and "Manage". Currently they say "All" and "Installed"
The left/right margins in the content area (below header) should be white. Currently it appears to be using EuiPageBackgroundColor.
Remove the double/nested header on the Integration detail page (Nginx). The header with the graphic should only appear on the Integrations home page.
Remove Fleet's top bar navigation on the "Add Integration" page.
On the "Add Integration" page, the root breadcrumb should say Integrations rather than Fleet since this is the place users started from.

Annotated screenshots

kpollich · 2021-06-04T15:51:41Z

Thanks @kpollich this is a great start and I appreciate the before/after screenshots. I haven't tested this locally, but I noticed a few layout + minor details in looking at the screenshots. I'll summarize here, and i've included annotated screenshots below for more detail. Also, just in case, here's the link to the Figma designs

Breadcrumbs on Integrations home page should match the tabs, "Browse" and "Manage". Currently they say "All" and "Installed"

The left/right margins in the content area (below header) should be white. Currently it appears to be using EuiPageBackgroundColor.

Remove the double/nested header on the Integration detail page (Nginx). The header with the graphic should only appear on the Integrations home page.

Remove Fleet's top bar navigation on the "Add Integration" page.

On the "Add Integration" page, the root breadcrumb should say Integrations rather than Fleet since this is the place users started from.

Annotated screenshots

Thanks so much for this, @hbharding - extremely helpful and actionable feedback. I've just pushed up fixes for everything here, and I'm including screenshots of the updates below. Thanks again!

Screenshots

kibanamachine · 2021-06-07T14:32:28Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	458	472	+14

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	980	981	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	747.7KB	713.7KB	-34.0KB
`securitySolution`	6.9MB	6.9MB	+6.0B
total			-33.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	347.5KB	425.1KB	+77.6KB

Unknown metric groups

API count

id	before	after	diff
`fleet`	1070	1071	+1

async chunk count

id	before	after	diff
`fleet`	5	6	+1

History

💔 Build #129452 failed 20ff389
💚 Build #129295 succeeded 2e47795
💔 Build #129273 failed a361845
💚 Build #129241 succeeded 457f450
💔 Build #129233 failed ae06c8c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

kibanamachine · 2021-06-07T14:46:21Z

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 99848

* WIP: Re-create separation of integrations app * Fix breadcrumbs * Fix add integration button/routing * Fix integrations test paths * Fix failing policy tab tests * Fix type errors * Fix more type errors * Fix integrations home page redirect * Fix circular import * Fix i18n errors * Fix FTR paths * Fix more deep fleet import paths * Remove unneeded state set * Fix more type errors * Fix failing security_solutions tests * Address redirect back path todo * Fix page path in FTR * Fix type error * 🤞 Fix FTR failures * Fix package details path in endpoint tests * Fix test import * Fix add integration route + breadcrumbs * Fix missing layout for create package policy page * Fixup Kibana feature declaration + fix app registry arrays * Fix Kibana startup error from feature registration * Fix telemetry schema * Remove integrations from privilege tests * Increase Fleet bundle limit by 20kb to fix CI error * Use correct updated Fleet bundle limit * Update limits.yml via script * Address design feedback * Fix i18n error * Fix failing security solution tests * Fix type error Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # packages/kbn-optimizer/limits.yml

* master: (90 commits) Fix UI breaks on providing long search keyword in 'Search Box' (elastic#101385) Adds css class to EuiDescriptionListDescription in order to break word on exception details card (elastic#101481) [Lens] Increase timings for drag and drop tests (elastic#101380) [Lens] Fix editor react error on configuration panel (elastic#101367) [Fleet] Move integrations to a separate app (elastic#99848) Fix incorrect message displayed on importing Timeline Templates (elastic#101288) [Cases] RBAC (elastic#95058) [APM] Visual improvements for new APM layout with left navigation (elastic#101360) [master] More precise alerts matching (elastic#99820) [Lens] Value in legend (elastic#101353) Revert "[Reporting] ILM policy for managing reporting indices (elastic#100130)" (elastic#101358) [Discover] Fix header row of data grid in Firefox (elastic#101374) Add link to advanced setting in Discover (elastic#101154) Url service locators (elastic#101045) [Timelion] Update the removal message to mention the exact version (elastic#100994) [Security Solution][Detection Engine] Test cases for alias failure test cases where we don't copy aliases correctly (elastic#101437) [Event Log] Adding `type_id` to saved object array in event log (elastic#100939) [Reporting] Add `location.url` info to console message logs (elastic#101427) [Security Solutions][Detection Engine] Fixes timestamp bugs within source indexes when the formats are not ISO8601 format (elastic#101349) Improve Task Manager instrumentation (elastic#99160) ...

* WIP: Re-create separation of integrations app * Fix breadcrumbs * Fix add integration button/routing * Fix integrations test paths * Fix failing policy tab tests * Fix type errors * Fix more type errors * Fix integrations home page redirect * Fix circular import * Fix i18n errors * Fix FTR paths * Fix more deep fleet import paths * Remove unneeded state set * Fix more type errors * Fix failing security_solutions tests * Address redirect back path todo * Fix page path in FTR * Fix type error * 🤞 Fix FTR failures * Fix package details path in endpoint tests * Fix test import * Fix add integration route + breadcrumbs * Fix missing layout for create package policy page * Fixup Kibana feature declaration + fix app registry arrays * Fix Kibana startup error from feature registration * Fix telemetry schema * Remove integrations from privilege tests * Increase Fleet bundle limit by 20kb to fix CI error * Use correct updated Fleet bundle limit * Update limits.yml via script * Address design feedback * Fix i18n error * Fix failing security solution tests * Fix type error Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # packages/kbn-optimizer/limits.yml

kpollich added v8.0.0 Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 auto-backport Deprecated: Automatically backport this PR after it's merged labels May 11, 2021

kpollich mentioned this pull request May 13, 2021

[Fleet] Support browsing granular integrations #99866

Merged

kpollich requested review from jen-huang, nchaulet and jfsiii May 14, 2021 12:51

kpollich mentioned this pull request May 25, 2021

[Fleet] Improve Egonomics and Clarity for Imports #100566

Open

kpollich force-pushed the integrations-separate-app branch 2 times, most recently from e18ca4b to d48b017 Compare May 31, 2021 23:08

kpollich self-assigned this Jun 1, 2021

kpollich force-pushed the integrations-separate-app branch from d48b017 to b67d3d4 Compare June 1, 2021 12:12

jloleysens self-requested a review June 1, 2021 15:04

kpollich marked this pull request as ready for review June 1, 2021 15:13

kpollich requested a review from a team as a code owner June 1, 2021 15:13

kpollich changed the title ~~[Fleet] [WIP] Move integrations to a separate app~~ [Fleet] Move integrations to a separate app Jun 1, 2021

nchaulet reviewed Jun 1, 2021

View reviewed changes

jen-huang added the release_note:feature Makes this part of the condensed release notes label Jun 1, 2021

kpollich requested a review from a team as a code owner June 1, 2021 20:30

jloleysens reviewed Jun 2, 2021

View reviewed changes

nchaulet self-requested a review June 2, 2021 14:27

paul-tavares self-requested a review June 2, 2021 15:13

legrego self-requested a review June 2, 2021 20:01

kpollich added 4 commits June 4, 2021 08:13

Fix Kibana startup error from feature registration

056fb68

Fix telemetry schema

5a82b22

Remove integrations from privilege tests

9398073

Increase Fleet bundle limit by 20kb to fix CI error

4bc15a6

kpollich force-pushed the integrations-separate-app branch from fb0ecc2 to 4bc15a6 Compare June 4, 2021 12:14

jbudz approved these changes Jun 4, 2021

View reviewed changes

kibanamachine and others added 2 commits June 4, 2021 10:15

Merge branch 'master' into integrations-separate-app

44b2b9d

Use correct updated Fleet bundle limit

e51cbf9

kpollich added 2 commits June 4, 2021 10:54

Update limits.yml via script

da285a6

Address design feedback

ae06c8c

kpollich added 2 commits June 4, 2021 12:00

Fix i18n error

457f450

Merge branch 'master' into integrations-separate-app

a361845

kpollich requested a review from legrego June 4, 2021 18:14

legrego approved these changes Jun 4, 2021

View reviewed changes

kpollich added 3 commits June 4, 2021 15:13

Fix failing security solution tests

2e47795

Merge branch 'master' into integrations-separate-app

20ff389

Fix type error

a258250

kpollich merged commit dc1d98b into elastic:master Jun 7, 2021

kpollich deleted the integrations-separate-app branch June 7, 2021 14:44

kpollich mentioned this pull request Jun 7, 2021

[7.x] [Fleet] Move integrations to a separate app (#99848) #101493

Merged

kpollich mentioned this pull request Jun 15, 2021

[Fleet + Integrations UI] Address UI Regressions in Fleet/Integrations #102250

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Move integrations to a separate app #99848

[Fleet] Move integrations to a separate app #99848

kpollich commented May 11, 2021 •

edited

Loading

kpollich commented May 14, 2021 •

edited

Loading

elasticmachine commented Jun 1, 2021

kpollich commented Jun 1, 2021 •

edited

Loading

nchaulet Jun 1, 2021

kpollich Jun 1, 2021

jen-huang Jun 1, 2021

jloleysens left a comment •

edited

Loading

jloleysens commented Jun 2, 2021

kpollich commented Jun 2, 2021

kpollich commented Jun 2, 2021

kpollich commented Jun 4, 2021

jbudz left a comment

kpollich commented Jun 4, 2021

hbharding commented Jun 4, 2021

kpollich commented Jun 4, 2021

kibanamachine commented Jun 7, 2021

API count

async chunk count

kibanamachine commented Jun 7, 2021

[Fleet] Move integrations to a separate app #99848

[Fleet] Move integrations to a separate app #99848

Conversation

kpollich commented May 11, 2021 • edited Loading

Screenshot Comparison

kpollich commented May 14, 2021 • edited Loading

elasticmachine commented Jun 1, 2021

kpollich commented Jun 1, 2021 • edited Loading

nchaulet Jun 1, 2021

Choose a reason for hiding this comment

kpollich Jun 1, 2021

Choose a reason for hiding this comment

jen-huang Jun 1, 2021

Choose a reason for hiding this comment

jloleysens left a comment • edited Loading

Choose a reason for hiding this comment

jloleysens commented Jun 2, 2021

kpollich commented Jun 2, 2021

kpollich commented Jun 2, 2021

kpollich commented Jun 4, 2021

jbudz left a comment

Choose a reason for hiding this comment

kpollich commented Jun 4, 2021

hbharding commented Jun 4, 2021

kpollich commented Jun 4, 2021

kibanamachine commented Jun 7, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

async chunk count

History

kibanamachine commented Jun 7, 2021

💔 Backport failed

kpollich commented May 11, 2021 •

edited

Loading

kpollich commented May 14, 2021 •

edited

Loading

kpollich commented Jun 1, 2021 •

edited

Loading

jloleysens left a comment •

edited

Loading