Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Logs onboarding] Removing custom_logs from routes and making them more generic #162706

Merged
merged 7 commits into from
Aug 1, 2023
Merged

[Logs onboarding] Removing custom_logs from routes and making them more generic #162706

merged 7 commits into from
Aug 1, 2023

Conversation

yngrdyn
Copy link
Contributor

@yngrdyn yngrdyn commented Jul 28, 2023 •
edited
Loading

This PR is a preparation work for #154929.

Changes:

  • Rearranged code to make routes more generic (Most of the changes).
  • Added type to onboarding savedObject.
  • Removed status route since it was not being used.

@yngrdyn yngrdyn requested a review from a team as a code owner July 28, 2023 12:30
@yngrdyn yngrdyn added the release_note:skip Skip the PR/issue when compiling release notes label Jul 28, 2023
@apmmachine
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@kibanamachine kibanamachine requested a review from a team as a code owner July 28, 2023 12:48
@yngrdyn yngrdyn changed the title Removing custom_logs from routes and making them more generic [Logs onboarding] Removing custom_logs from routes and making them more generic Jul 28, 2023
@yngrdyn yngrdyn force-pushed the 154929-logs-onboarding-system-logs-refactor-server-routes branch 3 times, most recently from 33e9a52 to 936413d Compare July 28, 2023 17:25
@yngrdyn yngrdyn force-pushed the 154929-logs-onboarding-system-logs-refactor-server-routes branch from 4ca872e to ae48838 Compare July 31, 2023 08:59
@yngrdyn yngrdyn force-pushed the 154929-logs-onboarding-system-logs-refactor-server-routes branch from e78cfcb to 83aaaa6 Compare July 31, 2023 11:04
ogupte
ogupte approved these changes Jul 31, 2023
View reviewed changes
Copy link
Contributor

@ogupte ogupte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, a few suggestions, but non blocking.

packages/kbn-check-mappings-update-cli/current_mappings.json
@@ -2249,6 +2249,9 @@
},
"observability-onboarding-state": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to rename this observability-onboarding-state to observability-onboarding-flow to match the new pattern for routes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to rename the savedObject but got the following error
Removing mapped properties is disallowed >8.8

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok...well it's more of a cosmetic thing i guess. i figured now would be the time since it's technically still not released yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not released to serverless yet but I believe modifying the name of a saved object will trigger a more expensive migration that affects customers. This is mainly a problem for heavy users of cases and/or alerts. CC @gsoldevila

That being said, I think it'd be good to get some "practice" of what it'd be like to be running a live service so I agree with the decision to rather forgo the cosmetic change :)

x-pack/plugins/observability_onboarding/server/routes/flow/route.ts Outdated Show resolved Hide resolved
@yngrdyn yngrdyn force-pushed the 154929-logs-onboarding-system-logs-refactor-server-routes branch 3 times, most recently from e7d4878 to 162ae28 Compare July 31, 2023 15:56
@yngrdyn yngrdyn force-pushed the 154929-logs-onboarding-system-logs-refactor-server-routes branch from 162ae28 to b02287b Compare July 31, 2023 16:34
jloleysens
jloleysens approved these changes Aug 1, 2023
View reviewed changes
packages/kbn-check-mappings-update-cli/current_mappings.json
@@ -2249,6 +2249,9 @@
},
"observability-onboarding-state": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not released to serverless yet but I believe modifying the name of a saved object will trigger a more expensive migration that affects customers. This is mainly a problem for heavy users of cases and/or alerts. CC @gsoldevila

That being said, I think it'd be good to get some "practice" of what it'd be like to be running a live service so I agree with the decision to rather forgo the cosmetic change :)

achyutjhunjhunwala
achyutjhunjhunwala reviewed Aug 1, 2023
View reviewed changes
...plugins/observability_onboarding/public/components/app/custom_logs/wizard/api_key_banner.tsx Outdated
Copy link
Contributor

@achyutjhunjhunwala achyutjhunjhunwala Aug 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) you are renaming the APIs from custom_logs to logs, what about the folder names ? Do you still want to keep them as custom_logs ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in the server we identified the reusability of the endpoints specifically from the elastic-agent but in the front the separation is still needed. In the server most of the generic endpoints (the one used by the agent) were moved to route flow while the specifics to logs remain in logs/*.

achyutjhunjhunwala
achyutjhunjhunwala approved these changes Aug 1, 2023
View reviewed changes
Copy link
Contributor

@achyutjhunjhunwala achyutjhunjhunwala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
observabilityOnboarding 101.6KB 101.5KB -38.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id before after diff
observability-onboarding-state - 2 +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@yngrdyn yngrdyn merged commit 378cd6f into elastic:main Aug 1, 2023
@kibanamachine kibanamachine added v8.10.0 backport:skip This commit does not require backporting labels Aug 1, 2023
@yngrdyn yngrdyn linked an issue Aug 1, 2023 that may be closed by this pull request
[Logs Onboarding] System logs onboarding flow #154929
Closed
ThomThomson pushed a commit to ThomThomson/kibana that referenced this pull request Aug 1, 2023
@yngrdyn
[Logs onboarding] Removing custom_logs from routes and making them mo…
69dd0cc 
…re generic (elastic#162706)

This PR is a preparation work for
elastic#154929.

### Changes:
- Rearranged code to make routes more generic (Most of the changes).
- Added `type` to onboarding savedObject.
- Removed `status` route since it was not being used.
@yngrdyn yngrdyn mentioned this pull request Aug 2, 2023
Merged
yngrdyn added a commit that referenced this pull request Aug 4, 2023
@yngrdyn
[Logs onboarding] Generate elastic-agent.yml file for system logs (#1…
c54d297 
…62972)

Closes #154929.

This PR along with #162654,
#162706 and
#162600 completes the work
required for collect system logs.

### Changes
- `ObservabilityOnboardingType` now could be `logFiles | systemLogs`.
This help us to identify (without changing the script) whether we need
to retrieve the yaml configuration for customLogs or for systemLogs.
- Added `generateSystemLogsYml` which generates a specific configuration
for system logs.
- `get_has_logs.ts` was modified so we are querying the proper index
depending on the type of logs.

#### Demo


https://github.com/elastic/kibana/assets/1313018/47eca890-37b2-401e-9e41-67c978ab50ad
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogupte ogupte ogupte approved these changes

@achyutjhunjhunwala achyutjhunjhunwala achyutjhunjhunwala approved these changes

@jloleysens jloleysens jloleysens approved these changes

Assignees
No one assigned
Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes v8.10.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Logs Onboarding] System logs onboarding flow
7 participants
@yngrdyn @apmmachine @kibana-ci @ogupte @achyutjhunjhunwala @jloleysens @kibanamachine