-
Notifications
You must be signed in to change notification settings - Fork 8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fleet] Provide license info in telemetry #182150
[Fleet] Provide license info in telemetry #182150
Conversation
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
/ci |
@elasticmachine merge upstream |
/ci |
@elasticmachine merge upstream |
Pinging @elastic/fleet (Team:Fleet) |
|
||
await this.send( | ||
events, | ||
eventsWithLicenseInfo, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this sender is only used for sending upgrade events, we should probably add the license info to FleetUsageSender too.
You can see that logged as
[2024-05-01T15:13:14.420+02:00][DEBUG][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":1,"healthy":0,"unhealthy":0,"offline":1,"inactive":0,"unenrolled":2,"total_all_statuses":3,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":0,"unhealthy":0,"offline":1,"updating":0,"total_all_statuses":1,"num_host_urls":1}}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey, can I ask for a clarification?
Is it correct that the FleetUsageSender sends telemetry data through e.g.
core.analytics.reportEvent(FLEET_USAGES_EVENT_TYPE, fleetUsageData); |
Just wondering, as the
TelemetryEventsSender
uses a simple POST request to the telemetry URL.
Edit: think I answered my own question, will push change :)
Is it possible to add “Organization ID” and “Deployment ID” too as requested here? https://github.com/elastic/ingest-dev/issues/2866#issuecomment-2014029639 |
Hey @juliaElastic I've been struggling to get those via the internal clients, know if I might be missing anything? I've provisionally added the cluster info to the usage sender. Also, I've checked for 3 things as detailed in the description (fleet usage sender, fleet usage logger, upgrade sender). I'm puzzled why changing the interval in |
I'll take a look.
I think you have to increase the task version to let kibana pick up the change:
When I'm running your pr locally, I'm seeing cluster_info and license_info logged out, you could test in cloud by adding the
You also have to add new fields to the fleet_usages_schema.ts |
Thank you, it works! Should we increase the version for this change? |
Yes. |
I found a way to add kibana/x-pack/plugins/fleet/server/services/preconfiguration/fleet_server_host.ts Lines 31 to 40 in 75c7f11
I didn't find organization id anywhere in kibana code. I think we probably don't need example
There are a few more fields in this CloudSetup type that might be useful, e.g. |
@elasticmachine merge upstream |
thank you @jillguyonnet and @juliaElastic for your efforts on this. If deploymentID is available in the telemetry, for ESs customers, the only way I know how to get to the cluster is by looking up the deploymentID via admin.found.no. License issued to (I think already exists in our telemetry) is less reliable unfortunately. It may resolve to "cloud" in cases where the license was issued via API (which seems often). I will follow up over email with some other information that may help. |
@nimarezainia In the context of the recent discussions, would there be any value in adding any of the already available fields? e.g. |
Yes please add those and we can perhaps call this issue closed for now. I believe that the deploymentID in particular at least would give us the opportunity to correctly identify the user/customer and take it from there. Thanks you again. |
@elasticmachine merge upstream |
Tested this on a cloud deployment:
and from a package install:
Does |
@jillguyonnet the deploymentID maps to cluster: kibana-pr-182150 does this sound right to you? @jlind23 I'm wondering if you see it differently. |
@nimarezainia I found the same result. FWIW it is an internal deployment and a non paying organization under a |
@nimarezainia |
From my perspective, yes. |
@elasticmachine merge upstream |
@@ -61,6 +63,8 @@ export const fetchFleetUsage = async ( | |||
...(await getPanicLogsLastHour(esClient)), | |||
...(await getAgentLogsTopErrors(esClient)), | |||
agents_per_output_type: await getAgentsPerOutput(soClient, esClient), | |||
license_issued_to: (await esClient.license.get()).license.issued_to, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: would it make sense to use the same fetchLicenseInfo
function to get the license info and catch errors?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
@jillguyonnet I want to backport #184056 to 8.14 branch so I'm going to backport this one first to avoid conflicts. The more telemetry the better :) |
### Summary Relates elastic/ingest-dev#2866 This PR adds license information to Fleet telemetry events stored in the `fleet-usages*` index. The `license. issued_to` field provides the customer name. ### Testing locally In a local setup, only the new `license_issued_to` field is defined. Fleet usage logger: 1. Change the [interval](https://github.com/elastic/kibana/blob/de755234957b8df5bb773587150014b076fe5551/x-pack/plugins/fleet/server/services/fleet_usage_logger.ts#L59C46-L59C48) of the `FleetUsageLogger` to e.g. `1m`. 2. Wait for the log line to show up: ``` Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":3,"healthy":3,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":3,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":1,"unhealthy":0,"offline":0,"updating":0,"total_all_statuses":1,"num_host_urls":1},"license_issued_to":"elasticsearch"} ``` Fleet usage sender: 1. Change the interval of the `FleetUsageSender` to e.g. `1m`. 2. Wait for the following log line: ``` [2024-05-03T15:24:11.843+02:00][DEBUG][plugins.fleet] Fleet usage telemetry: {"agents_enabled":true,"agents":{"total_enrolled":3,"healthy":3,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":3,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":1,"unhealthy":0,"offline":0,"updating":0,"total_all_statuses":1,"num_host_urls":1},"packages":[{"name":"system","version":"1.55.2","enabled":true},{"name":"synthetics","version":"1.2.1","enabled":false},{"name":"fleet_server","version":"1.5.0","enabled":true},{"name":"elastic_agent","version":"1.18.0","enabled":false},{"name":"nginx","version":"1.20.0","enabled":false}],"agent_checkin_status":{"error":0,"degraded":0},"agents_per_policy":[2,1],"agents_per_os":[{"name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","count":3}],"fleet_server_config":{"policies":[{"input_config":{}}]},"agent_policies":{"count":3,"output_types":["elasticsearch"]},"agent_logs_panics_last_hour":[],"agent_logs_top_errors":[],"fleet_server_logs_top_errors":[],"license_issued_to":"elasticsearch"} ``` Upgrade sender: 1. Change the [interval](https://github.com/elastic/kibana/blob/b8d8c737e6cc7889c19a6e7984d618bf378ee617/x-pack/plugins/fleet/server/services/telemetry/fleet_usage_sender.ts#L30) of the `FleetUsageSender` to e.g. `1m`. 4. (Re)install a package and wait for the telemetry log (DEBUG level), which should contain license information: ``` [2024-05-03T15:20:41.087+02:00][DEBUG][plugins.fleet.telemetry_events] [{"packageName":"nginx","currentVersion":"1.20.0","newVersion":"1.20.0","status":"success","dryRun":false,"eventType":"package-install","installType":"reinstall","errorMessage":[],"license_issued_to":"elasticsearch"}] ``` ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com> (cherry picked from commit 755a37d)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
# Backport This will backport the following commits from `main` to `8.14`: - [[Fleet] Provide license info in telemetry (#182150)](#182150) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jill Guyonnet","email":"jill.guyonnet@elastic.co"},"sourceCommit":{"committedDate":"2024-05-15T13:05:39Z","message":"[Fleet] Provide license info in telemetry (#182150)\n\n### Summary\r\n\r\nRelates elastic/ingest-dev#2866 PR adds license information to Fleet telemetry events stored in the\r\n`fleet-usages*` index. The `license. issued_to` field provides the\r\ncustomer name.\r\n\r\n### Testing locally\r\n\r\nIn a local setup, only the new `license_issued_to` field is defined.\r\n\r\nFleet usage logger:\r\n1. Change the\r\n[interval](https://github.com/elastic/kibana/blob/de755234957b8df5bb773587150014b076fe5551/x-pack/plugins/fleet/server/services/fleet_usage_logger.ts#L59C46-L59C48)\r\nof the `FleetUsageLogger` to e.g. `1m`.\r\n2. Wait for the log line to show up:\r\n```\r\nFleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":3,\"healthy\":3,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":3,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"total_all_statuses\":1,\"num_host_urls\":1},\"license_issued_to\":\"elasticsearch\"}\r\n```\r\n\r\nFleet usage sender:\r\n1. Change the interval of the `FleetUsageSender` to e.g. `1m`.\r\n2. Wait for the following log line:\r\n```\r\n[2024-05-03T15:24:11.843+02:00][DEBUG][plugins.fleet] Fleet usage telemetry: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":3,\"healthy\":3,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":3,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"total_all_statuses\":1,\"num_host_urls\":1},\"packages\":[{\"name\":\"system\",\"version\":\"1.55.2\",\"enabled\":true},{\"name\":\"synthetics\",\"version\":\"1.2.1\",\"enabled\":false},{\"name\":\"fleet_server\",\"version\":\"1.5.0\",\"enabled\":true},{\"name\":\"elastic_agent\",\"version\":\"1.18.0\",\"enabled\":false},{\"name\":\"nginx\",\"version\":\"1.20.0\",\"enabled\":false}],\"agent_checkin_status\":{\"error\":0,\"degraded\":0},\"agents_per_policy\":[2,1],\"agents_per_os\":[{\"name\":\"Ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"count\":3}],\"fleet_server_config\":{\"policies\":[{\"input_config\":{}}]},\"agent_policies\":{\"count\":3,\"output_types\":[\"elasticsearch\"]},\"agent_logs_panics_last_hour\":[],\"agent_logs_top_errors\":[],\"fleet_server_logs_top_errors\":[],\"license_issued_to\":\"elasticsearch\"}\r\n```\r\n\r\nUpgrade sender:\r\n1. Change the\r\n[interval](https://github.com/elastic/kibana/blob/b8d8c737e6cc7889c19a6e7984d618bf378ee617/x-pack/plugins/fleet/server/services/telemetry/fleet_usage_sender.ts#L30)\r\nof the `FleetUsageSender` to e.g. `1m`.\r\n4. (Re)install a package and wait for the telemetry log (DEBUG level),\r\nwhich should contain license information:\r\n```\r\n[2024-05-03T15:20:41.087+02:00][DEBUG][plugins.fleet.telemetry_events] [{\"packageName\":\"nginx\",\"currentVersion\":\"1.20.0\",\"newVersion\":\"1.20.0\",\"status\":\"success\",\"dryRun\":false,\"eventType\":\"package-install\",\"installType\":\"reinstall\",\"errorMessage\":[],\"license_issued_to\":\"elasticsearch\"}]\r\n```\r\n\r\n### Checklist\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>","sha":"755a37d25e133a3189c1c349f88b562084445c59","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","backport:prev-minor","v8.14.0","v8.15.0"],"title":"[Fleet] Provide license info in telemetry","number":182150,"url":"#182150 Provide license info in telemetry (#182150)\n\n### Summary\r\n\r\nRelates elastic/ingest-dev#2866 PR adds license information to Fleet telemetry events stored in the\r\n`fleet-usages*` index. The `license. issued_to` field provides the\r\ncustomer name.\r\n\r\n### Testing locally\r\n\r\nIn a local setup, only the new `license_issued_to` field is defined.\r\n\r\nFleet usage logger:\r\n1. Change the\r\n[interval](https://github.com/elastic/kibana/blob/de755234957b8df5bb773587150014b076fe5551/x-pack/plugins/fleet/server/services/fleet_usage_logger.ts#L59C46-L59C48)\r\nof the `FleetUsageLogger` to e.g. `1m`.\r\n2. Wait for the log line to show up:\r\n```\r\nFleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":3,\"healthy\":3,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":3,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"total_all_statuses\":1,\"num_host_urls\":1},\"license_issued_to\":\"elasticsearch\"}\r\n```\r\n\r\nFleet usage sender:\r\n1. Change the interval of the `FleetUsageSender` to e.g. `1m`.\r\n2. Wait for the following log line:\r\n```\r\n[2024-05-03T15:24:11.843+02:00][DEBUG][plugins.fleet] Fleet usage telemetry: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":3,\"healthy\":3,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":3,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"total_all_statuses\":1,\"num_host_urls\":1},\"packages\":[{\"name\":\"system\",\"version\":\"1.55.2\",\"enabled\":true},{\"name\":\"synthetics\",\"version\":\"1.2.1\",\"enabled\":false},{\"name\":\"fleet_server\",\"version\":\"1.5.0\",\"enabled\":true},{\"name\":\"elastic_agent\",\"version\":\"1.18.0\",\"enabled\":false},{\"name\":\"nginx\",\"version\":\"1.20.0\",\"enabled\":false}],\"agent_checkin_status\":{\"error\":0,\"degraded\":0},\"agents_per_policy\":[2,1],\"agents_per_os\":[{\"name\":\"Ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"count\":3}],\"fleet_server_config\":{\"policies\":[{\"input_config\":{}}]},\"agent_policies\":{\"count\":3,\"output_types\":[\"elasticsearch\"]},\"agent_logs_panics_last_hour\":[],\"agent_logs_top_errors\":[],\"fleet_server_logs_top_errors\":[],\"license_issued_to\":\"elasticsearch\"}\r\n```\r\n\r\nUpgrade sender:\r\n1. Change the\r\n[interval](https://github.com/elastic/kibana/blob/b8d8c737e6cc7889c19a6e7984d618bf378ee617/x-pack/plugins/fleet/server/services/telemetry/fleet_usage_sender.ts#L30)\r\nof the `FleetUsageSender` to e.g. `1m`.\r\n4. (Re)install a package and wait for the telemetry log (DEBUG level),\r\nwhich should contain license information:\r\n```\r\n[2024-05-03T15:20:41.087+02:00][DEBUG][plugins.fleet.telemetry_events] [{\"packageName\":\"nginx\",\"currentVersion\":\"1.20.0\",\"newVersion\":\"1.20.0\",\"status\":\"success\",\"dryRun\":false,\"eventType\":\"package-install\",\"installType\":\"reinstall\",\"errorMessage\":[],\"license_issued_to\":\"elasticsearch\"}]\r\n```\r\n\r\n### Checklist\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>","sha":"755a37d25e133a3189c1c349f88b562084445c59"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"#182150 Provide license info in telemetry (#182150)\n\n### Summary\r\n\r\nRelates elastic/ingest-dev#2866 PR adds license information to Fleet telemetry events stored in the\r\n`fleet-usages*` index. The `license. issued_to` field provides the\r\ncustomer name.\r\n\r\n### Testing locally\r\n\r\nIn a local setup, only the new `license_issued_to` field is defined.\r\n\r\nFleet usage logger:\r\n1. Change the\r\n[interval](https://github.com/elastic/kibana/blob/de755234957b8df5bb773587150014b076fe5551/x-pack/plugins/fleet/server/services/fleet_usage_logger.ts#L59C46-L59C48)\r\nof the `FleetUsageLogger` to e.g. `1m`.\r\n2. Wait for the log line to show up:\r\n```\r\nFleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":3,\"healthy\":3,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":3,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"total_all_statuses\":1,\"num_host_urls\":1},\"license_issued_to\":\"elasticsearch\"}\r\n```\r\n\r\nFleet usage sender:\r\n1. Change the interval of the `FleetUsageSender` to e.g. `1m`.\r\n2. Wait for the following log line:\r\n```\r\n[2024-05-03T15:24:11.843+02:00][DEBUG][plugins.fleet] Fleet usage telemetry: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":3,\"healthy\":3,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":3,\"updating\":0},\"fleet_server\":{\"total_enrolled\":1,\"healthy\":1,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"total_all_statuses\":1,\"num_host_urls\":1},\"packages\":[{\"name\":\"system\",\"version\":\"1.55.2\",\"enabled\":true},{\"name\":\"synthetics\",\"version\":\"1.2.1\",\"enabled\":false},{\"name\":\"fleet_server\",\"version\":\"1.5.0\",\"enabled\":true},{\"name\":\"elastic_agent\",\"version\":\"1.18.0\",\"enabled\":false},{\"name\":\"nginx\",\"version\":\"1.20.0\",\"enabled\":false}],\"agent_checkin_status\":{\"error\":0,\"degraded\":0},\"agents_per_policy\":[2,1],\"agents_per_os\":[{\"name\":\"Ubuntu\",\"version\":\"20.04.6 LTS (Focal Fossa)\",\"count\":3}],\"fleet_server_config\":{\"policies\":[{\"input_config\":{}}]},\"agent_policies\":{\"count\":3,\"output_types\":[\"elasticsearch\"]},\"agent_logs_panics_last_hour\":[],\"agent_logs_top_errors\":[],\"fleet_server_logs_top_errors\":[],\"license_issued_to\":\"elasticsearch\"}\r\n```\r\n\r\nUpgrade sender:\r\n1. Change the\r\n[interval](https://github.com/elastic/kibana/blob/b8d8c737e6cc7889c19a6e7984d618bf378ee617/x-pack/plugins/fleet/server/services/telemetry/fleet_usage_sender.ts#L30)\r\nof the `FleetUsageSender` to e.g. `1m`.\r\n4. (Re)install a package and wait for the telemetry log (DEBUG level),\r\nwhich should contain license information:\r\n```\r\n[2024-05-03T15:20:41.087+02:00][DEBUG][plugins.fleet.telemetry_events] [{\"packageName\":\"nginx\",\"currentVersion\":\"1.20.0\",\"newVersion\":\"1.20.0\",\"status\":\"success\",\"dryRun\":false,\"eventType\":\"package-install\",\"installType\":\"reinstall\",\"errorMessage\":[],\"license_issued_to\":\"elasticsearch\"}]\r\n```\r\n\r\n### Checklist\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>","sha":"755a37d25e133a3189c1c349f88b562084445c59"}}]}] BACKPORT--> Co-authored-by: Jill Guyonnet <jill.guyonnet@elastic.co>
Summary
Relates https://github.com/elastic/ingest-dev/issues/2866
This PR adds license information to Fleet telemetry events stored in the
fleet-usages*
index. Thelicense. issued_to
field provides the customer name.Testing locally
In a local setup, only the new
license_issued_to
field is defined.Fleet usage logger:
FleetUsageLogger
to e.g.1m
.Fleet usage sender:
FleetUsageSender
to e.g.1m
.Upgrade sender:
FleetUsageSender
to e.g.1m
.Checklist