Stack Monitoring Tech Debt Plan

[jasonrhodes]

Over the past year, the Stack Monitoring app has endured a number of massive changes, from a completely new team that owns it and works on, the complete removal of Angular, and a massive set of 8.0 backwards-compatible architecture and data collection changes. Because of this, we've decided to spend some focused time on stabilizing the Stack Monitoring app in the near future. To that end, we've prioritized the following work.

Bugs

• [Stack Monitoring] Enterprise search module showing up in Cluster overview with no data #121192
• [Stack monitoring] Timepicker doesn't correctly pick defaults values for refresh interval #126496
• [Stack Monitoring] ES nodes can't be sorted by status #122338
• [Docs] Monitoring settings doesn't mention service account tokens #124576
• [Stack monitoring] Do not expose monitoring.ui.elasticsearch configuration to the browser #123269
• [Stack Monitoring] All nodes get the "master" star on instance listing #126557
• [Stack Monitoring] Use BasePath.publicBaseUrl for links generated by SM rules #127126
• Stack Monitoring rule types failing due to empty buckets #120111
• [Stack Monitoring] Elasticsearch version displays incorrect version #126741
• [Stack Monitoring] Kibana should not report healthy when recent data is missing #126386

More recent bugs, could defer to later epic

• [Stack Monitoring] Mapping for elasticsearch.node.stats.os.cgroup.memory.limit.bytes is incorrect beats#31765
• [Stack Monitoring] Missing remote_cluster_client role is reported as "access denied" #129546
• [Stack Monitoring] Shard Activity for completed recoveries shows N/A on Total Time column #135041

Tech Debt

• [Stack Monitoring] Replace usages of deprecated index pattern service and related types #124108

Typescript

We've made good progress on TS conversion over the past year, but there's still a good chunk to do. Mainly:

• Server has ~50 .js files remaining, mostly in the following directories:
  • /server/routes
• iots implemenation is required if we want the client to share types with the server

Looking at this, I think we should prioritize getting the server 100% TS and looking into iots implementation. We'll save the public folder for another day :)

Note: Here is an example of a recent PR that does some TS conversion in the server directory for routes, for reference.

• [Stack Monitoring] Convert files in server/routes/setup, ui.js to typescript  #117764
• [Stack Monitoring] Convert files in server/routes/logstash to typescript  #117763
• [Stack Monitoring] Convert files in server/routes/kibana to typescript  #117762
• [Stack Monitoring] Convert files in server/routes/elasticsearch_settings to typescript  #117761
• [Stack Monitoring] Convert files in server/routes/elasticsearch to typescript  #117760
• [Stack Monitoring] Convert files in server/routes/beats to typescript  #117757
• [Stack Monitoring] Convert files in server/routes/apm to typescript #117756
• [Stack Monitoring] Convert files in server/routes/enterprise_search to TypeScript #132644
• [Stack Monitoring] Create scoped issues for "All stack monitoring server routes use iots for type-safe validation" #102437

Documentation

Organization

• https://github.com/elastic/observability-dev/issues/2005
  * [x] Terminology document added to the above docs section
  * [x] Up to date architecture diagram for the different collection methods (which endpoints get queried should be covered by module doc issues below)

Documenting our modules

• [Stack Monitoring] Document data stored for ES module #127209
• [Stack Monitoring] Document data stored for Logstash module #127808
• [Stack Monitoring] Document data stored for Kibana module #127807
• [Stack Monitoring] Document data stored for beat (beats, apm-server, fleet) module #128035
• [Stack Monitoring] Document data stored for enterprisesearch module #130520

Enhancements

• [Stack Monitoring] Create a Stack Monitoring health endpoint #127235
• [Stack Monitoring] Add notes about running a kibana-less deployment to cloud_setup.md #127938
• [Stack Monitoring] Update monitoring template indices docs #130221

Knowledge Sessions

These are NOT going to be well-polished presentations, but rather a live working session of going through the steps described in that session's description and thinking aloud for others to witness how we problem solve each of these areas.

We'll record each of these. I expect folks will need to rewatch them a few times before they start to sink in, and we can do similar sessions repeatedly until things make a bit more sense.

• High level overview of the app, focusing on ES + Kibana running locally (Mat)
• Data collection architecture (Jason)
• Overview of the beat modules (Kevin)
• Understanding our own access to customer deployments in cloud, and how to interpret them
• Logstash (Jason)
• APM (Silvia or Stuart?)
• Stack monitoring in cloud (Mat/Jason)

Deprioritized items

Bugs

• "Large shard size" Stack Monitoring rule is missing "Look at the average over X minutes" option #111889
• [Stack Monitoring] Metricbeat writes to .monitoring-*-8-mb indice instead of datastream after 8.0 upgrade beats#30769 - waiting on feedback from ES team, nothing we can do yet
• [metricbeat] Stack monitoring modules may ignore xpack configuration on error beats#30809 - maybe better to consider once the health endpoint exists
• [Stack Monitoring] Rules do not support standalone clusters  #125303

Tech Debt

• [Monitoring] Allow users to configure CCS pattern #120384
• [Stack Monitoring] Support monitoring.ui.debug_mode for rules #127683
• [Stack Monitoring] Elasticsearch module should attempt to parse the request.body by default (or as an option) beats#30678
• [Stack Monitoring] Address outstanding ESTF failures #128757

Typescript (we'll wait on public conversion)

• Public has ~200 non-test .js{x} files remaining, mostly in the following directories:
  • /public/components
  • /public/lib

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

Is there an e2e testing issue we should get in here? cc @klacabane since he drafted the doc, not sure if issues are open or not yet.

[matschaffer]

I addd 2 bugs from https://github.com/elastic/support-known-issues/issues/1050 to the bug list on this issue.

[jasonrhodes]

@Kerry350 / @matschaffer would you say that #130097 will close the last two items on the TS list (APM routes and iots plan)? If so, that should (hopefully) make the rest of the routes folder be much simpler to get through, IIUC?

[Kerry350]

@jasonrhodes Correct. In fact I'm going to close #102437 and instead just drop a note / AC to say that io-ts should be used for validation to each of the remaining route conversion tickets.

#117756 did the groundwork (adding directories, helpers, runtime types etc) that makes adding io-ts validation to routes easy. Felix has further expanded on this a little in his work by adding even more common shared runtime types etc.

Between #130725 and #117756 route conversion should be pretty simple.

[jasonrhodes]

@Kerry350 thanks so much! This is looking really great 👏

[smith]

Not everything on here is done, but we're closing this as an issue we're tracking and will handle future technical debt items individually. 🚀