Add rate and percentile aggregations to docs for custom threshold rule

[dedemorton]

Adds changes requested in https://github.com/elastic/observability-docs/issues/3530.

TODO:

• Add rate and percentile aggregations.
• Create a dedicated section/page for rate aggregation that:
  • Explains why you would use rate instead of average for something like a counter.
  • Shows how the rate is calculated.
  • Provides a realistic example.
    Then point to the page from both rules.
• Refine the new content about rate aggregation and add example (with input from development team)
• Port this change to the serverless docs, too (making sure to add the missing header). https://github.com/elastic/staging-serverless-observability-docs/pull/241/

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[mergify]

This pull request does not have a backport label. Could you fix it @dedemorton? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[dedemorton]

@maryam-saeidi I think I've taken this as far as I can without help from the development team. I looked through all the resources you referenced in the issue, but it's really a lot to slog through. I did add a new page called "Understanding rate aggregations" but I suspect that this should probably be refined (written?) by someone who didn't just learn what a rate aggregation was yesterday. :-) The quality of the content perhaps reflects my ignorance. At any rate, it's a starting point. I welcome your comments and guidance to make it better. Plus I would appreciate your input on a good example to provide on that page.

[maryam-saeidi]

Hi @dedemorton,

Thanks for adding this page! 🙏🏻

How would you like to collaborate, shall I just write information for the sections that you provided in a Google doc and share it here or do you prefer it differently? I will also get feedback from one of our SREs who worked with this aggregation to see what he thinks would be a nice addition to this document. :)

[dedemorton]

@maryam-saeidi I'm totally OK with you pushing your changes directly to my fork to make them part of the PR review. If you prefer to work in google docs, that's OK too. I've added you to my fork as a contributor so you can push changes to the branch there (if you want).

[maryam-saeidi]

@dedemorton I had a chat with @jasonrhodes and we think it would be better to put this document under a separate category like aggregations, or a separate item. The current place of this document does not match the rest of the items in the same group (which are different rule types):

So, either we can add another item in the following list or a separate group like aggregations:

Current	Suggestion
	Create and manage rules Aggregations > rate View alerts

[dedemorton]

The current place of this document does not match the rest of the items in the same group

@maryam-saeidi Oh I totally agree. I've shoehorned it into the wrong place. The problem with this content is that it's an outlier compared to how we cover other types of aggregations. We are very uneven in how we cover aggregations and don't provide comprehensive reference documentation about all the aggregations we support in alerting rules. I don't want to have Aggregations at the top level in the nav right now because that will raise expectations for users who want to learn about other aggregations and will be disappointed if we only cover rate aggregations. (Plus it's not a best practice to have a nav container that contains a single page.)

For now, I can move the new page up a level in the TOC and rename it. So we'll have something like:

• Alerting
  • Create and manage rules
  • Learn about rate aggregations*
  • View Alerts

*I want to avoid calling the new page "Rate aggregations" to keep the wording parallel in the nav and prevent folks from accidentally reading "rate" as a verb initially.

In the future, we probably need a reference section somewhere in the alerting docs that covers aggregations in more depth, but that's out-of-scope for this PR. If you think it would be valuable, please open an issue, and we'll add it to our sprint planning. Thanks!

[jasonrhodes]

@dedemorton could we make that something like this:

• Alerting
  • Create and manage rules
  • Aggregation options
    • Rate
    • [later, we can add 99th percentile, etc]
  • View Alerts

[dedemorton]

@jasonrhodes I can do that if it would help your team build out more content later, but I think users are going to be misled and potentially annoyed by the missing info (plus the extra clicks). But maybe they feel that way already?

There's been some scope creep with this work (originally sized at <1 day based on the original description in the issue, which was edited after we did our sizings). That's OK, but it means that the container topic for the new section is going to be very basic. Does that sound OK?

[jasonrhodes]

Of course, whatever makes sense on the scope side of things. I also defer to your expertise on the navigation choices. My perspective is that the "Information Architecture" being communicated by having "Learn about rate aggregations" by itself in the nav would, for me, be more confusing and frustrating than having some of the aggregations missing, but I understand what the page is in part because the IA makes sense to my brain (compared to that drop down that I use when I create/edit rules, in other words).

Either way, I'll create a new issue in this repo to represent making a few more pages for the aggregations that seem like they could benefit the most from a similar explanation, @maryam-saeidi I'll probably need help choosing which ones make the most sense.

Update: with the new table structure DeDe added, I don't think we need a new issue for more pages of aggregation explanations. I'll not make that issue, and we can add them if they naturally arise.

Thanks for your help, @dedemorton!

[dedemorton]

I'll add a commit so you can see how it looks, and we can go from there.

[jasonrhodes]

Thank you @dedemorton for all of the help and back and forth on this, it's looking great imo <3

[bmorelli25]

Pending the last question for @maryam-saeidi, this lgtm!

[maryam-saeidi]

LGTM! I've added one comment for Rate aggregations.

Thanks for all your effort to improve our documentation ❤️