[Maps] observability real user monitoring solution layer #64949

nreese · 2020-04-30T18:46:42Z

Implements part of #64445.

This PR adds Observability layer wizard card making it easy to create map layers from Observability data. This PR provides two basic layer types for RUM transactions. Expect a lot more in this space in future PRs.

The card is only displayed when APM index pattern is installed. The APM index pattern is automatically installed when opening the APM application with data in apm-* index.

elasticmachine · 2020-04-30T18:46:44Z

Pinging @elastic/kibana-gis (Team:Geo)

elasticmachine · 2020-04-30T18:47:23Z

Pinging @elastic/apm-ui (Team:apm)

nreese · 2020-05-04T19:48:59Z

@jsanz

The default green does not match very well with any of the ramps used, can I advocate for a darker grey instead? something like #3d3d3d?

done

Not really sure about this one because I see more cases where I don't like it, but for the APM RUM Performance -> SLA percentage cluster option, labeling would be nice

Added

nreese · 2020-05-04T19:49:32Z

@thomasneirynck

+1 on renaming some things. e.g.
"Observabiliby layers" -> "APM layers"
"Choropleth" -> "World boundaries"

done

nreese · 2020-05-04T19:56:49Z

@alexfrancoeur

For the performance layers I'm wondering if we should style green to red (fastest to slowest).

done

nreese · 2020-05-04T21:24:01Z

@elasticmachine merge upstream

thomasneirynck

lgtm for Maps-code. Add-layer flow works really well, and good concept to start using for other "curated layers".

Main feedback is on the actual data-display side.

Use proper formatting of values

RUM Performance > SLA Percentage layer.

Format the % fields as actual percentages / trim decimal places

When these percentage values are shown as decimals, it looks funny on the map and the legend.

e.g.: This is how duration_sla_pct field formatter would look like

Map-formatting looks "better" imho.

Rum Performance > Transaction Duration layer.

Use duration formatter.
e.g.

This would format ms properly

@sqren : Is it possible to specify these formatters in the APM index pattern? The Maps-application will automatically pick these up to display values/format legends/tooltips etc...

Use custom ramps

It seems the performance-metrics are diverging. Presumably, a decrease is "better" (smaller than 0), and an increase is worse (larger than 0) (??) . Ideally, can we center the ramp at value 0 then. This would require a custom ramp iso using an out-of-the-box one.

@sqren Can you confirm if this is the case? Are the metrics for "transaction duration" and "SLA percentage" diverging? (either below or above 0.0)?

cc @alexfrancoeur Are the above suggestions required?

sorenlouv · 2020-05-05T12:40:55Z

Is it possible to specify these formatters in the APM index pattern? The Maps-application will automatically pick these up to display values/format legends/tooltips etc...

@thomasneirynck

I think so - but since the APM app doesn't use index patterns I'm not the most knowledgable about this.
My understanding is that the index pattern is created by beats via the APM Server, and then semi-automatically added to kibana. Example: #64232
There might be a place in the APM Server settings where these formatters can be added.
@elastic/apm-server might know more (or know who to ask).

thomasneirynck · 2020-05-05T13:31:23Z

Seems like this would be the place to add a field-formatter to the apm-* index pattern (?)

kibana/x-pack/plugins/apm/server/tutorial/index_pattern.json

Line 3 in 103a3cd

    
           "fieldFormatMap": "{\"client.bytes\":{\"id\":\"bytes\"},\"client.nat.port\":{\"id\":\"string\"},\"client.port\":{\"id\":\"string\"},\"destination.bytes\":{\"id\":\"bytes\"},\"destination.nat.port\":{\"id\":\"string\"},\"destination.port\":{\"id\":\"string\"},\"event.duration\":{\"id\":\"duration\",\"params\":{\"inputFormat\":\"nanoseconds\",\"outputFormat\":\"asMilliseconds\",\"outputPrecision\":1}},\"event.sequence\":{\"id\":\"string\"},\"event.severity\":{\"id\":\"string\"},\"http.request.body.bytes\":{\"id\":\"bytes\"},\"http.request.bytes\":{\"id\":\"bytes\"},\"http.response.body.bytes\":{\"id\":\"bytes\"},\"http.response.bytes\":{\"id\":\"bytes\"},\"http.response.status_code\":{\"id\":\"string\"},\"log.syslog.facility.code\":{\"id\":\"string\"},\"log.syslog.priority\":{\"id\":\"string\"},\"network.bytes\":{\"id\":\"bytes\"},\"package.size\":{\"id\":\"string\"},\"process.parent.pgid\":{\"id\":\"string\"},\"process.parent.pid\":{\"id\":\"string\"},\"process.parent.ppid\":{\"id\":\"string\"},\"process.parent.thread.id\":{\"id\":\"string\"},\"process.pgid\":{\"id\":\"string\"},\"process.pid\":{\"id\":\"string\"},\"process.ppid\":{\"id\":\"string\"},\"process.thread.id\":{\"id\":\"string\"},\"server.bytes\":{\"id\":\"bytes\"},\"server.nat.port\":{\"id\":\"string\"},\"server.port\":{\"id\":\"string\"},\"source.bytes\":{\"id\":\"bytes\"},\"source.nat.port\":{\"id\":\"string\"},\"source.port\":{\"id\":\"string\"},\"system.cpu.total.norm.pct\":{\"id\":\"percent\"},\"system.memory.actual.free\":{\"id\":\"bytes\"},\"system.memory.total\":{\"id\":\"bytes\"},\"system.process.cpu.total.norm.pct\":{\"id\":\"percent\"},\"system.process.memory.rss.bytes\":{\"id\":\"bytes\"},\"system.process.memory.size\":{\"id\":\"bytes\"},\"url.port\":{\"id\":\"string\"},\"view spans\":{\"id\":\"url\",\"params\":{\"labelTemplate\":\"View Spans\"}}}",

Would this impact any of the existing code @graphaelli (?)

kibanamachine · 2020-05-05T13:50:15Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d692faf

History

💔 Build #45279 failed 425943f
💔 Build #45228 failed c0d168e
💔 Build #45207 failed 03dc864
💔 Build #45155 failed 5f292aa
💚 Build #44936 succeeded 6686c3f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

sorenlouv · 2020-05-05T13:53:23Z

Seems like this would be the place to add a field-formatter to the apm-* index pattern (?)

That thing is auto-generated, so adding anything there would be overwritten next time apm-server updates the index pattern.

sorenlouv · 2020-05-05T13:58:20Z

Tangent: This shows why index pattern in the current form is such a bad fit. Having somewhere (like index patterns) to store field formatters only relevant to the UI, and having that live in Kibana, is a good idea. But persisting the field mappings in a secondary location (away from the actual mappings) is a terrible idea.
Since the index pattern now needs to be updated every single time a field in APM Server is added/updated we run into this problem where we need to merge two different responsibilities into one giant object.

Moving field formatters out of index patterns (or removing mappings from index patterns) would be a huge step forward, and resolve a lot of issues imo.

thomasneirynck · 2020-05-05T14:00:53Z

@sqren 100% agreed

the functionality captured by index-patterns is really useful, but having IPs stored in a central location creates friction for use-cases like this. I think the kibana-app team is thinking in a similar direction in how they want to evolve e.g. Lens as well. cc @timroes

alexfrancoeur · 2020-05-05T14:40:19Z

@thomasneirynck I agree that a better long term solution be metric specific field formatters. Similar to Lens, Canvas and TSVB. It's a common request that we don't want to rely on global formatters for visualizing data. I'll add as a topic for discussion to our 7.9 planning. It's a feature I've wanted for nearly every map I build, and I know I'm not alone. For this PR, maybe we merge without addressing formatting initially?

thomasneirynck · 2020-05-05T14:45:32Z

For this PR, maybe we merge without addressing formatting initially?

+1

thomasneirynck

+1 maps-side.

There seems to be a little uncertainty about this sla_duration_pct field, where exactly this comes from. I would change the color-ramp to a custom diverging color-ramp that centers on 0. Depending on whether we get clarification on this field, we could consider this a bug fix and merge after FF. It seems that below 0 would indicate beating "sla", above 0 not meeting the SLA (?)

cc @alexfrancoeur

simitt · 2020-05-05T15:31:21Z

@thomasneirynck the index pattern is generated in APM Server via some libbeat functionality, and then copied over to Kibana to ensure the bundled index pattern is aligned with the APM Server index templates. The correct place to update any kind of formatting etc would be directly in APM Server, e.g. in the metricsets fields.yml.

Do you have a list of fields and formats for which the format should be changed?

thomasneirynck · 2020-05-05T18:14:20Z

hi @simitt

two additions would be useful imho

Field: duration_sla_pct
Formatter: Percentage
Numeral js format pattern: 0,0.[]%

Field: transaction_duration_us
Formatter: Duration
Input format: Microseconds
Output format: Milliseconds
Decimal places: 2

I guess details can be tweaked, but those two fields are used in the solution-layers. It would make the labels on the map and the legend more readable.

I can create a separate issue for this as well, if that's easier. Thx!

gchaps · 2020-05-05T18:27:05Z

I have a suggestion for the text underneath the Format field:

Formats allow you to control the display of specific values. Formats can also change the value and turn off highlighting in Discover.

If these are just "numeric" values, use that word instead of specific in the first sentence. Also in the second sentence, what does "change the value" mean? Change the value of the data?

* [Maps] observability real user monitoring solution layer * make stateful component * rename RUM to observability * layer select * metrics select * display select * clean up * create cholopleth map layer descriptor * get styling working * fix unique count agg * create geo grid source * render as heatmap * clusters * tslint errors * last tslint fix * only show card when APM index exists * layer label * clean up * fix getLayerWizards * add parans * review feedback * add cluster labels * use green to red ramp * tslint fix Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

nreese · 2020-05-05T20:55:26Z

I have a suggestion for the text underneath the Format field

@gchaps Would you mind making an issue for these field formatter text changes? That UI is not a part of this UI. The screen shots where just included to show how @thomasneirynck configured some field formatters.

…5337) * [Maps] observability real user monitoring solution layer * make stateful component * rename RUM to observability * layer select * metrics select * display select * clean up * create cholopleth map layer descriptor * get styling working * fix unique count agg * create geo grid source * render as heatmap * clusters * tslint errors * last tslint fix * only show card when APM index exists * layer label * clean up * fix getLayerWizards * add parans * review feedback * add cluster labels * use green to red ramp * tslint fix Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

nreese added 18 commits April 29, 2020 09:44

[Maps] observability real user monitoring solution layer

f711cf4

make stateful component

eca88a0

rename RUM to observability

33400cc

layer select

e50ab31

metrics select

4d5fc6a

display select

3e3274b

clean up

dc42f71

create cholopleth map layer descriptor

19c8c96

get styling working

1e8064c

fix unique count agg

e943094

create geo grid source

33f1af0

render as heatmap

e41685c

clusters

3ed7a03

tslint errors

ea802bb

last tslint fix

777216c

Merge branch 'master' of github.com:elastic/kibana into solution_layers

bd83501

only show card when APM index exists

f8ae4b3

layer label

1479aab

nreese added release_note:enhancement [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation v8.0.0 v7.8.0 labels Apr 30, 2020

nreese requested review from jsanz and thomasneirynck April 30, 2020 18:46

nreese requested review from a team as code owners April 30, 2020 18:46

nreese added the Team:APM All issues that need APM UI Team support label Apr 30, 2020

clean up

2a3bb0b

nreese requested a review from thomasneirynck May 4, 2020 19:48

use green to red ramp

c0d168e

Merge branch 'master' into solution_layers

425943f

thomasneirynck reviewed May 4, 2020

View reviewed changes

tslint fix

d692faf

miukimiu self-requested a review May 5, 2020 13:51

thomasneirynck approved these changes May 5, 2020

View reviewed changes

nreese merged commit 891e27e into elastic:master May 5, 2020

nreese mentioned this pull request May 5, 2020

[7.x] [Maps] observability real user monitoring solution layer (#64949) #65337

Merged

simitt mentioned this pull request May 6, 2020

Add format to Kibana index pattern elastic/apm-server#3751

Closed

thomasneirynck mentioned this pull request Jun 4, 2020

[Maps] Observability - Real User Monitoring layer template #64445

Closed

nreese mentioned this pull request Jul 2, 2020

[Maps] Heatmap layer has no metrics #70399

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] observability real user monitoring solution layer #64949

[Maps] observability real user monitoring solution layer #64949

nreese commented Apr 30, 2020

elasticmachine commented Apr 30, 2020

elasticmachine commented Apr 30, 2020

nreese commented May 4, 2020

nreese commented May 4, 2020

nreese commented May 4, 2020

nreese commented May 4, 2020

thomasneirynck left a comment

sorenlouv commented May 5, 2020

thomasneirynck commented May 5, 2020 •

edited

Loading

kibanamachine commented May 5, 2020

sorenlouv commented May 5, 2020 •

edited

Loading

sorenlouv commented May 5, 2020 •

edited

Loading

thomasneirynck commented May 5, 2020

alexfrancoeur commented May 5, 2020

thomasneirynck commented May 5, 2020

thomasneirynck left a comment •

edited

Loading

simitt commented May 5, 2020

thomasneirynck commented May 5, 2020

gchaps commented May 5, 2020

nreese commented May 5, 2020

[Maps] observability real user monitoring solution layer #64949

[Maps] observability real user monitoring solution layer #64949

Conversation

nreese commented Apr 30, 2020

elasticmachine commented Apr 30, 2020

elasticmachine commented Apr 30, 2020

nreese commented May 4, 2020

nreese commented May 4, 2020

nreese commented May 4, 2020

nreese commented May 4, 2020

thomasneirynck left a comment

Choose a reason for hiding this comment

Use proper formatting of values

RUM Performance > SLA Percentage layer.

Rum Performance > Transaction Duration layer.

Use custom ramps

sorenlouv commented May 5, 2020

thomasneirynck commented May 5, 2020 • edited Loading

kibanamachine commented May 5, 2020

💚 Build Succeeded

History

sorenlouv commented May 5, 2020 • edited Loading

sorenlouv commented May 5, 2020 • edited Loading

thomasneirynck commented May 5, 2020

alexfrancoeur commented May 5, 2020

thomasneirynck commented May 5, 2020

thomasneirynck left a comment • edited Loading

Choose a reason for hiding this comment

simitt commented May 5, 2020

thomasneirynck commented May 5, 2020

gchaps commented May 5, 2020

nreese commented May 5, 2020

thomasneirynck commented May 5, 2020 •

edited

Loading

sorenlouv commented May 5, 2020 •

edited

Loading

sorenlouv commented May 5, 2020 •

edited

Loading

thomasneirynck left a comment •

edited

Loading