Nailing Correlation Analysis

[neilkakkar]

As noted in #6360, our ambitious goal this coming sprint is to build the best possible quant analysis tool.

In prep for this, we did some user testing on the MVP, and an industry analysis + feature brainstorm. (thanks Paolo!)

I interpret best as a tool that does its job without getting in the way. And the job for correlation analysis is to help diagnose causes.

There's two things we need to accomplish to be successful here:

1. Surface meaningful & easy to understand results
2. Easily allow further actions on surfaced results to make them actionable.

For example, I might see that "Video Played" is a signal for success, but I can't understand why: Maybe it's specific videos that specially convert well, or something else to do with them. So, once I see "Video Played", I expect to be able to answer the following question: What is it about Video Played that makes it a good signal?

Implicit in the 2 things above is not cluttering the UI with 1000s of different options that might be useful: If it isn't clear how to answer your question, the tool isn't useful.

---

Things we ought to do: (rough prioritisation for now, evolving list). Ideally we'd do all of these, but given the 2 week constraint, we want to choose maximum impact things first. I think both lists go in parallel, and we should pick up the first thing from either and continue & see where we get till the end of the sprint.

Surface meaningful & easy to understand results

We're already in a place where results are useful to some people. Making these better involves allowing discarding 'obvious' events, surfacing things users are familiar with (actions), and making the numbers (if any) easy to understand & removing discrepancies.

3. Allow global property exclusions from Property correlation
4. Allow excluding events from Events Correlation
5. Polish our UI, make things clearer.
   1. Make a call on what all information to show: Entire contingency tables, success & failure counts, or just the odds ratio.
   2. Actionable tips to reduce skew (a.k.a. give more confident results) vs mentioning a confidence level
6. Include Actions in Event signals
   1. Figure out how to do this & timings & how long this takes, since action matching has moved back to plugin-server
7. [ ]

Allow further actions on surfaced results

Where we're lacking right now is what to do with the results. It's hard to go from "oh, this is a signal" to "What I should do about it". Drilling down on signals via paths & breakdowns; and then qualitative feedback on this small set, like session recordings, help figure out the problems.

For example, on a property breakdown like browser_version=87 being a signal of failures, I'd love to see some session recordings to see what went wrong.

1. Given an event, allow breaking down by event properties - @neilkakkar
   1. Figure out technical constraints around this - can we do all event properties (would be unique to us) quickly?
   2. Figure out how & where to display this
   3. Some special automatic handling for internal events like $autocapture and $pageview
2. Given an event / property signal, show me all persons who did this,
   1. and link to relevant session recordings
   2. [ ]
3. Given a signal, show me the paths these specific users took.
4. (questionable value): Add this event signal to the funnel

---

I think executing the above things gets us to a place where we are integrated pretty well into other tools. It complements, and shortcuts what users would anyways be doing to diagnose causes, thus achieving its goal of helping diagnose causes.

To judge success: Get 3 users to LOVE correlation analysis

cc: @clarkus @paolodamico @marcushyett-ph @hazzadous @liyiy @EDsCODE @macobo for any ideas on what we should / should not be doing.

[neilkakkar]

rm events in an actions funnel - events that make up actions for the funnel.

[neilkakkar]

Hey @clarkus - some technical updates that should unblock design:

1. We want to allow drilling down on an event, to run correlation on its properties.
2. We can surface all properties of an event (too much crap, I don't want to do this), or max top 10 property values (much nicer, more directed)
3. For now, I don't want to give the option to choose specific properties of an event to run correlation on. We just do it on all of them, every time. (let me know if you feel the design is better if we allow selecting properties). (I felt it cluttered UI too much)
   Edit: this might become necessary given the current state on production - I can now see it being icky to use if we have lots of random properties defined. Something to resolve with further testing.
4. The above is true^ except for special events, like $autocapture and $pageview. I'm yet to investigate what all properties should do here, and how much noise comes with this, but safe to say this will all happen on the backend, and shouldn't affect frontend designs. I'd probably exclude properties defined in the global exclusion (even if they're for person properties) in here as well.

regarding timings for PostHog sized org: It takes about ~1.5 second to run correlation analysis on all properties for a given event.

A more ambitious analysis is for running property correlations on all events & all properties. This takes ~5 seconds, so for now I'm ditching this functionality. We can revisit if there's a good enough usecase for it.

---

About connecting to other insights: I was thinking of showing Person modals on the success and dropoff counts, and linking to Paths & session recordings via the modal. Does this make sense? (very open to ideas here, and if we should represent it differently / making it 1-click vs 2-click / etc.). The problem we want to solve here is to allow diving deeper: given a correlation analysis result, I want to see what else these specific people are doing: their paths & screen recordings.

[paolodamico]

Taking a short step back with some thoughts (happy to discuss sync too, might be easier),

• IMO the most ambitious thing we can accomplish here is leverage what others can't do (recordings & autocapture). Particularly in terms of autocapture (conceptually), if we could surface relevant actions that end users took that may not even be on the radar, it would be killer. I think the approach here would be to whitelist stuff like clicking on buttons or other relevant elements. In addition, we could consider start tracking stuff like dead clicks, loads & performance, ... that we could surface automatically here and are relevant for other analytics (like rage clicks).
• Priorities-wise on the stuff from your list, I would consider something like this:
  1. Link relevant recordings & persons modal.
  2. Include actions in event signals.
  3. Breaking down on event properties.
  4. Event exclusions.
• I would treat UX improvements as a separate thing, and we should definitely do it. @clarkus and I can tackle them together if you want so you can focus on the main functionality. I don't anticipate a ton of engineering work anyways.

Details/rationale for priorities order

Rationale for prioritization 👇

1. Uniquely position to solve this, way to deep dive and get qualitative input.
2. Autocapture is a huge value prop for PH and we would be losing this, we have a few focus customers heavily relying on them, can add a ton more power to correlation to form hypotheses by creating actions.
3. @neilkakkar explained it best (above).
4. Helps surface more relevant insights over noise.

[clarkus]

About connecting to other insights: I was thinking of showing Person modals on the success and dropoff counts, and linking to Paths & session recordings via the modal. Does this make sense? (very open to ideas here, and if we should represent it differently / making it 1-click vs 2-click / etc.). The problem we want to solve here is to allow diving deeper: given a correlation analysis result, I want to see what else these specific people are doing: their paths & screen recordings.

This aligns with our other uses of the person modal so far. I think that's a good start. We offer this in the core funnels visualization, so we might consider how we clarify the different origins of person lists.

[clarkus]

I was just trying this with our data - the current functionality seems to be there. I think you could expand the drill-down section once the analysis finishes running - it's just a stronger signal that things are done and ready for a look. Regarding the scale of drill-down properties, maybe we just show the top N items and then allow another action to "show more". I think a lot of the scale issues might be resolved by creating some reasonable defaults for filtering out known noisy events / properties. Secondary to that, users could flag events that are most critical to their use. That would be a strong signal on what matters from a user perspective. I'm happy to jump in on design iterations once you think we're ready for that.

[paolodamico]

Sounds good @clarkus! I think the exclusions thing will be pretty helpful right now and we can then explore something like "critical" properties once we have some more feedback/data here.

---

The other thing (pending @neilkakkar's comments here) I'd love to explore design-wise is how can we better display the results. In particular,

1. The way other platforms display the results do a great job with FOMO to prompt you to dive deeper and fix issues sooner (e.g. $ loss).
2. How to display some deeper metrics on the results? e.g. correlation factor, FP/FN matrix table, ....

The benchmark doc will provide a lot more context here.

[neilkakkar]

Had a sync chat about this:

1. For now, we want to support $autocapture elements_chain as just another nested property. Thus, it makes priority for nested properties the highest. The insight here is that solving the more general problem of relevant event properties helps better unlock $autocapture properties.
2. In the future we might think about giving this its own special place on the UX, if we see decent results coming out of them
3. Idea for future: Creating other events (like $rageclick) out of autocapture, like dead clicks powers correlation analysis more.

[neilkakkar]

Growing list of default Property Exclusions:

$initial_geoip_postal_code
$initial_geoip_latitude
$initial_geoip_longitude
$geoip_latitude
$geoip_longitude
$geoip_postal_code
$geoip_continent_code
$geoip_continent_name
$initial_geoip_continent_code
$initial_geoip_continent_name
$geoip_time_zone
$geoip_country_code
$geoip_subdivision_1_code
$initial_geoip_subdivision_1_code
$geoip_subdivision_2_code
$initial_geoip_subdivision_2_code
$geoip_subdivision_1_code
$initial_geoip_subdivision_1_code
$geoip_subdivision_name
$initial_geoip_subdivision_name
$initial_geoip_city_name

All the above are irrelevant in the face of $geoip_country_name and $geoip_city_name

---

Rule properties in for $autocapture, vs out:

Only elements_chain seems useful so far?

Same for $pageview?: Nah, let $pageview be default place for all these random autocaptured properties to show up.

$event_type
$pathname
$current_url
$device
UTM_*

[posthog-bot]

This issue hasn't seen activity in two years! If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[posthog-bot]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.