Add event-flow visualization #3102

williaster · 2017-07-10T19:53:09Z

This PR adds an event flow visualization for analysis of event sequences. The vis is mostly built in @data-ui with @vx and there is a light wrapper for Superset integration.

Overall the vis supports the following:

Takes raw user (or more generically 'entity') event sequences and computes aggregate paths (shown in top panel of vis)
Allows alignment of event sequences by an arbitrary event index and event type (e.g., the 2nd click event)
Allows selection of arbitrary subtrees of the aggregated graph; this filters to the selected subtree in the aggregate panel and shows the corresponding/selected raw event sequences in a bottom panel
Gives a breakdown of event type counts in the right panel; by clicking on the legend you can filter to / filter out specific types of events.
X-axis: by default aggregate nodes are positioned according to the mean elapsed time from the previous node, for closely-spaced events you can also position by sequence number (1st, 2nd, 3rd, etc)
Node sorting: by default nodes are ordered top -> bottom based on high -> low event count meaning most common nodes appear at the top; you can also order by short -> long elapsed time to the next event.

note that builds, etc. will fail until the @data-ui/event-flow package is published (which I should get to later today), I just wanted to get some 👀 s on the PR

williaster · 2017-07-10T22:53:24Z

and here are some pics in Superset:

Explore controls

Dashboard

mistercrunch · 2017-07-10T23:59:33Z

The image should be square, ideally 1024*1024 to render well

mistercrunch · 2017-07-11T00:00:35Z

This is awesome btw! I'll try to review soon but I'm a little swamped at the moment.

mistercrunch · 2017-07-11T00:04:26Z

superset/assets/visualizations/EventFlow.jsx

+  TS,
+  EVENT_NAME,
+  ENTITY_ID,
+} from '@data-ui/event-flow';


what does '@%' do? where does the event-flow code live?

it's just a name-spaced package on npm, under the "@data-ui" user but mainly so all the @data-ui packages are in the same repo.

it will live at https://github.com/williaster/data-ui/packages/event-flow but I haven't quite merged the chart into the master @data-ui repo, adding one more feature. this is the (massive 🙈) PR

williaster · 2017-07-12T07:25:28Z

The package is published now so the build should be fixed (you can demo it here).

For visual clarity + performance, with larger number of events expected with use in Superset (10,000s) I also added functionality to "trim" nodes in the visualization based on a minimum event count threshold. This means that sequences are aggregated together but "leaf" nodes are hidden from view.

Before / after trimming

mistercrunch

Sorry for the delay on the review, but this is really amazing and the kind of visualization that is a key "selling" point for Superset. I have a few notes for otherwise LGTM!

I'm curious about your CSS in your module and want to confirm that it's namespace alright so it doesn't spill on other visualization (the same way treemap did on yours).

mistercrunch · 2017-07-17T16:02:20Z

superset/assets/javascripts/explore/stores/controls.jsx

@@ -1273,5 +1273,11 @@ export const controls = {
    hidden: true,
    description: 'The number of seconds before expiring the cache',
  },
+
+  generic_checkbox: {


This is the name that will be use in the saved slice's json. Having something generic here will make it harder if/when trying to generate slice dynamically, and harder to read the json and understand what that maps to. Also references in the backend/frontend code won't be as explicit. Let's use a name that describe the actual feature this triggers.

mistercrunch · 2017-07-17T16:04:43Z

superset/assets/package.json

@@ -56,7 +57,7 @@
    "jquery": "^3.2.1",
    "jsdom": "9.12.0",
    "lodash.throttle": "^4.1.1",
-    "mapbox-gl": "^0.26.0",
+    "mapbox-gl": "^0.27.0",


I'm guessing this was an attempt at fixing the npm install bug. It's been fixed on master by using yarn instead of npm install so this is probably not needed anymore. Though if you've tested that the mapbox viz renders alright I'm not against the bump here even though it would belong in another PR ideally.

mistercrunch · 2017-07-17T16:08:32Z

superset/assets/visualizations/EventFlow.jsx

+  const hasData = json.data && json.data.length > 0;
+
+  // the slice container overflows ~80px in explorer, so we have to correct for this
+  const isExplorer = (/explore/).test(window.location.pathname);


not sure if I understand why, but I'm wondering if using slice.width() and slice.height() would address the 80px issue?

I tried this to begin with but it doesn't seem to work/couldn't come up with any fix for the css. The panel dom looks like:

<div class="panel"> <div class="panel-heading">...</div> <div class="panel-body">...</div> </div>

panel-body is what the slice is in and it seems to have the same height as the parent panel. that means it overflows by the height of panel-heading :/

williaster · 2017-07-17T21:04:09Z

@mistercrunch I made the following changes

removed mapbox version bumps (these broke the build for a while, looks good now 👌)
updated the thumbnail png to 1024 x 1024
made the checkbox more specific to event-flow
added one more control that determines the initial min event count for graph trimming

mistercrunch · 2017-07-19T02:45:57Z

I fixed the coveralls token in master, please rebase+push

…ments, add min event count form.

coveralls · 2017-07-19T06:19:01Z

Coverage decreased (-0.07%) to 69.195% when pulling 4aef4eb on williaster:chris--event-flow into d7e4191 on apache:master.

coveralls · 2017-07-19T06:19:01Z

Coverage decreased (-0.07%) to 69.195% when pulling 4aef4eb on williaster:chris--event-flow into d7e4191 on apache:master.

coveralls · 2017-07-19T06:19:01Z

Coverage decreased (-0.07%) to 69.195% when pulling 4aef4eb on williaster:chris--event-flow into d7e4191 on apache:master.

coveralls · 2017-07-19T06:19:01Z

Coverage decreased (-0.07%) to 69.195% when pulling 4aef4eb on williaster:chris--event-flow into d7e4191 on apache:master.

williaster · 2017-07-19T06:20:02Z

@mistercrunch done

williaster · 2017-07-19T06:26:02Z

let me know if I should add a test for the coverage.

coveralls · 2017-07-20T00:19:59Z

Coverage decreased (-0.09%) to 69.175% when pulling 4550d06 on williaster:chris--event-flow into d7e4191 on apache:master.

vnnw · 2017-11-24T03:24:53Z

Can you add a demo into Superset distribution?

williaster · 2017-11-29T21:08:03Z

@vnnw yes this would be a good addition to get people up and going a little more easily.

mistercrunch reviewed Jul 11, 2017

View reviewed changes

williaster force-pushed the chris--event-flow branch 2 times, most recently from a5f82ba to 7d24928 Compare July 15, 2017 01:17

williaster mentioned this pull request Jul 16, 2017

fix babel parsing for superset build williaster/data-ui#35

Closed

mistercrunch requested changes Jul 17, 2017

View reviewed changes

williaster force-pushed the chris--event-flow branch from fcab639 to abc09ee Compare July 17, 2017 19:25

williaster added 10 commits July 18, 2017 22:52

[event-flow] add event flow visualizaton type from @data-ui/event-flow.

b77566b

[event-flow] update vis thumbnail

4314a54

[event-flow] update row limit label, remove duplicate chart controls

45f25ea

[dependencies] add @data-ui/event-flow 0.0.2

7ff7270

[linting] fix multiple imports

f1caccf

[deps] bump mapbox-gl and react-map-gl to fix build

7b9eb37

[event-flow] bump to 0.0.3 for es2015 + stage-0 babel presets

e869ef3

[deps] revert mapbox version bumps

347a4dd

[event-flow] update png, bump to newest version, address reviewer com…

d54cd89

…ments, add min event count form.

[event-flow] pin version

4aef4eb

williaster force-pushed the chris--event-flow branch from e72aebc to 4aef4eb Compare July 19, 2017 05:53

[event-flow][spec] add test for coveralls

66970b1

mistercrunch merged commit 40d9e15 into apache:master Jul 21, 2017

williaster deleted the chris--event-flow branch July 24, 2017 16:25

lpatruno mentioned this pull request Apr 10, 2020

Any documentation on the Event Flow visualization tool? #5873

Closed

edmooney mentioned this pull request May 13, 2022

[Snyk] Fix for 1 vulnerabilities edmooney/incubator-superset#64

Open

mariasnyk mentioned this pull request May 14, 2022

[Snyk] Fix for 1 vulnerabilities mariasnyk/incubator-superset#83

Open

terrorizer1980 mentioned this pull request May 14, 2022

[Snyk] Fix for 1 vulnerabilities terrorizer1980/superset#65

Closed

gregnrobinson mentioned this pull request May 14, 2022

[Snyk] Fix for 1 vulnerabilities gregnrobinson/superset#74

Open

terrorizer1980 mentioned this pull request May 14, 2022

[Snyk] Fix for 1 vulnerabilities Exhorder6/superset#77

Closed

lili2311 mentioned this pull request May 14, 2022

[Snyk] Fix for 1 vulnerabilities lili-test/incubator-superset#64

Open

mariasnyk mentioned this pull request Oct 26, 2022

[Snyk] Fix for 2 vulnerabilities mariasnyk/incubator-superset#101

Open

snyk-bot mentioned this pull request Oct 27, 2022

[Snyk] Fix for 2 vulnerabilities lili-test/incubator-superset#103

Open

m-ajay mentioned this pull request Jun 28, 2023

[Snyk] Fix for 1 vulnerabilities m-ajay/superset#148

Open

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.19.0 labels Feb 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add event-flow visualization #3102

Add event-flow visualization #3102

williaster commented Jul 10, 2017 •

edited

williaster commented Jul 10, 2017 •

edited

mistercrunch commented Jul 10, 2017

mistercrunch commented Jul 11, 2017

mistercrunch Jul 11, 2017

williaster Jul 11, 2017 •

edited

williaster commented Jul 12, 2017

mistercrunch left a comment

mistercrunch Jul 17, 2017

mistercrunch Jul 17, 2017

mistercrunch Jul 17, 2017

williaster Jul 17, 2017

williaster commented Jul 17, 2017 •

edited

mistercrunch commented Jul 19, 2017

coveralls commented Jul 19, 2017

coveralls commented Jul 19, 2017

coveralls commented Jul 19, 2017

coveralls commented Jul 19, 2017

williaster commented Jul 19, 2017

williaster commented Jul 19, 2017

coveralls commented Jul 20, 2017

vnnw commented Nov 24, 2017

williaster commented Nov 29, 2017

Add event-flow visualization #3102

Add event-flow visualization #3102

Conversation

williaster commented Jul 10, 2017 • edited

williaster commented Jul 10, 2017 • edited

mistercrunch commented Jul 10, 2017

mistercrunch commented Jul 11, 2017

mistercrunch Jul 11, 2017

Choose a reason for hiding this comment

williaster Jul 11, 2017 • edited

Choose a reason for hiding this comment

williaster commented Jul 12, 2017

mistercrunch left a comment

Choose a reason for hiding this comment

mistercrunch Jul 17, 2017

Choose a reason for hiding this comment

mistercrunch Jul 17, 2017

Choose a reason for hiding this comment

mistercrunch Jul 17, 2017

Choose a reason for hiding this comment

williaster Jul 17, 2017

Choose a reason for hiding this comment

williaster commented Jul 17, 2017 • edited

mistercrunch commented Jul 19, 2017

coveralls commented Jul 19, 2017

coveralls commented Jul 19, 2017

coveralls commented Jul 19, 2017

coveralls commented Jul 19, 2017

williaster commented Jul 19, 2017

williaster commented Jul 19, 2017

coveralls commented Jul 20, 2017

vnnw commented Nov 24, 2017

williaster commented Nov 29, 2017

williaster commented Jul 10, 2017 •

edited

williaster commented Jul 10, 2017 •

edited

williaster Jul 11, 2017 •

edited

williaster commented Jul 17, 2017 •

edited