Add Aggregation user input _index #2031

bpostlethwaite · 2017-09-22T18:01:06Z

fixes #2024
and adds _index to fullData aggregation.aggregations to make back referencing into user data possible.

This necessitated a change in plots.js to prevent the supply defaults step running twice with the second run acting on fullData rather then userData.

etpinard · 2017-09-22T19:47:12Z

Very interesting fix. Transforms that do not expand traces shouldn't have to go through supplyTransformDefaults twice. At the moment, all transforms but groupby don't expand traces. So from a pref standpoint, this PR is 👌

cc @rreusser who in #1830 had to go-around a few transform design problems.

rreusser · 2017-09-28T19:03:58Z

src/plots/plots.js

@@ -1039,9 +1039,21 @@ plots.supplyTransformDefaults = function(traceIn, traceOut, layout) {
            _module = transformsRegistry[type],
            transformOut;

+        /*
+         * Supply defaults may run twice. First pass runs all supply defaults steps


Questions:

I assume this refers to the way supplyTraceDefaults runs on traces and then again on expanded traces?

Which _module is added? I think that sometimes refers to the trace module and sometimes to the transform module (and every now and then to the base plot module, though usually it's called baseModule or something, I think)

Answer to second: the transform module.

Yep that is what it is referring to

rreusser · 2017-09-28T19:04:09Z

src/plots/plots.js

+         * and adds the _module to any output transforms.
+         * If transforms exist another pass is run so that any generated traces also
+         * go through supply defaults. This has the effect of rerunning
+         * supplyTransforms. If the transform does not have a `transform` function


*supplyTransformDefaults?

rreusser · 2017-09-28T19:09:48Z

src/plots/plots.js

+         * it could not have generated any new traces and the second stage is
+         * unnecessary. We detect this case with the following variables.
+         */
+        var isSecondStage = transformIn._module && transformIn._module === _module,


Is a third stage possible if multiple transforms expand things multiple times? (e.g. groupby + groupby) I think this makes sense to me, but just making sure transform module identity equates to isSecondStage. (Or is it isNotFirstStage?) My understanding is just a little loose here.

right, more than 2 stages... need to think about that. Suggestions?

Right I think we are ok. I should call this variable isNotFirstStage though. Thanks

Things get fundamentally a bit shaky after one level. Like groupby + groupby. It does mostly what you'd expect, but it turns out styling is the big challenge. Think SQL. You massage the query until you have a nice query that spits out data. Then you insert it into a plot and you're happy. Plotly's transforms incrementally apply styles as a sequence of transforms is applied so that you'd have to track styles as well as the pathway a data point took through the query to really sort things out after the fact. Though I agree that I think we're okay.

Sorry, specifics: groupby adds a _group property to traces so that you can figure out which group split generated a trace. But it should really be tracking that as a transform index and a group value. This comes into play with transform vs. calcTransform since you start having to worry about when a trace was split/grouped/styled/etc.

Apologies if that's still a bit loose on the specifics. /cc @alexcjohnson since we've talked this through before in a fair bit of detail. I think our conclusion at the time was that the basic use cases are handled just fine and account for 97+% of realistic use-cases.

bpostlethwaite · 2017-09-28T19:26:21Z

@rreusser some changes here: ec0a65f

Does that seem more clear? It seems to be more honest about the possibility of more than 2 stages.

rreusser · 2017-09-28T19:28:45Z

test/jasmine/tests/transform_aggregate_test.js

+        });
+
+        expect(enabledAggregations[0].target).toEqual('x');
+        expect(enabledAggregations[0]._index).toEqual(0);


Is there any way to test this that doesn't rely upon an underscored property? Like could supplyTransformDefaults delete the first aggregation if it's overridden by another? Would that be adequate to sort things out in the workspace or does the workspace need to know indices?

The only reason for indicies is for Workspace. supplyTransformDefaults doesn't care about them.

I need a way in Workspace to take a fullData transform Aggregration and write back to the correct userData index. I need some way to correlate userData indicies with the fullData equivalents.

maybe it shouldn't be underscored? Something like .inputIndex. the only problem with that is there are precedents for things being added to fullData for consumption by users that have an underscore. _input for example.

So the workspace accesses indices directly? (That's okay, just making sure. plotly.js's 'private' properties are fairly publicly accessed, so it's nothing new. Just means they need to be modified with great care.)

yep, and the workspace is the only thing that needs them. Plotly.js doesn't care as it rebuilds new arrays each time.

rreusser · 2017-09-28T19:30:01Z

Above corner-case comments notwithstanding, this all looks reasonable to me.

etpinard · 2017-10-02T13:40:59Z

Ok. Looks like @rreusser approves this.

💃 for me @bpostlethwaite

This will be part of 1.31.0 set to be released no-later than October 5.

missing aggregations array no longer throws - fix #2024

e555e0a

bpostlethwaite requested a review from etpinard September 22, 2017 18:01

bpostlethwaite force-pushed the aggregationInputIndicies branch from 9e3088d to 13dd5c6 Compare September 22, 2017 18:09

add and test aggregation back _index.

75e6e48

This necessitated a change in plots.js to prevent the supply defaults step running twice with the second run acting on fullData rather then userData.

bpostlethwaite force-pushed the aggregationInputIndicies branch from 13dd5c6 to 75e6e48 Compare September 22, 2017 18:17

etpinard added status: reviewable type: bug type: performance labels Sep 25, 2017

etpinard added this to the v1.31.0 milestone Sep 28, 2017

rreusser reviewed Sep 28, 2017

View reviewed changes

documentation and variable name improvements. Thanks @rreusser

ec0a65f

rreusser reviewed Sep 28, 2017

View reviewed changes

etpinard approved these changes Oct 2, 2017

View reviewed changes

bpostlethwaite merged commit f617aef into master Oct 2, 2017

bpostlethwaite deleted the aggregationInputIndicies branch October 2, 2017 13:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Aggregation user input _index #2031

Add Aggregation user input _index #2031

bpostlethwaite commented Sep 22, 2017

etpinard commented Sep 22, 2017 •

edited

Loading

rreusser Sep 28, 2017

rreusser Sep 28, 2017

bpostlethwaite Sep 28, 2017

rreusser Sep 28, 2017

rreusser Sep 28, 2017

bpostlethwaite Sep 28, 2017

bpostlethwaite Sep 28, 2017

rreusser Sep 28, 2017 •

edited

Loading

rreusser Sep 28, 2017

rreusser Sep 28, 2017

bpostlethwaite commented Sep 28, 2017

rreusser Sep 28, 2017

bpostlethwaite Sep 28, 2017

bpostlethwaite Sep 28, 2017

bpostlethwaite Sep 28, 2017 •

edited

Loading

rreusser Sep 28, 2017

bpostlethwaite Sep 28, 2017

rreusser commented Sep 28, 2017

etpinard commented Oct 2, 2017 •

edited by jackparmer

Loading

Add Aggregation user input _index #2031

Add Aggregation user input _index #2031

Conversation

bpostlethwaite commented Sep 22, 2017

etpinard commented Sep 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rreusser Sep 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpostlethwaite commented Sep 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpostlethwaite Sep 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rreusser commented Sep 28, 2017

etpinard commented Oct 2, 2017 • edited by jackparmer Loading

etpinard commented Sep 22, 2017 •

edited

Loading

rreusser Sep 28, 2017 •

edited

Loading

bpostlethwaite Sep 28, 2017 •

edited

Loading

etpinard commented Oct 2, 2017 •

edited by jackparmer

Loading