density contours stroke & fill #948

Fil · 2022-06-22T15:06:43Z

As a complement to #943:

group by z, stroke, fill
apply other styles as well (constant like mixBlendMode, channel-based like title…)
use fill: "density" or stroke: "density" to color by "density level" (note that the numbers ~~are not interpretable by the reader, so the legend is not recommended—or maybe we need to normalize~~ are normalized by the maximum density across all facets and series)

todo:

add a weight channel
- (and an example)
document
make sense of the density value (as a percentage of maximum density?)
test initializer composition
1d density contours
wait for a new release of d3-contour? respect thresholds passed to contourDensity d3/d3-contour#57

src/marks/density.js

Fil · 2022-06-23T15:21:11Z

I've tested the initializer composition with the bin and the hexbin transforms… I don't think we want to add them to the unit tests because they don't have value as dataviz, but it's fun

src/marks/density.js

Fil · 2022-06-24T09:02:25Z

rebased on main

Fil · 2022-06-24T10:15:26Z

I want to have 1-d KDE, but it's not in the scope of this PR, which in my view is complete.

mbostock · 2022-06-24T15:42:31Z

src/marks/density.js

+      .bandwidth(bandwidth)
+      .thresholds(thresholds);
+
+    // First pass: seek the maximum density across all facets and series; memoize for performance.


“Memoizing” refers to a caching technique where, given a function that computes a consistent return value for its given arguments, the return values are cached by their arguments so as to avoid invoking the function repeatedly for the same arguments. This isn’t really memoizing here; there’s not a function that represents the computational task. I haven’t quite followed what this code is doing yet but this is probably something like “cache the initial set of contours to avoid recomputing them in the second pass”. I need to read this code more thoroughly (I’m making an edit pass now) but I’ll add more comments here.

Yes, cache the contours and reuse those that ended up with the same threshold selection.

mbostock · 2022-06-24T15:45:04Z

In the future, please don’t delete my PR before I’ve had a chance to review your contributions. I understand the desire to rebase to keep things up-to-date, but it’s now harder for me to review this code because I can’t easily see the changes you made on top of the previous PR I put up. Thank you. 🙏

note: we have to go around this bug in d3-contour: d3/d3-contour#57

(in the future we might route those to 1-dimensional transforms—KDE)

…composing with the hexbin transform) * avoids a crash when there is no contour

(not entirely sure if it's good practice to have all the yarn.lock changes in the PR, beyond those that are relevant)

mbostock · 2022-06-24T16:35:35Z

Okay, I was able to rebase my PR to main, then this PR to my PR, and we’re back to me being able to review all the changes here. 😅 Thanks for being patient with me.

mbostock · 2022-06-24T16:39:46Z

src/marks/density.js

-              .thresholds(thresholds)
-            (index))
+        .call(g => g.selectAll()
+          .data(Array.from(index, i => [i]))


If we use applyChannelStyles instead of applyGroupedChannelStyles, we won’t need to do this. (I’ll make this change shortly.)

mbostock · 2022-06-24T16:43:47Z

src/marks/density.js

+    const W = channels.weight?.value;
+    const Z = channels.z?.value;
+    const {z} = this;
+    const [cx, cy] = applyFrameAnchor(options, dimensions);


applyFrameAnchor takes this as the first argument, not options. (I’ll fix this.)

mbostock · 2022-06-24T16:47:29Z

We might also want to support fillOpacity = density and strokeOpacity = density? Could be left as a future improvement though; definitely not urgent. (I’ll consider implementing this now.)

mbostock · 2022-06-24T16:56:28Z

src/marks/density.js

+    // normalize colors to a thresholds scale
+    const m = max * (maxn + 1) / maxn;
+    if (f) channels.fill.value = channels.fill.value.map(v => v / m);
+    if (s) channels.stroke.value = channels.stroke.value.map(v => v / m);


I’m thinking maybe we don’t do this, and just leave the units as “points per square pixel” or “value per square pixel” or perhaps “points per 1,000 square pixels” as a more human-readable value. Even though the value is arbitrary and resolution-dependent, it’s still probably more useful than 0 = 0, 1 = maximum observed density because it allows you to have a consistent scale across charts.

I guess it depends if you want to compare distributions or actual quantities, but I'm OK with reverting this. If needed, we could add the normalization as an option, but I'm not convinced it's that useful—I was just trying to make sense of the values.

100 square pixels is probably a better choice for a unit than 1000, as one can picture a 10x10 square hosting 1 or a few points.

Fil · 2022-06-24T17:06:57Z

I'm not sure about fillOpacity being useful, since the shapes are not exclusive bands the areas are overlaid and having a constant fillOpacity=0.1 is what results in a "transparent to opaque" display. (But not opposed to adding it for consistency.)

mbostock · 2022-06-24T18:12:41Z

src/marks/density.js

+    // Second pass: generate contours with the thresholds derived above
+    density.thresholds(thresholds);
+    for (const {facetIndex, index, c: memoc, top} of memo) {
+      const c = top < max ? density(index) : memoc;


I’m not sure how much this caching is helping; I think most commonly, only one series (or facet-series) will be cacheable as having the max threshold. All the others will need to be recomputed. I guess that’s still helpful in the common case of only a single series and single facet, but I suspect we could make this significantly faster if d3.contourDensity had a way to compute and store the blurred grid, and then compute individual contours for a particular threshold afterward. That way, we wouldn’t have to recompute the underlying grid when computing new contours, and we could cache individual contours rather than sets.

mbostock

I made some edits in abfd86e. To summarize:

For options that are only needed by the initializer, I moved the defaults and coercion into the initializer.
I fixed a bug where if the fill was specified as “density”, the default stroke is now “none” (not “currentColor”).
I removed this.path in favor of creating a path in render.
I added a comment explaining why mark.filter is overridden. I wonder if we also want to change how the channels are grouped: rather than picking the value associated with the first index for each facet-series, we should pick the first defined value? But this is probably fine as-is.
I adopted applyChannelStyles instead of applyGroupedChannelStyles to avoid constructing wrapper index arrays for each contour.
I added z to the set of channels that are not automatically grouped (since z is not needed to render the computed contours), and adopted a Set (since the explicit enumeration of channel names in the code was feeling a bit unwieldy with four named channels).
I adopted the coerceNumbers and identity technique used by the hexbin transform for the x, y, and weight channels.
I fixed the first argument to applyFrameAnchor.
I tweaked how the newChannels are constructed: it’s an object rather than an array, and I handle the fill and stroke = “density” cases separately.
I removed the use of constant(cx) and constant(cy), since d3-contour does that for us.
I removed then normalization of the color scale values (in the color = density case) and instead scaled by a fix factor of 100. Mostly because this is simpler and faster and I’m not really sure what we should do here.
I removed the caching logic in favor of skipping the threshold-finding pass when there’s only one facet-series. If we want to make this faster, we should instead consider making changes to the d3-contour API to avoid recomputing the blurred grid.
I forced the color scale to include zero in the color = density case, rather than the default which is that the color range will start at the smallest contour value.

* density mark * density contours stroke & fill (#948) * density contours as an initializer: first pass * density by z, carrying styles and channels (no reducer, only "first") * fill: "density" * density weight * consistent thresholds across facets and series note: we have to go around this bug in d3-contour: d3/d3-contour#57 * allow initializer composition; move initializer to the class; clean up * density weight example * error if x or y is undefined (in the future we might route those to 1-dimensional transforms—KDE) * document * reduce img * * don't apply the scales if we are already in pixel space (e.g. when composing with the hexbin transform) * avoids a crash when there is no contour * 1d density contours with frameAnchor * adopt d3@7.4.5 for https://github.com/d3/d3-contour/releases/tag/v3.0.2 (not entirely sure if it's good practice to have all the yarn.lock changes in the PR, beyond those that are relevant) * replace example image * mike’s edits * fix image dimensions * isDensity * distinct * tweak tests Co-authored-by: Mike Bostock <mbostock@gmail.com> Co-authored-by: Philippe Rivière <fil@rezo.net>

Fil · 2022-06-24T20:59:20Z

Superb. Thank you for the detailed list of changes!

Fil requested a review from mbostock June 22, 2022 15:06

Fil mentioned this pull request Jun 22, 2022

density mark #943

Merged

1 task

mbostock reviewed Jun 23, 2022

View reviewed changes

src/marks/density.js Outdated Show resolved Hide resolved

mbostock reviewed Jun 23, 2022

View reviewed changes

src/marks/density.js Outdated Show resolved Hide resolved

Fil force-pushed the fil/density-contours branch from 3484ec8 to 774191f Compare June 24, 2022 08:56

Fil changed the base branch from mbostock/density-contours to main June 24, 2022 08:56

mbostock reviewed Jun 24, 2022

View reviewed changes

mbostock force-pushed the fil/density-contours branch from 56a0e32 to 8dad905 Compare June 24, 2022 15:48

mbostock changed the base branch from main to mbostock/density-contours June 24, 2022 15:48

mbostock force-pushed the mbostock/density-contours branch from 8286adb to 22e478c Compare June 24, 2022 15:51

Fil and others added 14 commits June 24, 2022 11:54

density contours as an initializer: first pass

ff27647

density by z, carrying styles and channels (no reducer, only "first")

0448962

fill: "density"

9f34cda

density weight

39dad32

consistent thresholds across facets and series

d97b954

note: we have to go around this bug in d3-contour: d3/d3-contour#57

allow initializer composition; move initializer to the class; clean up

282bc03

density weight example

21d2b75

error if x or y is undefined

e0673aa

(in the future we might route those to 1-dimensional transforms—KDE)

document

e231d38

reduce img

175ed46

* don't apply the scales if we are already in pixel space (e.g. when …

0d206c2

…composing with the hexbin transform) * avoids a crash when there is no contour

1d density contours with frameAnchor

dcc9784

adopt d3@7.4.5 for https://github.com/d3/d3-contour/releases/tag/v3.0.2

09c20da

(not entirely sure if it's good practice to have all the yarn.lock changes in the PR, beyond those that are relevant)

replace example image

d327591

mbostock force-pushed the fil/density-contours branch from 8dad905 to d327591 Compare June 24, 2022 15:59

mbostock reviewed Jun 24, 2022

View reviewed changes

mike’s edits

abfd86e

mbostock force-pushed the fil/density-contours branch from dcadd28 to abfd86e Compare June 24, 2022 19:10

mbostock approved these changes Jun 24, 2022

View reviewed changes

mbostock added 4 commits June 24, 2022 15:24

fix image dimensions

452d148

isDensity

b59e720

distinct

a932e4e

tweak tests

269a70a

mbostock merged commit 563ce90 into mbostock/density-contours Jun 24, 2022

mbostock deleted the fil/density-contours branch June 24, 2022 20:38

Fil mentioned this pull request Apr 19, 2023

1-d KDE transform? #1469

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

density contours stroke & fill #948

density contours stroke & fill #948

Fil commented Jun 22, 2022 •

edited

Fil commented Jun 23, 2022

Fil commented Jun 24, 2022

Fil commented Jun 24, 2022

mbostock Jun 24, 2022

Fil Jun 24, 2022

mbostock commented Jun 24, 2022

mbostock commented Jun 24, 2022

mbostock Jun 24, 2022

mbostock Jun 24, 2022

mbostock commented Jun 24, 2022

mbostock Jun 24, 2022

Fil Jun 24, 2022

Fil commented Jun 24, 2022

mbostock Jun 24, 2022

mbostock left a comment

Fil commented Jun 24, 2022

density contours stroke & fill #948

density contours stroke & fill #948

Conversation

Fil commented Jun 22, 2022 • edited

Fil commented Jun 23, 2022

Fil commented Jun 24, 2022

Fil commented Jun 24, 2022

mbostock Jun 24, 2022

Choose a reason for hiding this comment

Fil Jun 24, 2022

Choose a reason for hiding this comment

mbostock commented Jun 24, 2022

mbostock commented Jun 24, 2022

mbostock Jun 24, 2022

Choose a reason for hiding this comment

mbostock Jun 24, 2022

Choose a reason for hiding this comment

mbostock commented Jun 24, 2022

mbostock Jun 24, 2022

Choose a reason for hiding this comment

Fil Jun 24, 2022

Choose a reason for hiding this comment

Fil commented Jun 24, 2022

mbostock Jun 24, 2022

Choose a reason for hiding this comment

mbostock left a comment

Choose a reason for hiding this comment

Fil commented Jun 24, 2022

Fil commented Jun 22, 2022 •

edited