Redesign "materialization" of data specs by mattpap · Pull Request #10235 · bokeh/bokeh

mattpap · 2020-06-27T08:49:13Z

This PR achieves several goals:

All data initialization is now explicit, i.e. no guessing in super-classes what properties a glyph may implement.
What can be a typed array is a typed array. This is to allow almost zero-copy data transfer between threads.
- Nested arrays are stored as flat RaggedArrays (typed arrays with offsets into subarrays).
- Colors are stored as packed 32-bit integers.
Transforms are computed in-place whenever it is allowed.

bryevdv · 2020-07-02T19:20:31Z

@mattpap How close is this to being ready? I would like to start in earnest on some WebGL experiments but this work definitely intersects relevant codepaths. I can branch of of this if you think it will be awhile (but then it would be nice if force-pushes can be avoided)

bryevdv · 2020-07-02T19:37:38Z

  }
+}
+
+export function $project_xsys(xs: Arrayable<number>[], ys: Arrayable<number>[], merc_xs?: Arrayable<number>[], merc_ys?: Arrayable<number>[]): void {


Does the $ signify non-allocating?

Yes, though I have mixed feelings about this notation.

My preference would be to make those as /** @internal */ (this has meaning in TS, by excluding such entities from generated declaration files), but that would make them untestable in the current setup. However, I've been experimenting with a different approach to compiling tests, in which internal declarations would be usable, so this may get resolved at some point.

I don't have any strong opinion about the notation. Is there a possibility that extension developers might want to be able to use the non-allocating version?

mattpap · 2020-07-02T19:41:31Z

How close is this to being ready?

It's feature ready, but it seems it breaks a few things. I can get this finished next few days, maybe even tomorrow, especially as it blocks my other work as well.

mattpap · 2020-07-03T08:33:54Z

On the other hand, I wonder what kind of changes you expect to be doing? Because, if I was doing this work, I would start from improving/replacing the existing webgl engine, within the existing integration framework, and then work on improving the integration itself. I also need to know the extent of your changes, because this PR is just one in a series of PRs that are going to touch this part of bokehjs (for color mapping, indexing, web workers, etc.).

bryevdv · 2020-07-03T17:23:10Z

On the other hand, I wonder what kind of changes you expect to be doing? Because, if I was doing this work, I would start from improving/replacing the existing webgl engine, within the existing integration framework, and then work on improving the integration itself. I also need to know the extent of your changes, because this PR is just one in a series of PRs that are going to touch this part of bokehjs (for color mapping, indexing, web workers, etc.).

@mattpap absolutely, I need to do a little experimentation to get to that point, and I just didn't want to start that process on top of code that is about to change in this PR. To give you an early idea of thoughts though: My main observation about the webgl is simply that the codepath branches off at an wild tangent in an awkward place and starts up a set of tasks that in a completely different structure than the canvas rendering. What would like to do is try to make the rendering steps more granular so that both webgl and and can as rendering can be described by them (though individual steps might end up as be no-ops on one or the other side, e.g. a step to do a data-to-screen mapping would be a no-op on the webgl side, because that mapping will be done in the shader with the gl program executes) But the point is to try to bring canvas and webgl rendering under some conceptual alignment (as much as possible).

bryevdv · 2020-07-06T19:02:13Z

 export class YCoordinateSeqSpec extends CoordinateSeqSpec { readonly dimension = "y" }

+export class XCoordinateSeqSeqSeqSpec extends CoordinateSeqSeqSeqSpec { readonly dimension = "x" }
+export class YCoordinateSeqSeqSeqSpec extends CoordinateSeqSeqSeqSpec { readonly dimension = "y" }


Just tossing out as an idea: what do you think of something like XCoordinateSeqSpec, XCoordinateSeq2Spec, XCoordinateSeq3Spec? I actually find SeqSeqSeqSpec harded to quickly visually parse

I will add this to the overall "naming things" issue.

bryevdv · 2020-07-06T19:15:00Z

+    ctx.lineJoin = join
+    ctx.lineCap = cap
+    ctx.lineDash = dash
+    ctx.lineDashOffset = offset


With these changes what is the need for explicit intermediate storage here (besides color and alpha)?

bryevdv · 2020-07-06T19:37:47Z


                # for positional arg properties, default=None means no default
-                default=Parameter.empty if default is None else default
+                default=Parameter.empty if default is None or is_field(default) else default


This condition is a bit circumspect, I had to think about it for a bit. Maybe update the comment to explain that "args" that are explicitly specified as field are intended to be required in glyph methods?

I changed this, so that arguments are processed from last to first. This way first non-default argument will make all following non-default, even if they have defaults (as it is now possible), thus avoiding gaps in positional arguments with and without defaults values.

bryevdv · 2020-07-06T20:04:05Z

      })
      const source = new ColumnDataSource({data: {x: [["a", "b"], ["b", "c"]], y: [1, 2]}})
-      p.vbar({x: {field: "x"}, top: {field: "y"}, source})
+      p.vbar({x: {field: "x"}, top: {field: "y"}, width: 0.1, source})


what is this addition for?

The test originally depended on null to 0 conversion, and produced zero width bars, which due to non-zero line width appeared to have non-zero width. The more I test bokehjs alone, the more cases like this get revealed.

I see, I thought there was supposed to be a default width of 1 for bars?

There is a default, but a single fat bar (of width 1) doesn't seem appropriate for this test. It looks like it's hiding something underneath the bar.

bryevdv · 2020-07-06T20:27:45Z

@mattpap Lot of great stuff here. I only had a few comments/questions. Do you have an updated ETA? The examples report looks good and the failing tests seem like are not much to fix up?

mattpap · 2020-07-06T20:35:41Z

Graphs are broken, because they circumvent the well defined initialization of glyphs. This seems to be the last thing to fix.

* Explicitly define glyph coordinates * Use NumberArray.set() in ImageURL * Add default values * Don't check for nulls in MultiLine._index_data() * Wrap only plain object default values * Make data projection explicit * Clean up HexTile._set_data() * Add support for ragged arrays * Encode dimension in coordinate specs * Explicit data materialization * Update incomplete unit tests * Update graph layout providers * Fail safe when a column is missing * Update incomplete glyph tests * Assert number arrays in corner cases * Fix glyphs' defaults * Explicit defaults in {V,H}Bar glyphs * Respect packed colors in webgl backend * Unify handling of colors and alpha * Use CoordinateSeqSeqSeqSpec for multi polygons * Make toStringTag static * Increase pixel allowance * Respect glyph protocol in graph renderer * Ignore early glyph positional defaults * Mark export selenium tests as flaky

github-actions · 2024-10-27T04:10:21Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

mattpap added status: WIP type: task tag: component: bokehjs labels Jun 27, 2020

mattpap added this to the 2.2 milestone Jun 27, 2020

bryevdv reviewed Jul 2, 2020

View reviewed changes

Comment thread bokehjs/src/lib/models/glyphs/glyph.ts

mattpap force-pushed the mattpap/materialize branch from a0bdc0f to fea6ac4 Compare July 5, 2020 13:07

bryevdv reviewed Jul 6, 2020

View reviewed changes

Comment thread bokehjs/src/lib/core/visuals.ts

bryevdv reviewed Jul 6, 2020

View reviewed changes

mattpap force-pushed the mattpap/materialize branch from 6fd226d to 82550f5 Compare July 7, 2020 10:47

mattpap added 11 commits July 8, 2020 19:11

Explicitly define glyph coordinates

7b2c9d5

Add default values

49fbf63

Don't check for nulls in MultiLine._index_data()

2511c9d

Wrap only plain object default values

5f03f5a

Make data projection explicit

6d12421

Clean up HexTile._set_data()

9196019

Add support for ragged arrays

8f3a4e0

Encode dimension in coordinate specs

f267b68

Explicit data materialization

1be6b51

Update incomplete unit tests

35ec95a

Update graph layout providers

e2c650c

mattpap added 11 commits July 8, 2020 19:11

Fail safe when a column is missing

b20dfb7

Use NumberArray.set() in ImageURL

66549c6

Update incomplete glyph tests

7f79d55

Assert number arrays in corner cases

bc46b45

Fix glyphs' defaults

84a3d68

Explicit defaults in {V,H}Bar glyphs

54d988e

Respect packed colors in webgl backend

361686b

Unify handling of colors and alpha

3569c22

Use CoordinateSeqSeqSeqSpec for multi polygons

e059b4a

Make toStringTag static

709eed9

Increase pixel allowance

fbf4f0c

mattpap force-pushed the mattpap/materialize branch from 82550f5 to 75dcee0 Compare July 8, 2020 20:23

Respect glyph protocol in graph renderer

5f1c988

mattpap force-pushed the mattpap/materialize branch from 75dcee0 to 5f1c988 Compare July 8, 2020 22:38

mattpap added 2 commits July 9, 2020 10:24

Ignore early glyph positional defaults

81aa324

Mark export selenium tests as flaky

d19f28c

mattpap added status: ready status: accepted reso: completed and removed status: WIP status: ready labels Jul 9, 2020

mattpap merged commit 06be48a into branch-2.2 Jul 9, 2020

mattpap deleted the mattpap/materialize branch July 9, 2020 12:29

github-actions Bot locked as resolved and limited conversation to collaborators Oct 27, 2024

Conversation

mattpap commented Jun 27, 2020

Uh oh!

bryevdv commented Jul 2, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattpap commented Jul 2, 2020

Uh oh!

Uh oh!

mattpap commented Jul 3, 2020

Uh oh!

bryevdv commented Jul 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryevdv Jul 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryevdv commented Jul 6, 2020

Uh oh!

mattpap commented Jul 6, 2020

Uh oh!

github-actions Bot commented Oct 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bryevdv commented Jul 3, 2020 •

edited

Loading

bryevdv Jul 6, 2020 •

edited

Loading