Skip to content

Commit

Permalink
Merge acaad2d into b842628
Browse files Browse the repository at this point in the history
  • Loading branch information
@ibgreen
ibgreen committed Oct 17, 2019
2 parents b842628 + acaad2d commit 09bec3c
Show file tree
Hide file tree
Showing 28 changed files with 4,213 additions and 747 deletions.
18 changes: 13 additions & 5 deletions .eslintrc.js
View file
Expand Up @@ -2,12 +2,20 @@
module.exports = {
plugins: ['react'],
extends: ['uber-jsx', 'uber-es2015', 'prettier', 'prettier/react', 'plugin:import/errors'],
overrides: {
files: ['*.spec.js', 'webpack.config.js', '**/bundle/*.js'],
rules: {
'import/no-extraneous-dependencies': 0
overrides: [
{
files: ['*.spec.js', 'webpack.config.js', '**/bundle/*.js'],
rules: {
'import/no-extraneous-dependencies': 0
}
},
{
files: ['**/test/**/*.js'],
rules: {
'import/no-unresolved': 0
}
}
},
],
settings: {
'import/core-modules': [
'@luma.gl/core',
Expand Down
7 changes: 7 additions & 0 deletions modules/gpu-table/README.md
View file
@@ -0,0 +1,7 @@
# @deck.gl/gpu-table (Experimental)

> This module will be moved out of deck.gl once it stabilizes, and will only be present in deck.gl 8.0-alpha releases
The `GPUTable` class manages binary columnar tables in GPU memory.

See [deck.gl](http://deck.gl) for documentation.
110 changes: 110 additions & 0 deletions modules/gpu-table/docs/arrow-mapping.md
View file
@@ -0,0 +1,110 @@
# Arrow Mapping

## Schemas

Arrow supports simple schemas, along the lines of

```js
const schema = [
{name: 'longitude', metadata: },
]
```

As will be seen below, the Apache Arrow type system, while quite sophisticated, does not directly support some key column types contain all the information needed to determine how to map Arrow table columns to GPU attributes. The missing information can be provided using Arrow column metadata, through the `schema` that is part of every Arrow table.

Apache Arrow specification itself does not define the names or semantics of any actual metadata fields. Instead, it is the `GPUTable` class that assigns semantics to a selection of metadata fields, to describe how various columns should be mapped to attributes.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| | | | |


The fact that the `metadata` object is an official part of the Apache Arrow format gives applications the ability to annotate Arrow tables either:
- server-side, at generation time, before tables are written to disk or streamed to the client
- client side, just after reading them but before passing them to the `GPUTable` class.

Key Features
- Preparation of vec2, vec3, vec4s from individual columns into an interleaved column.

Options for handling short arrays:
- Use arrow metadata?
- Use arrow structs?

## Arrow to GLSL Type Map

| Metadata Type | Arrow Type | Metadata | Data Texture | Comment |
| --- | --- | --- | --- | --- |
| `Float32` | `Float32` | | | |
| `Float32[2]` | `FixedSizeBinary(8)` | type: `GL.FLOAT`, size: 2 | | |
| `Float32[3]` | `FixedSizeBinary(12)` | type: `GL.FLOAT`, size: 3 | | |
| `Float32[4]` | `FixedSizeBinary(16)` | type: `GL.FLOAT`, size: 4 | | |
| `Float64` | `Float64` | | |
| `Float64`[2] | FixedSizeBinary(16) | type: `GL.DOUBLE`, size: 2 | | |
| `Float64`[3] | FixedSizeBinary(24) | type: `GL.DOUBLE`, size: 3 | | |
| `Float64`[4] | FixedSizeBinary(32) | type: `GL.DOUBLE`, size: 4 | | |
| `int` | Int32 (Int16, Int8) | | | |
| `ivec2` | FixedSizeBinary(8) | | | |
| `ivec3` | FixedSizeBinary(12) | | | |
| `ivec4` | FixedSizeBinary(16) | | | |
| `unsigned` | Uint32 (Uint16, Uint8) | | | |
| `uvec2` | FixedSizeBinary(8) | | | |
| `uvec3` | FixedSizeBinary(12) | | | |
| `uvec4` | FixedSizeBinary(16) | | | |
| `bool` | TBD | | | |
| `bvec2` | TBD | | | |
| `bvec3` | TBD | | | |
| `bvec4` | TBD | | | |
| `mat2` | FixedSizeBinary(16) | `vec4` | | |
| `mat3` | FixedSizeBinary(36) | `vec4`* 3 | | |
| `mat4` | FixedSizeBinary(64) | `vec4`* 4 | | |
| `dmat2` | FixedSizeBinary(32) | `vec4`* 2 | | |
| `dmat3` | FixedSizeBinary(72) | `vec4`* 6 | | TBD - Would take a huge chunk of attribute bank |
| `dmat4` | FixedSizeBinary(128) | `vec4`* 8 | | TBD - Would take a huge chunk of attribute bank |


| `mat`n`x`m | FixedSizeBinary(n*m*4) | TBD | | |
| `dmat`n`x`m | FixedSizeBinary(n*m*8) | TBD | | |

## Arrow to GLSL Type Map


| GLSL Type | Arrow Type | Attributes | Data Texture | Comment |
| --- | --- | --- | --- | --- |
| `float` | Float32 | | | |
| `vec2` | FixedSizeBinary(8) | | | |
| `vec3` | FixedSizeBinary(12) | | | |
| `vec4` | FixedSizeBinary(16) | | | |
| `double` | Float64 | | | |
| `dvec2` | FixedSizeBinary(16) | | | |
| `dvec3` | FixedSizeBinary(24) | | | |
| `dvec4` | FixedSizeBinary(32) | | | |
| `int` | Int32 (Int16, Int8) | | | |
| `ivec2` | FixedSizeBinary(8) | | | |
| `ivec3` | FixedSizeBinary(12) | | | |
| `ivec4` | FixedSizeBinary(16) | | | |
| `unsigned` | Uint32 (Uint16, Uint8) | | | |
| `uvec2` | FixedSizeBinary(8) | | | |
| `uvec3` | FixedSizeBinary(12) | | | |
| `uvec4` | FixedSizeBinary(16) | | | |
| `bool` | TBD | | | |
| `bvec2` | TBD | | | |
| `bvec3` | TBD | | | |
| `bvec4` | TBD | | | |
| `mat2` | FixedSizeBinary(16) | `vec4` | | |
| `mat3` | FixedSizeBinary(36) | `vec4`* 3 | | |
| `mat4` | FixedSizeBinary(64) | `vec4`* 4 | | |
| `dmat2` | FixedSizeBinary(32) | `vec4`* 2 | | |
| `dmat3` | FixedSizeBinary(72) | `vec4`* 6 | | TBD - Would take a huge chunk of attribute bank |
| `dmat4` | FixedSizeBinary(128) | `vec4`* 8 | | TBD - Would take a huge chunk of attribute bank |
| `mat`n`x`m | FixedSizeBinary(n*m*4) | TBD | | |
| `dmat`n`x`m | FixedSizeBinary(n*m*8) | TBD | | |

- GPU allows integer columns to be exposed as floats
- GPU allows integer columns exposed as floats to be normalized

### Handling of 64 bit data

On GLSL versions that do not support `double` and `dvec*` data types, double data types are split into two sets of attributes, one with the high 32 bit float and one with the suffix `_64low`.

Note that the splitting of `Float64` into two `Float32` values is done on the CPU as the attribute is uploaded to the GPU. The two values for each float are interleaved in memory are exposed as two separate attributes using `stride`.

70 changes: 70 additions & 0 deletions modules/gpu-table/docs/gpu-column.md
View file
@@ -0,0 +1,70 @@
# GPUColumn (Internal)

> This class is internal and not intended to be directly used by applications.
This class helps deck.gl manage attributes. It integrates into the luma.gl `Model.setGPUColumns()` method by implementing the `GPUColumn.getValue()` method. luma.gl checks for the presence of this method on any attribute passed in.


## Usage

Create model object by passing shaders, uniforms, geometry and render it by passing updated uniforms.

```js
import {GPUColumn} from '@deck.gl/gpu-table';
```

```js
// construct the model.
const positions = new GPUColumn({
id: 'vertexPositions',
size: 3,
value: new Float32Array([0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0])
});

// and on each frame update any uniforms (typically matrices) and call render.
model.setGPUColumns({positions});
model.draw();
```

## Methods

### constructor

The constructor for the GPUColumn class. Use this to create a new GPUColumn.

`new GPUColumn(gl, options);`

* `gl` - WebGL context.
* `size` (*number*) - The number of components in each element the buffer (1-4).
* `id` (*string*, optional) - Identifier of the attribute. Cannot be updated.
* `type` (*GLenum*, optional) - Type of the attribute. If not supplied will be inferred from `value`. Cannot be updated.
* `isIndexed` (*bool*, optional) - If the attribute is element index. Default `false`. Cannot be updated.
* `constant` (*bool*, optional) - If the attribute is a constant. Default `false`.
* `isInstanced` (*bool*, optional) - Whether buffer contains instance data. Default `false`.
* `normalized` (*boolean*, optional) - Default `false`
* `integer` (*boolean*, optional) - Default `false`
* `offset` (*number*, optional) - where the data starts in the buffer. Default `0`.
* `stride` (*number*, optional) - an additional offset between each element in the buffer. Default `0`.
* `value` (*TypedArray*) - value of the attribute.
- If `constant` is `true`, the length of `value` should match `size`
- If `constant` is `false`, the length of `value` should be `size` multiplies the number of vertices.
* `buffer` (*Buffer*) - an external buffer for the attribute.


### delete

Free WebGL resources associated with this attribute.


### update

```js
attribute.update({value: newValue});
```

Update attribute options. See `constructor` for possible options.


### getBuffer

Returns a `Buffer` object associated with this attribute, if any.

0 comments on commit 09bec3c

Please sign in to comment.