Skip to content

Commit

Permalink
Edits based on comments
Browse files Browse the repository at this point in the history
  • Loading branch information
tsherif committed Jul 3, 2019
1 parent 5175681 commit 13864ec
Showing 1 changed file with 30 additions and 58 deletions.
88 changes: 30 additions & 58 deletions dev-docs/RFCs/v7.3/data-management-rfc.md
Expand Up @@ -12,85 +12,57 @@ This RFC specifies subsystem for managing data in luma.gl and creating gl resour

## Background

Buffer and texture management in luma.gl is currently done ad hoc manner, with the application required to build these resources from data and to decide when and how they can be re-used. In deck.gl this has lead to the complexity of the `AttributeManager` that creates buffers from input data and manages their lifecycles.
Buffer and texture management in luma.gl is currently done in an ad hoc manner, with the application required to build these resources from data and to decide when and how they can be re-used. In deck.gl this has lead to the complexity of the `AttributeManager` that creates buffers from input data and manages their lifecycles.

The lack of a global mechanism for managing data resources also has performance implications in that buffers and textures can be allocated to store redundant data. In deck.gl this might occur due to multiple layers being created from the same data (where each layer would allocate its own gl resources) or multiple instances of the same layer each creating their own layer-specific geometry buffers or texture data. The deck.gl `TileLayer` uses a `TileCache` to avoid redundantly creating textures, and a similar mechanism could be applied more generally to avoid allocating textures and buffers if like resources with the same data have already been created anywhere in the application.


## Overview

The proposed data manager would allow the application to register arbitrary named data sets, optionally providing accessors to parse particular views of the data. It would provide API methods to create buffers or textures from the
registered data, returning cached versions if they are available.
The proposed data manager would allow the application to register arbitrary named data sets, optionally providing accessors to parse particular views of the data. It would provide API methods to create buffers or textures from the registered data, returning cached versions if they are available.


## Implementation

A `DataManager` class that supports the following methods:
- `addData(id, data, accessors)`: add a data set to the manager, optionally with accessors to create particular views.
- `getBuffer(dataId, viewId)`: create a buffer from a data set view. On first call, create the buffer and cache it, subsequently return cached version and increment usage count. If `viewId` is omitted, the entire data set is used.
- `getTexture(dataId, viewId)`: create a texture from a data set view. On first call, create the texture and cache it, subsequently return cached version and increment usage count. If `viewId` is omitted, the entire data set is used.
- `release(Buffer|Texture)`: indicate that a resource is no longer used. Decrements the usage count for the resource and deletes it if the usage count reaches 0.
- `addData(id, data)`: add a data set to the manager.
- `getBuffer(dataId|size)`: if `dataId` is provided, create a buffer from a data set, if `size` is provided create an empty buffer of the given size. On first call, create the buffer and cache it, subsequently return cached version and increment usage count.
- `getResizedBuffer(buffer, size)`: create a buffer of the given size with starting data copied from the given `buffer`. On first call, create the buffer and cache it, subsequently return cached version and increment usage count.
- `getUpdatedBuffer(buffer, dataId, offset)`: create a copy of the given buffer, data from the data set inserted at the given `offset`. On first call, create the buffer and cache it, subsequently return cached version and increment usage count.
- `getCopiedBuffer(srcBuffer, dstBuffer, srcOffset, dstOffset, bytes)`: create copy of `dstBuffer`, with `bytes` bytes of data copied from `srcBuffer` at `srcOffset` to `dstOffset`. On first call, create the buffer and cache it, subsequently return cached version and increment usage count.
- `getTexture(dataId)`: create a texture from a data set. On first call, create the texture and cache it, subsequently return cached version and increment usage count.
- `release(buffer|texture)`: indicate that a resource is no longer used. Decrements the usage count for the resource and deletes it if the usage count reaches 0.


## Example

```js
const dm = new DataManager(gl);

dm.addData('myTable', [
{
position: [0.4, 1.0],
color: [255, 0, 255]
},
{
position: [1.0, 2.0],
color: [0, 255, 255]
},
{
position: [-1.0, 3.0],
color: [255, 0, 0]
}
], {
positions: (data) => {
const result = new Float32Array(data.length * 2);
for (let i = 0, len = data.length; i < len; ++i) {
result.set(data[i].position, i * 2);
}
return result;
},
colors: (data) => {
const result = new Uint8Array(data.length * 3);
for (let i = 0, len = data.length; i < len; ++i) {
result.set(data[i].color, i * 3);
}
return result;
}
});

dm.addData('myImages', [
image1,
image2
], {
image1: (data) => data[0],
image2: (data) => data[1]
});

dm.addData('anotherImage', image3); // Accessors not provided because the only view is the entire data set

const positionBuffer = dm.getBuffer('myTable', 'positions');
const positionBuffer2 = dm.getBuffer('myTable', 'positions'); // Gets cached version of same buffer
const colorBuffer = dm.getBuffer('myTable', 'colors');
const texture1 = dm.getTexture('myImage', 'image1');
const texture2 = dm.getTexture('myImage', 'image2');
const texture3 = dm.getTexture('myImage', 'image2'); // Gets cached version of same texture
const texture4 = dm.getTexture('anotherImage'); // Omit view ID to use entire data set
dm.addData('myPositions', floatArray);
dm.addData('myColors', byteArray);
dm.addData('positionUpdate', updateArray);
dm.addData('myImage', image);

const positionBuffer = dm.getBuffer('myPositions');
const positionBuffer2 = dm.getBuffer('myPositions'); // Gets cached version of same buffer
const colorBuffer = dm.getBuffer('myColors');
const emptyBuffer = dm.getBuffer(256); // Create and cache an empty buffer of 256 bytes
const resizedBuffer = dm.getResizedBuffer(positionBuffer, 1024); // Create and cache a new 1024-byte buffer beginning with the contents of positionBuffer
const updatedBuffer = dm.getUpdatedBuffer(positionBuffer, 'positionUpdate', 64); // Create a new copy of positionBuffer with positionUpdate data inserted starting at byte 64
const copiedBuffer = dm.getCopiedBuffer(positionBuffer, emptyBuffer, 64, 128, 32); // Copy 32 bytes starting from byte 64 in positionBuffer to a copy of emptyBuffer starting at byte 128

const texture1 = dm.getTexture('myImage');
const texture2 = dm.getTexture('myImage'); // Gets cached version of same texture

dm.release(positionBuffer); // Still in memory, usage count decremented to 1
dm.release(positionBuffer2); // Delete buffer
dm.release(colorBuffer);
dm.release(texture1);
dm.release(texture2); // Still in memory, usage count decremented to 1
dm.release(texture3); // Delete texture
dm.release(texture4);
dm.release(emptyBuffer);
dm.release(resizeBuffer);
dm.release(updatedBuffer);
dm.release(copiedBuffer);
dm.release(texture1); // Still in memory, usage count decremented to 1
dm.release(texture2); // Delete texture

```

0 comments on commit 13864ec

Please sign in to comment.