Skip to content

Commit

Permalink
Merge pull request #5 from javagl/tools-ts-implicit
Browse files Browse the repository at this point in the history
Tileset processing with pipelines
  • Loading branch information
javagl committed Apr 11, 2023
2 parents a06814c + 9cf6faf commit 93f8775
Show file tree
Hide file tree
Showing 174 changed files with 6,820 additions and 859 deletions.
8 changes: 8 additions & 0 deletions IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ Parts of the current implementation may still change. This page is only a short
- These are implementations of the `TilesetSource` and `TilesetTarget` interface (see `./src/tilesetData`), based on 3TZ or 3DTILES

- `./src/pipelines`: **Preliminary** classes for modeling "processing pipelines" for tilesets
- The `Pipeline` class describes the pipeline with its input and output, and contains one or more `TilesetStage` objects
- The `TilesetStage` describes an operation that is applied to the tileset as a whole, usually focussing on modifications of the tileset JSON object. It may contain one or more `ContentStage` objects
- The `ContentStage` is an operation that may be applied to tile content (i.e. "files") that are part of the tileset
- Instances of these classes may be created with the `Pipelines`, `TilesetStages`, and `ContentStages` classes, respectively
- A pipeline may be executed by a `PipelineExecutor`.

- `./src/spatial`: Basic classes for dealing with tree structures, specifically with quadtrees and octrees

Expand All @@ -73,10 +78,13 @@ Parts of the current implementation may still change. This page is only a short
- `TilesetCombiner`: Used to "inline" external tilesets into a single one
- `TilesetMerger`: Used to create one tileset that refers to others as external tilesets
- `TilesetUpgrader`: Upgrade a tileset to a newer version (many aspects unspecified here)
- The (abstract) `TilesetProcessor` class and the (concrete) `BasicTilesetProcessor` class offer an infrastructure for generic operations on the tilesets and their content. These classes serve as the basis for the implementation of the pipeline execution functionality.

- `./src/tilesets`: Utility functions for tileset operations
- `Tiles` for traversing (explicit!) tile hierarchies
- `Tilesets` offering convenience functions for `merge/combine/upgrade`
- `Contents` with utility functions related to tile `content` objects
- `Extensions` for handling extensions and extension declarations in tilesets (and glTF objects)

- `./src/traversal`: Classes for traversing tilesets
- NOTE: The `SubtreeModel`/`SubtreeMetadataModel` interfaces _might_ at some point be moved into `implicitTiling`, but are currently tailored for the use in the traversal classes, and should be considered to be an "implementation detail" here.
Expand Down
109 changes: 102 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,43 @@ npx ts-node ./src/main.ts combine -i ./specs/data/combineTilesets/input -o ./spe

Merge multiple tilesets into a single one that refers to the input tilesets as external tilesets.
```
npx ts-node ./src/main.ts merge -i ./specs/data/mergeTilesets/input/TilesetA -i ./specs/data/mergeTilesets/input/sub/TilesetA -o ./specs/data/mergeTilesets/output
npx ts-node ./src/main.ts merge -i ./specs/data/mergeTilesets/TilesetA -i ./specs/data/mergeTilesets/sub/TilesetA -o ./specs/data/mergeTilesets/output
```

#### upgrade

Upgrade a tileset to the latest 3D Tiles version.
```
npx ts-node ./src/main.ts upgrade -i ./specs/data/TilesetOfTilesets/tileset.json -o ./output/upgraded
```
The exact behavior of the upgrade operation is not yet specified. But when B3DM- and I3DM tile content in the input tileset uses glTF 1.0 assets, then the upgrade step will try to upgrade these assets to glTF 2.0.

#### convert

<sup>(This replaces the `databaseToTileset` and `tilesetToDatabase` commands)</sup>

Convert between tilesets and tileset package formats.
```
npx ts-node ./src/main.ts upgrade -i ./specs/data/TilesetOfTilesets/tileset.json -o ./output/TilesetOfTilesets.3tz
```

The input- and output arguments for this command may be

- The name of a directory that contains a `tileset.json` file (or the full path to a tileset JSON file)
- The name of a `.3tz` file
- The name of a `.3dtiles` file

The input may also be a `.zip` file that contains a `tileset.json` file.

#### databaseToTileset

Deprecated. This functionality is now offered via the `convert` command.

#### tilesetToDatabase

Deprecated. This functionality is now offered via the `convert` command.



### Command line tools for tile content

Expand Down Expand Up @@ -116,7 +150,7 @@ npx ts-node ./src/main.ts optimizeB3dm -i ./specs/data/Textured/batchedTextured.
This example optimizes the b3dm and compresses the meshes using Draco, with a high compression level.


### optimizeI3dm
#### optimizeI3dm

Optimize a i3dm using [gltf-pipeline](https://github.com/CesiumGS/gltf-pipeline/blob/main/README.md).
```
Expand All @@ -125,18 +159,79 @@ npx ts-node ./src/main.ts optimizeI3dm -i ./specs/data/instancedWithBatchTableBi
See [optimizeB3dm](#optimizeb3dm) for further examples.


### upgrade
### Pipeline

Upgrade a tileset to the latest 3D Tiles version.
Execute a sequence of operations that are described in a JSON file.

> **Note:** The pipeline execution feature is preliminary. Many aspects of the pipeline definition, including the JSON representation and the exact set of operations that are supported as parts of pipelines may change in future releases.
The basic structure of a pipeline JSON file is summarized here:

- A pipeline has an `input` and `output`, which are the names of a tileset directory or package
- A pipeline has an array of 'tileset stages'
- A tileset stage has a `name` and a `description`
- A tileset stage has an array of 'content stages'
- A content stage has a `name` and a `description`
- A content stage can carry information about the content types that it is applied to

A simple example pipline may therefore look like this:
```
npx ts-node ./src/main.ts upgrade -i ./specs/data/TilesetOfTilesets/tileset.json -o ./output/upgraded
{
"input": "./specs/data/TilesetOfTilesetsWithUris",
"output": "./output/TilesetOfTilesetsWithUris.3tz",
"tilesetStages": [
{
"name": "_b3dmToGlb",
"description": "Convert B3DM to GLB",
"contentStages": [
{
"name": "b3dmToGlb",
"description": "Convert each B3DM content into GLB"
}
]
}
]
}
```
The exact behavior of the upgrade operation is not yet specified. But when B3DM- and I3DM tile content in the input tileset uses glTF 1.0 assets, then the upgrade step will try to upgrade these assets to glTF 2.0.

The `name` of a tileset- or content stage can refer to a predefined set of operations that can be executed. If a `name` is not one of the known operations, it should start with an `_` underscore.

The `description` of a tileset- or content stage is intended as a human-readable summary, to be shown as log output.

The predefined operations largely correspond to the command-line functionality.

The known tileset stages are:

- `upgrade`: Upgrade the input tileset to the latest version. Details about what that means are omitted here.
- `combine`: Combine all external tilesets of the input tileset, to create a single tileset

The known content stages are:

- Compression:
- `gzip`: Apply GZIP compression to all files (with optional filters)
- `ungzip`: Uncompress all files that are compressed with GZIP
- Conversion:
- `glbToB3dm`: Convert all GLB tile contents into B3DM
- `glbToI3dm`: Convert all GLB tile contents into I3DM (with the GLB being the only instance)
- `b3dmToGlb`: Convert all B3DM tile contents into GLB (assuming that the B3DM is only a wrapper around GLB)
- `i3dmToGlb`: Convert all I3DM tile contents into GLB (assuming that the I3DM is only a wrapper around GLB)
- `separateGltf`: Convert all GLB tile contents into `.gltf` files with external resources
- Optimization:

These operations receive an `options` object, which is an untyped object carrying the options that are passed to `gltf-pipeline` for the optimization.
- `optimizeGlb`: Optimize GLB tile content, using `gltf-pipeline`
- `optimizeB3dm`: Optimize the GLB payload of a B3DM tile content, using `gltf-pipeline`
- `optimizeI3dm`: Optimize the GLB payload of a I3DM tile content, using `gltf-pipeline`

An example of a pipeline that combines a sequence of multiple operations is shown in [`examplePipeline.json`](./specs/data/pipelines/examplePipeline.json).



---

**Draft** demos for the library usage:
## Demos

The `demos` folder contains some examples of how the functionality of the tools may be used as a library. This is intended as a preview. The functionality is not yet exposed as a public API.

### General tool functions

Expand Down
207 changes: 207 additions & 0 deletions demos/BinaryMetadataDemos.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
import { ClassProperty } from "../src/structure/Metadata/ClassProperty";
import { BinaryPropertyTables } from "../src/metadata/binary/BinaryPropertyTables";
import { BinaryPropertyTableModel } from "../src/metadata/binary/BinaryPropertyTableModel";

/**
* A test for the `BinaryPropertyTableModel` class.
*
* It creates a (binary) property table that contains a single
* property with the given structure and the given values.
* From that, it creates a `BinaryPropertyTable`, and goes
* through all its rows (as `MetadataEntityModel` instances),
* and prints the values that this entity model has for
* the given property.
*
* (These should be the same as the given input property values)
*
* @param name - A name for log messages
* @param classProperty - The `ClassProperty`
* @param propertyValues - The property values
*/
function runPropertyTableModelTest(
name: string,
classProperty: ClassProperty,
propertyValues: any
) {
const count = propertyValues.length;
const arrayOffsetType = "UINT32";
const stringOffsetType = "UINT32";
const binaryPropertyTable =
BinaryPropertyTables.createBinaryPropertyTableFromProperty(
"testProperty",
classProperty,
propertyValues,
arrayOffsetType,
stringOffsetType,
undefined
);
const propertyTableModel = new BinaryPropertyTableModel(binaryPropertyTable);
console.log("For " + name);
console.log(" Original values: " + JSON.stringify(propertyValues));
for (let i = 0; i < count; i++) {
const entity0 = propertyTableModel.getMetadataEntityModel(i);
const value0 = entity0.getPropertyValue("testProperty");
console.log(` Value from MetadataEntity ${i}: ` + JSON.stringify(value0));
}
}

/**
* Calls `runPropertyTableModelTest` for various class property types
*/
function runPropertyTableModelTests() {
const example_variable_length_FLOAT32_SCALAR_array = {
type: "SCALAR",
componentType: "FLOAT32",
array: true,
};
const example_fixed_length_FLOAT32_SCALAR_array = {
type: "SCALAR",
componentType: "FLOAT32",
array: true,
count: 5,
};
const example_STRING = {
type: "STRING",
};
const example_variable_length_STRING_array = {
type: "STRING",
array: true,
};
const example_fixed_length_STRING_array = {
type: "STRING",
array: true,
count: 5,
};
const example_BOOLEAN = {
type: "BOOLEAN",
};
const example_variable_length_BOOLEAN_array = {
type: "BOOLEAN",
array: true,
};
const example_fixed_length_BOOLEAN_array = {
type: "BOOLEAN",
array: true,
count: 5,
};

const example_variable_length_UINT32_VEC2_array = {
type: "VEC2",
componentType: "UINT32",
array: true,
};
const example_fixed_length_UINT32_VEC2_array = {
type: "VEC2",
componentType: "UINT32",
array: true,
count: 5,
};

const example_variable_length_FLOAT32_SCALAR_array_values = [
[-1.0, -0.5, 0.0, 0.5, 1.0],
[-1.0, 0.0, 1.0],
];
const example_fixed_length_FLOAT32_SCALAR_array_values = [
[-1.0, -0.5, 0.0, 0.5, 1.0],
[1.0, 2.0, 3.0, 4.0, 5.0],
];
const example_STRING_values = ["This is a test", "This is another test"];
const example_variable_length_STRING_array_values = [
["This", "is", "a", "test"],
["Another", "test"],
];
const example_fixed_length_STRING_array_values = [
["zero", "one", "two", "three", "four"],
["A", "B", "C", "D", "E"],
];
const example_BOOLEAN_values = [true, false];
const example_variable_length_BOOLEAN_array_values = [
[true, false, true, false],
[false, true, false],
];
const example_fixed_length_BOOLEAN_array_values = [
[true, false, true, false, true],
[false, true, false, true, false],
];

const example_variable_length_UINT32_VEC2_array_values = [
[
[0, 1],
[2, 3],
[4, 5],
],
[
[6, 7],
[8, 9],
],
];
const example_fixed_length_UINT32_VEC2_array_values = [
[
[0, 1],
[2, 3],
[4, 5],
[6, 7],
[8, 9],
],
[
[10, 11],
[12, 13],
[14, 15],
[16, 17],
[18, 19],
],
];

runPropertyTableModelTest(
"example_fixed_length_STRING_array",
example_fixed_length_STRING_array,
example_fixed_length_STRING_array_values
);
runPropertyTableModelTest(
"example_variable_length_BOOLEAN_array",
example_variable_length_BOOLEAN_array,
example_variable_length_BOOLEAN_array_values
);
runPropertyTableModelTest(
"example_fixed_length_UINT32_VEC2_array",
example_fixed_length_UINT32_VEC2_array,
example_fixed_length_UINT32_VEC2_array_values
);
runPropertyTableModelTest(
"example_variable_length_STRING_array",
example_variable_length_STRING_array,
example_variable_length_STRING_array_values
);
runPropertyTableModelTest(
"example_fixed_length_FLOAT32_SCALAR_array",
example_fixed_length_FLOAT32_SCALAR_array,
example_fixed_length_FLOAT32_SCALAR_array_values
);
runPropertyTableModelTest(
"example_STRING",
example_STRING,
example_STRING_values
);
runPropertyTableModelTest(
"example_variable_length_FLOAT32_SCALAR_array",
example_variable_length_FLOAT32_SCALAR_array,
example_variable_length_FLOAT32_SCALAR_array_values
);
runPropertyTableModelTest(
"example_BOOLEAN",
example_BOOLEAN,
example_BOOLEAN_values
);
runPropertyTableModelTest(
"example_fixed_length_BOOLEAN_array",
example_fixed_length_BOOLEAN_array,
example_fixed_length_BOOLEAN_array_values
);
runPropertyTableModelTest(
"example_variable_length_UINT32_VEC2_array",
example_variable_length_UINT32_VEC2_array,
example_variable_length_UINT32_VEC2_array_values
);
}

runPropertyTableModelTests();

0 comments on commit 93f8775

Please sign in to comment.