TypeScript rewrite #16

javagl · 2023-02-13T16:48:29Z

An initial draft of a possible 3D Tiles Tools rewrite. Further details will be added here soon.

Update (2023-04-12)

The previous update summarized the main part of the rewrite, with two further PRs still being pending. These PRs have been

They have been reviewed and merged into the ts-tools staging branch in the meantime.

Similarly, the following, additional PRs have been created, reviewed and merged into the the staging branch:

Both contain further, detailed descriptions of the changes. The README.md and IMPLEMENTATION.md in this branch have been updated to reflect these changes as well.

Previous updates:

Update (2023-03-14) :

The main part here has reviewed, and the summary of that first review is in the comment below. But for many reasons, it probably makes sense to not hastily merge this PR into main, but instead, wait until certain anticipated features are implemented, some futher cleanups have been made, and there have been more tests on actual data sets.

In the meantime, there are some PRs that are supposed to be added here, using the tools-ts branch as a sort of "staging" branch:

Cleanups, basic tests, comments...

Added basic specs for core functionality Ported/updated legacy specs Moved all spec data outuput to spec/data/output Started basic specs for 'upgrade' functionality

javagl · 2023-02-28T16:25:03Z

The current state is a rewrite based on TypeScript that replicates most of the previous command-line functionality.

The supported command line functions are described in the README at https://github.com/javagl/3d-tiles-tools/blob/tools-ts/README.md

Implementation notes for the current structure can be found at https://github.com/javagl/3d-tiles-tools/blob/tools-ts/IMPLEMENTATION.md

The test coverage (both in terms of specs as well as in terms of manual "integration tests" based on real data sets) has to be increased. There also are some future developments for extended functionality that are already on the radar. These can be described in dedicated issues and handled as dedicated PRs.

lilleyse · 2023-03-06T17:17:19Z

I wonder if the CLI should catch the error and print the message instead of showing a DeveloperError.

$ npx ts-node ./src/main.ts ungzip -i ./output/TilesetOfTilesets-gzipped/ -o ./output/TilesetOfTilesets-notgzipped/
C:\Users\slilley\Code\3d-tiles-tools\src\ToolsMain.ts:284
    throw new DeveloperError(
          ^
DeveloperError: File output\TilesetOfTilesets-notgzipped\ already exists. Specify -f or --force to overwrite existing files.
    at Function.ensureCanWrite (C:\Users\slilley\Code\3d-tiles-tools\src\ToolsMain.ts:284:11)
    at Function.ungzip (C:\Users\slilley\Code\3d-tiles-tools\src\ToolsMain.ts:184:15)
    at runCommand (C:\Users\slilley\Code\3d-tiles-tools\src\main.ts:196:21)
    at run (C:\Users\slilley\Code\3d-tiles-tools\src\main.ts:159:9)
    at Object.<anonymous> (C:\Users\slilley\Code\3d-tiles-tools\src\main.ts:212:1)
    at Module._compile (node:internal/modules/cjs/loader:1126:14)
    at Module.m._compile (C:\Users\slilley\Code\3d-tiles-tools\node_modules\ts-node\src\index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
    at Object.require.extensions.<computed> [as .ts] (C:\Users\slilley\Code\3d-tiles-tools\node_modules\ts-node\src\index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1004:32)
    at Function.Module._load (node:internal/modules/cjs/loader:839:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at phase4 (C:\Users\slilley\Code\3d-tiles-tools\node_modules\ts-node\src\bin.ts:649:14)
    at bootstrap (C:\Users\slilley\Code\3d-tiles-tools\node_modules\ts-node\src\bin.ts:95:10)
    at main (C:\Users\slilley\Code\3d-tiles-tools\node_modules\ts-node\src\bin.ts:55:10)
    at Object.<anonymous> (C:\Users\slilley\Code\3d-tiles-tools\node_modules\ts-node\src\bin.ts:800:3) {
  toString: [Function (anonymous)]
}

lilleyse

Just some initial comments from looking at the README.

lilleyse · 2023-03-06T16:22:00Z

README.md

+Quantize floating-point attributes and oct-encode normals:
+```
+npx ts-node ./src/main.ts optimizeB3dm -i ./specs/data/batchedWithBatchTableBinary.b3dm -o ./output/optimized.b3dm --options -q -n
+```
+
 To use tileset texture compression, pass the [`texcomp` flags](https://github.com/CesiumGS/gltf-pipeline/blob/main/README.md#command-line-flags)
 ```
 node ./bin/3d-tiles-tools.js optimizeB3dm -i ./specs/data/Textured/batchedTextured.b3dm -o ./output/optimized.b3dm --options --texcomp.dxt1.enable --texcomp.dxt1.quality=5 --texcomp.etc1.enable
 ```
 This example optimizes the b3dm and compresses the textures into `dxt1` and `etc1` formats.


A Draco example would be better since these options no longer exist in gltf-pipeline >= 2.0.0

I changed that to be a draco example, but also noticed that there had been some issues with passing the options to gltf-pipeline: The old version exposed a parseArguments function that was applied to the "raw" string[] to obtain the actual options object. I now tried to "emulate" that, with some quirks (the name is draco.foo at the CLI, but dracoOptions.foo in the actual options object...).

In a test with the example that is now in the README,

npx ts-node ./src/main.ts optimizeB3dm -i ./specs/data/Textured/batchedTextured.b3dm -o ./output/optimized.b3dm --options --draco.compressMeshes --draco.compressionLevel=9

the output indeed was a B3DM where the GLB contained the Draco part, so that seems to work now...

lilleyse · 2023-03-06T16:46:48Z

src/main.ts

+    "upgrade",
+    "Upgrades the input tileset to the latest version of the 3D Tiles spec. Embedded glTF models will be upgraded to glTF 2.0.",
+    { i: inputStringDefinition }


Should upgrade be included in the README?

I added an example, with the disclaimer that The exact behavior of the upgrade operation is not yet specified....

lilleyse · 2023-03-06T16:48:53Z

README.md


-* To debug the tests in Webstorm, open the Gulp tab, right click the `test` task, and click `Debug 'test'`.
-* To run a single test, change the test function from `it` to `fit`.


Are there any building, testing, or generating documentation instructions that should still be included in the README?

I extended the package.json with the same commands as in the validator, and added notes in IMPLEMENTATION.md accordingly

lilleyse · 2023-03-06T16:52:35Z

package.json

-  "homepage": "https://github.com/CesiumGS/3d-tiles-tools",
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/CesiumGS/3d-tiles-tools.git"
-  },
-  "bugs": {
-    "url": "https://github.com/CesiumGS/3d-tiles-tools/issues"
-  },
-  "main": "index.js",
-  "engines": {
-    "node": ">=16.0.0"
-  },


Will these fields be added back prior to publishing on npm?

lilleyse · 2023-03-06T17:37:44Z

@javagl is this roughly what's left after this PR is merged? Anything big I'm missing?

Analyze tile data: Add command to analyze tile data #17
Reworked pipeline functionality
Finish tileset traversal
Support for implicit tiling
Any more changes from 2.0-tools, 2.0-tools-fix, 2.0-tools-hack, 2.0-tools-unlit that need to be incorporated into the upgrade stage?
Publish on npm
Tileset statistics stage (number of tiles, etc)

Generalize tileset processing for building pipelines

Extended pipeline processing

Tileset processing with pipelines

lilleyse

@javagl Awesome work here, this will be a much better foundation for all our 3D Tiles tooling needs. I was only able to scratch the surface in my review and just left some comments for the README, which you can address and then merge.

In the near future I'm looking forward to the 3D Tiles 1.0 to 1.1 upgrade path and deciding on the public API.

README.md

lilleyse · 2023-04-14T02:05:20Z

README.md

+  - `gzip`: Apply GZIP compression to all files (with optional filters)
+  - `ungzip`: Uncompress all files that are compressed with GZIP


Do gzip/ungzip also apply to the root tileset.json? Since they're content stages I wasn't sure.

Yes. We could try to find a better term than "content" here. It's all files/entries of a tileset data set.

Well, this is embarassing... :-( #22

I'm currently trying to see whether it makes sense to convert them into tileset stages. Not only would that avoid confusion, it would also resolve a bunch of questions about the special handling of the tileset.json when they are supposed to be a "content stage"...

Co-authored-by: Sean Lilley <lilleyse@gmail.com>

javagl added 16 commits February 13, 2023 17:03

Initial draft of TypeScript rewrite

242f459

Updates from usage in validator

1485002

Cleanups, basic tests, comments...

Update gitignore and include omitted spec data

3950cae

Cleanups and generalizations for content type detection

f7d62fe

Updates for content types, pipeline drafts - highly WIP

1dc0ad0

Cleanups. Emulate most of original CLI.

4a073a3

Cleanup for the demos. Move benchmarks to own directory.

9bbddad

Get rid of unnnecessary defined checks and assertions

e667bd4

Trying to revive 'optimize' from gltf-pipeline

50f61be

Fix missing await in TilesetCombiner

4f86d9e

Add function to upgrade tileset in place

a9b187e

Demand output at CLI

be3dbb9

Update tileset processing demos to use Tilesets class

5afd1cf

Updated README with current tools and demos

6133c75

Cleanups and additions for specs...

d7edca6

Added basic specs for core functionality Ported/updated legacy specs Moved all spec data outuput to spec/data/output Started basic specs for 'upgrade' functionality

Cleanups and tests. Mainly for 'upgrade'.

2e67c47

javagl force-pushed the tools-ts branch from 1390689 to 2e67c47 Compare February 28, 2023 16:22

javagl marked this pull request as ready for review February 28, 2023 16:29

javagl added 2 commits March 1, 2023 16:22

Removed debug output

4348245

Added first draft of 'analyze' command

9d443d2

javagl mentioned this pull request Mar 1, 2023

Add command to analyze tile data #17

Closed

javagl added 2 commits March 3, 2023 18:03

Fix option check in TilesetUpgrader

9823100

Merge branch 'tools-ts' into tools-ts-analyze

32fd11d

lilleyse reviewed Mar 6, 2023

View reviewed changes

javagl added 3 commits March 6, 2023 22:18

Update and clean up README.md - add upgrade step.

96a2d19

Update implementation notes with build instructions

d758e92

Update package JSON for build, docs and packaging

748ffd9

javagl and others added 18 commits April 9, 2023 16:30

Extended tests, mainly implicit tileset processing

d09f18c

Merge pull request #3 from javagl/tools-ts-implicit-processing

6ed6728

Generalize tileset processing for building pipelines

Removed duplicate call in demo

953e02d

Removed another duplicate method in demo

b6dfb5c

Cleanups for pipeline experiments demo

1ecc983

Minor clarification in demo comment

dd55620

Basic specs for packages as inputs and outputs

89d9028

Add option to set temp directory

fb6191b

Cleanups for demos

98ca36f

More content operations. Update of CLI.

3c7f8da

Update README and implementation notes

74c7344

Minor cleanups in demos

f97245d

Clean up imports

97dd025

Minor documentation updates

86776d4

Basic specs for implicit tilings

3c6314e

Update API definition file

33624ef

Add example pipelines

384fe41

Merge pull request #4 from javagl/tools-ts-implicit-processing-pipelines

9cf6faf

Extended pipeline processing

This was referenced Apr 11, 2023

Tileset processing with pipelines javagl/3d-tiles-tools#5

Merged

Update validator to use 3D Tiles Tools as dependency CesiumGS/3d-tiles-validator#270

Merged

javagl and others added 6 commits April 11, 2023 20:04

Merge pull request #5 from javagl/tools-ts-implicit

93f8775

Tileset processing with pipelines

Update TSDoc format

e309bd9

Update command line help

d13594e

Added .npmignore

94e1aff

Update README and implementation notes

628635f

Added analyze command description in README

5572377

lilleyse approved these changes Apr 14, 2023

View reviewed changes

javagl and others added 2 commits April 14, 2023 14:33

Update README.md

f25ba1a

Co-authored-by: Sean Lilley <lilleyse@gmail.com>

Removed outdated example

3a5853a

javagl merged commit 4406ef6 into CesiumGS:main Apr 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeScript rewrite #16

TypeScript rewrite #16

javagl commented Feb 13, 2023 •

edited

Loading

javagl commented Feb 28, 2023 •

edited

Loading

lilleyse commented Mar 6, 2023

lilleyse left a comment

lilleyse Mar 6, 2023

javagl Mar 6, 2023

lilleyse Mar 6, 2023

javagl Mar 6, 2023

lilleyse Mar 6, 2023

javagl Mar 6, 2023

lilleyse Mar 6, 2023

javagl Mar 6, 2023

lilleyse commented Mar 6, 2023

lilleyse left a comment •

edited

Loading

lilleyse Apr 14, 2023

javagl Apr 14, 2023

javagl Apr 21, 2023


		* To debug the tests in Webstorm, open the Gulp tab, right click the `test` task, and click `Debug 'test'`.
		* To run a single test, change the test function from `it` to `fit`.

		- `gzip`: Apply GZIP compression to all files (with optional filters)
		- `ungzip`: Uncompress all files that are compressed with GZIP

TypeScript rewrite #16

TypeScript rewrite #16

Conversation

javagl commented Feb 13, 2023 • edited Loading

javagl commented Feb 28, 2023 • edited Loading

lilleyse commented Mar 6, 2023

lilleyse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilleyse commented Mar 6, 2023

lilleyse left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javagl commented Feb 13, 2023 •

edited

Loading

javagl commented Feb 28, 2023 •

edited

Loading

lilleyse left a comment •

edited

Loading