I3dm updates #4101

lasalvavida · 2016-07-08T12:59:28Z

This is the first set of spec migrations from #33.

The i3dm header now has 6 double-precision fields for translating and scaling quantized coordinates.

Each instance is now composed of x, y, z 16-bit quantized coordinates along with 2 oct-encoded vectors that form an orthonormal basis for rotation.

Sample models updated to demonstrate all of these features.

@pjcozzi, @lilleyse, can you take a look at this when you get a chance?

Tasklist:

Integrate any changes from the finalized #100
Regenerate composite tiles (I want to make sure that nothing else in #100 changes before I do this.)

pjcozzi · 2016-07-08T14:02:15Z

Sounds awesome, thanks @lasalvavida!

@lilleyse can you please do the first review?

pjcozzi · 2016-07-08T14:02:41Z

For CesiumGS/3d-tiles#33

lilleyse · 2016-07-08T15:33:47Z

Source/Scene/Instanced3DModel3DTileContent.js

@@ -192,15 +198,41 @@ define([
        var batchTableByteLength = view.getUint32(byteOffset, true);
        byteOffset += sizeOfUint32;

+        //>>includeStart('debug', pragmas.debug);
        var gltfByteLength = view.getUint32(byteOffset, true);


This line should be outside the pragma.

lilleyse · 2016-07-08T15:34:27Z

You'll need to update the composite tilesets as well.

lilleyse · 2016-07-08T15:36:58Z

That's all from me, I like the new instanced tileset!

lilleyse · 2016-07-08T15:43:13Z

Just a note here... it also looks like another 3d tiles test is failing:
replacement refinement - selects root when sse is not met and subtree is not refinable (3)

But it's unrelated to this PR.

lasalvavida · 2016-07-08T20:25:28Z

Updated @lilleyse, I'm still working on the composite tilesets

pjcozzi · 2016-07-10T15:50:50Z

Source/Scene/Instanced3DModel3DTileContent.js

+            AttributeCompression.octDecode(normalTwoX, normalTwoY, normalRight);
+
+            // Compute third normal
+            Cartesian3.cross(normalRight, normalUp, normalOut);


"Out" here is confusing since, if I am following correctly, we would usually call it "Up" here, e.g., east/north/up; however, "Up" is already used, and perhaps rightfully so.

I don't have a better idea now; will let you know if I come up with something.

pjcozzi · 2016-07-10T15:57:07Z

@lasalvavida very nice start here. This still needs:

Using TILES3D_SCALE and TILES3D_NON_UNIFORM_SCALE semantics in the batch table when available. Instanced 3D Model updates 3d-tiles#33 (comment)
Updating these semantics at runtime (e.g., with Cesium3DTileFeature) - or explicitly not allowing it. They do not need to be stored in the batch table textures since they are handled by the instancing code. @lilleyse can advise. Let me know if you need my input.

pjcozzi · 2016-07-10T16:00:40Z

@lasalvavida perhaps add a tasklist to the top of this PR with #4101 (comment), composite, and anything else that still needs to be done before merging.

lasalvavida · 2016-07-18T16:15:38Z

@pjcozzi, @lilleyse Can you take a look at the implementation changes from CesiumGS/3d-tiles#100?

Particularly the Cesium3DTileFeatureTableResources. The way I have it set up is that values can be provided in the JSON, or be referenced into the binary if byteOffset is present. Let me know what you think, and if we like that approach I'll roll it into the batchTableResources as well.

pjcozzi · 2016-07-18T16:46:58Z

Source/Scene/Instanced3DModel3DTileContent.js

-            Cartesian3.fromRadians(longitude, latitude, height, ellipsoid, position);
-            var modelMatrix = Transforms.eastNorthUpToFixedFrame(position);
+            var featureTableResources = new Cesium3DTileFeatureTableResources(featureTableJSON, featureTableBinary);
+            this.featureTableResources = featureTableResources;


Are you sure this.featureTableResources is needed? Can't the feature table just be created at load time for parsing into the proper data structures, e.g., creating Cesium primitives, and then discarded? This is a very heavy object and the memory consumption is not palatable to keep it for the lifetime of a tile.

pjcozzi · 2016-07-18T16:57:27Z

We might want to unit test Cesium3DTileFeatureTableResources separately. I'll let you know after the next round of edits.

lilleyse · 2016-07-18T18:29:12Z

Source/Scene/Cesium3DTileFeatureTableResources.js

+    /**
+     * @private
+     */
+    function Cesium3DTileFeatureTableResources(featureTableJSON, featureTableBinary) {


Right now the approach of generating a JS array that holds onto the property's binary data just seems too expensive. I think it would be easier if this class stored a TypedArray view for each property, and then when getting an instance's property it just indexes into the typed array. To avoid allocations, the getter can take an optional results array parameter.

This is also make it easier to work with Points3DTileContent which wants TypedArrays for the color and position properties.

Oh I see this comment: #4101 (comment) kind of alludes to that too.

@lasalvavida can we write unit tests that test this file in isolation without an entire tile?

lasalvavida · 2016-07-19T14:13:32Z

I went ahead and pushed some of these changes. Feel free to comment. I'm finishing up rewriting my tile generation code for the new spec, so we should have tests again shortly.

lilleyse · 2016-07-19T14:41:51Z

Source/Scene/Cesium3DTileFeatureTableResources.js

+        return arrayBuffer;
+    };
+
+    Cesium3DTileFeatureTableResources.prototype.getGlobalProperty = function(semantic, componentType, count) {


Would a global property ever need to be stored in binary? If not, this function could just return the JSON.

Probably not; I think that's probably better for making the implementation more streamlined

I don't think we should assume it will always be JSON. A global property could be a big array, even if it isn't per-feature data, so we shouldn't make the spec this limited unless there is a significant implementation reason, which I do not see here since we already need to handle binary for the other properties.

pjcozzi · 2016-07-20T02:05:19Z

@lilleyse let me know when this is ready for me to review.

lasalvavida · 2016-07-20T20:05:53Z

Updated with some test models for semantics. Once I have #4121, I'll add a test model for oct-encoding, and a test model with all semantics and update the tests.

lilleyse · 2016-08-10T21:25:16Z

Specs/Scene/Cesium3DTileFeatureTableResourcesSpec.js

+        var feature = featureTable.getProperty('TEST', 1, ComponentDatatype.UNSIGNED_BYTE, 2);
+        expect(feature).toEqual([2, 3]);
+    });
+});


For this file also test that the DeveloperErrors are hit.

lasalvavida · 2016-08-11T15:15:43Z

Updated. The composite tile samples have been updated as well. Once there's a verdict on the FeatureTable array issue, this should be ready to go.

pjcozzi · 2016-08-15T19:25:33Z

@lasalvavida please merge in 3d-tiles.

pjcozzi · 2016-08-15T19:37:43Z

Does the Sandcastle example cover most i3dm use cases? If not, we can add another one or two small tilesets like we did in #4183? We don't need exhaustive testing in the Sandcastle example, but it is good to cover command use cases for manual testing.

lasalvavida · 2016-08-15T19:39:17Z

Does the Sandcastle example cover most i3dm use cases?

The Sandcastle example only does basic instancing, but I do have test models in specs for all semantics, so it would be trivial to add one or two of those.

pjcozzi · 2016-08-15T19:42:04Z

I have the following test failures:

Scene/Composite3DTileContent rejects readyPromise on error
DeveloperError: glTF byte length is zero, i3dm must have a glTF to instance.

Scene/Composite3DTileContent renders composite
Expected undefined to be defined.

Scene/Composite3DTileContent renders composite of composite
Expected undefined to be defined.

pjcozzi · 2016-08-15T19:42:32Z

Does the Sandcastle example cover most i3dm use cases?
The Sandcastle example only does basic instancing, but I do have test models in specs for all semantics, so it would be trivial to add one or two of those.

Yes, please.

Think representative, not exhaustive.

pjcozzi · 2016-08-15T19:47:23Z

There are a lot of .i3dm files in this PR that are 100-400 KB. In the future, please please please focus on creating very small test files so we do not bloat the repo and we keep the tests really fast. For example, if a dataset for a test has an embedded texture, resize it to be small or if a tile has 16K instances change it to be 32 instances.

Short-term, is it possible to change the git history in this branch and use smaller i3dm files?

pjcozzi · 2016-08-15T19:52:22Z

Just those comments. Overall nice work here.

lasalvavida · 2016-08-15T19:54:59Z

Short-term, is it possible to change the git history in this branch and use smaller i3dm files?

Yup, no problem.

pjcozzi · 2016-08-19T01:20:31Z

What's left before this can be merged?

lasalvavida · 2016-08-19T14:10:10Z

I have all of my sample tilesets regenerated except for gltfExternal. I will have this updated shortly.
*edited

lasalvavida · 2016-08-19T18:13:28Z

Okay, tests should be passing now; and this should be ready for final review. As far as not bloating the git history, I think if you git merge --squash when you merge this, it will flatten out the history.

mramato · 2016-08-19T18:19:06Z

@lasalvavida you can squash yourself very easily. (assuming origin is the name of the official cesium remote)

Merge in origin/3d-tiles into i3dm-updates
Run git reset origin/3d-tiles (This will make your branch identical to 3d-tiles but with all of your changes edited locally.)
Commit everything providing a good overall description.
run git push --force to override the remote branch with your new squashed branch.

Let me know if you want me to stop by and walk you through it.

lasalvavida · 2016-08-19T18:25:50Z

Thanks @mramato. This is all squashed and ready for review.

pjcozzi · 2016-08-19T20:37:57Z

@lasalvavida @lilleyse is this ready to merge?

lilleyse · 2016-08-19T21:00:39Z

@lasalvavida I think you forgot to add Cesium3DTileFeatureTableResources back after squashing.

lasalvavida · 2016-08-22T13:42:33Z

Missing files re-added this should be ready for a look for real this time.

lilleyse · 2016-08-22T15:33:50Z

One minor suggestion - can you place the instanced models fully above the ground? The shadows look wrong at the moment.

Other that that, everything looks good.

lasalvavida · 2016-08-22T16:37:02Z

Updated, sample tiles are now on the ground.

pjcozzi · 2016-08-22T17:25:19Z

@lilleyse please merge when ready.

@lasalvavida nice work here!

pjcozzi mentioned this pull request Jul 8, 2016

Instanced 3D Model updates CesiumGS/3d-tiles#33

Closed

3 tasks

lilleyse reviewed Jul 8, 2016
View reviewed changes

pjcozzi reviewed Jul 10, 2016
View reviewed changes

lasalvavida mentioned this pull request Jul 11, 2016

I3dm updates CesiumGS/3d-tiles#100

Merged

lilleyse mentioned this pull request Jul 18, 2016

3D Tiles - Support tile transform #4130

Closed

3 tasks

pjcozzi reviewed Jul 18, 2016
View reviewed changes

lilleyse reviewed Jul 18, 2016
View reviewed changes

lilleyse reviewed Jul 19, 2016
View reviewed changes

lilleyse reviewed Aug 10, 2016
View reviewed changes

Squash commits so that test model revisions don't bloat the repository

99f6f38

lasalvavida force-pushed the i3dm-updates branch from c021627 to 99f6f38 Compare August 19, 2016 18:23

Revert bad change

12edc38

pjcozzi mentioned this pull request Aug 19, 2016

Add git squash instructions to guide #4219

Closed

Added missing files from squash

dedbdae

lasalvavida added 2 commits August 22, 2016 12:20

Instanced test models are now on the ground

b067c1b

Composite tiles are now on the ground

f4f9232

lilleyse merged commit 33b05c2 into CesiumGS:3d-tiles Aug 22, 2016

lilleyse deleted the i3dm-updates branch August 22, 2016 17:37

I3dm updates #4101

I3dm updates #4101

Conversation

lasalvavida commented Jul 8, 2016 • edited

pjcozzi commented Jul 8, 2016

pjcozzi commented Jul 8, 2016

Choose a reason for hiding this comment

lilleyse commented Jul 8, 2016

lilleyse commented Jul 8, 2016

lilleyse commented Jul 8, 2016

lasalvavida commented Jul 8, 2016

Choose a reason for hiding this comment

pjcozzi commented Jul 10, 2016

pjcozzi commented Jul 10, 2016

lasalvavida commented Jul 18, 2016 • edited

Choose a reason for hiding this comment

pjcozzi commented Jul 18, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lasalvavida commented Jul 19, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pjcozzi commented Jul 20, 2016

lasalvavida commented Jul 20, 2016

Choose a reason for hiding this comment

lasalvavida commented Aug 11, 2016

pjcozzi commented Aug 15, 2016

pjcozzi commented Aug 15, 2016

lasalvavida commented Aug 15, 2016

pjcozzi commented Aug 15, 2016

pjcozzi commented Aug 15, 2016

pjcozzi commented Aug 15, 2016

pjcozzi commented Aug 15, 2016

lasalvavida commented Aug 15, 2016

pjcozzi commented Aug 19, 2016

lasalvavida commented Aug 19, 2016 • edited

lasalvavida commented Aug 19, 2016

mramato commented Aug 19, 2016

lasalvavida commented Aug 19, 2016

pjcozzi commented Aug 19, 2016

lilleyse commented Aug 19, 2016

lasalvavida commented Aug 22, 2016

lilleyse commented Aug 22, 2016

lasalvavida commented Aug 22, 2016

pjcozzi commented Aug 22, 2016

lasalvavida commented Jul 8, 2016 •

edited

lasalvavida commented Jul 18, 2016 •

edited

lasalvavida commented Jul 19, 2016 •

edited

lasalvavida commented Aug 19, 2016 •

edited