feat(core): Remove redundant instance picking colors

[ibgreen-openai]

Summary

WebGL2 and WebGPU shaders can derive picking colors directly from builtin gl_InstanceID / instance_index variables, so no need to create and initialize instancePickingColors for most layers.

This removes the default instancePickingColors buffer for layers where the value is just the standard encoded instance index. while layers that need non-instance logical ids keep explicit picking color attributes.

First of 3 planned PRs

• 1. Remove instancePickingColors
• 2. switch from pickingColors to pickingIndexes buffers
• 3. Adopt luma.gl multi-render-target picking module (removes the current 256 layer picking limit)

pickMultipleObjects keeps the old buffer-mutation behavior for explicit picking color buffers. Builtin-index layers instead pass the already-picked object ids as a small disabled-index uniform list during repeated picking passes.

Changes

• Adds picking shader helpers for deriving standard picking colors from builtin instance ids.
• Removes core auto-registration of instancePickingColors.
• Migrates 1:1 instanced layer shaders to builtin instance picking.
• Preserves explicit instancePickingColors/pickingColors paths for logical feature ids, grouped text/path ids, and tessellated per-vertex picking.
• Updates custom layer docs and core tests for the new default behavior.

Layer Name	Type	instance_index	Comment
ScatterplotLayer	instanced	✅	Falls back to explicit instancePickingColors only when binary data supplies it.
IconLayer	instanced	✅	Falls back to explicit instancePickingColors only when binary data supplies it.
PointCloudLayer	instanced	✅	Standard 1:1 instance picking.
LineLayer	instanced	✅	Standard 1:1 instance picking.
ArcLayer	instanced	✅	Standard 1:1 instance picking.
ColumnLayer	instanced	✅	Standard 1:1 instance picking.
GridCellLayer	instanced	✅	Standard 1:1 instance picking.
HexagonCellLayer	instanced	✅	Standard 1:1 instance picking.
ScreenGridCellLayer	instanced	✅	Standard 1:1 instance picking.
SimpleMeshLayer	instanced	✅	Standard 1:1 instance picking.
ScenegraphLayer	instanced	✅	Standard 1:1 instance picking.
TextBackgroundLayer	instanced	✅	One background per text object.
CARTO RasterColumnLayer	instanced	✅	Already lays cells out from gl_InstanceID.
MeshLayer	instanced	✅	Uses instance_index unless pickFeatureIds is enabled.
PathLayer	instanced	❌	Keeps instancePickingColors; path segments can map to source object ids.
MultiIconLayer / text characters	instanced	❌	Keeps instancePickingColors; glyph instances map back to text labels.
GeoJSON binary point sublayers	instanced	❌	Keep supplied colors for global feature ids.
GeoJSON binary line / outline sublayers	instanced	❌	Keep supplied colors for global feature ids.
SolidPolygonLayer	primitive	N/A	Keeps pickingColors; tessellated vertices need source-object colors.
BitmapLayer	primitive	N/A	Does not need default instancePickingColors.

Validation

• npx tsc -p modules/core/tsconfig.json --noEmit
• npx tsc -p modules/layers/tsconfig.json --noEmit
• npx tsc -p modules/aggregation-layers/tsconfig.json --noEmit
• npx tsc -p modules/mesh-layers/tsconfig.json --noEmit
• npx tsc -p modules/geo-layers/tsconfig.json --noEmit
• npx tsc -p modules/carto/tsconfig.json --noEmit
• yarn -s lint
• yarn -s test --project node test/modules/core/shaderlib/picking.spec.ts test/modules/core/lib/layer.spec.ts test/modules/core/lib/layer-extension.spec.ts
• yarn -s test --project node test/modules/core/lib/pick-layers.spec.ts test/modules/core/lib/picking.spec.ts test/modules/layers/geojson-layer.spec.ts test/modules/layers/bitmap-layer.spec.ts

[coveralls]

coverage: 83.67% (-0.08%) from 83.746% — ib/remove-instance-picking-colors into master

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3b2c86d40

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[felixpalmer]

could do mat4 disabledPickingIndices instead?

[ibgreen-openai]

Hmm, perhaps, that would give us a few more. They could probably be an open ended array at the end of the buffer if we really wanted to generalized.
I'll consider it as I keep working on this.