Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Globe final PR #3963

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

Globe final PR #3963

wants to merge 21 commits into from

Conversation

HarelM
Copy link
Member

@HarelM HarelM commented Apr 10, 2024

Launch Checklist

Keeping this as draft right now, this PR will allow merging of the entire globe branch into main.

  • Confirm your changes do not include backports from Mapbox projects (unless with compliant license) - if you are not sure about this, please ask!
  • Briefly describe the changes in this PR.
  • Link to related issues.
  • Include before/after visuals or gifs if this PR includes visual changes.
  • Write tests for all new functionality.
  • Document any changes to public APIs.
  • Post benchmark scores.
  • Add an entry to CHANGELOG.md under the ## main section.

kubapelc and others added 7 commits March 15, 2024 13:32
* Port changes from main globe branch - basics

Fix minor issues so that it compiles.

* Fix PI redefinitions

* Fix stencil shader

* Port adaptation of raster layer for globe from main globe branch

* Add globe.html example from pheonor's repo

Minor changes (remove terrain, set initial zoom 0, change title and description)

* Better map projection parameter doc comment, warn when using unknown projection

* Mercator projectionData handles negative zoom correctly

* Comment clarification

* Fix spelling of "granularity"

* Add missing docs

* Convert ProjectionBase to an interface

* Do not leak GL object in globe projection error measurement, add a destroy method to projection

* Fix chrome performance warning, refactor error measurement

Warning fixed by changing ring buffer size to 1, making ring buffer pointless, so I removed it.

* Fix granularity capitalization

* Fix capitalization

* Fix typo

* Fix stencil mask triangle index order (this was causing failing render tests)

* Cleanup vertex shader projection interface

* Move projection creation function into its own file

* Remove getProjectionName

* Added comment for deduplicateWrapped

* Remove unused vertex-buffer-related code from image source

* Add globe raster layer render test

* More render tests - test transition to mercator

* Remove pointless test, add test descriptions

* Render test for rendering poles on globe

* SubdivisionGranularitySetting constructor takes an object

* Remove "defines" parameter from useProgram

* Refactor useProgram and Program constructor

* Properly format translatePosMatrix comment

* Refactor globe-specific code outside projection classes, remove stencil-specific granularity settings

* Refactor granularity settings to be more readable

* Minor refactor of ProjectionErrorMeasurement

* Refactor draw_raster.ts

* Move globe utility functions to utils.ts, use easeCubicInOut instead of smoothStep

* Simplify imports in globe.ts

* globe.ts refactor

* Move ProjectionErrorMeasurement to a separate file

* Refactor ProjectionErrorMeasurement

Change parseRGBA8float to a private static function, use isWebGL2 function instead of instanceof

* Refactor draw_raster.ts

* Refactor globe projection error measurement to not use Painter

* Painter.clearStencil creates custom ProjectionData instead of calling getProjectionData(null, null)

* Remove "deduplicateWrapped" functionality from source_cache.ts

* Globe projection no longer requires a map instance

* Painter doesn't pass `this` to `updateGPUdependent`

* isRenderingDirty is now a function

* Rename ProjectionBase to Projection

* Replace globeView property with setGlobeViewAllowed

* Add mercator and globe projection unit tests

* Remove tests that test for exact clipping planes

* Update build test with new bundle size

* isRenderingDirty is now a function
* Port changes from main globe branch - basics

Fix minor issues so that it compiles.

* Fix PI redefinitions

* Fix stencil shader

* Port adaptation of raster layer for globe from main globe branch

* Add globe.html example from pheonor's repo

Minor changes (remove terrain, set initial zoom 0, change title and description)

* Better map projection parameter doc comment, warn when using unknown projection

* Mercator projectionData handles negative zoom correctly

* Comment clarification

* Fix spelling of "granularity"

* Add missing docs

* Convert ProjectionBase to an interface

* Do not leak GL object in globe projection error measurement, add a destroy method to projection

* Fix chrome performance warning, refactor error measurement

Warning fixed by changing ring buffer size to 1, making ring buffer pointless, so I removed it.

* Fix granularity capitalization

* Fix capitalization

* Fix typo

* Fix stencil mask triangle index order (this was causing failing render tests)

* Cleanup vertex shader projection interface

* Move projection creation function into its own file

* Remove getProjectionName

* Added comment for deduplicateWrapped

* Remove unused vertex-buffer-related code from image source

* Add globe raster layer render test

* More render tests - test transition to mercator

* Remove pointless test, add test descriptions

* Render test for rendering poles on globe

* SubdivisionGranularitySetting constructor takes an object

* Remove "defines" parameter from useProgram

* Refactor useProgram and Program constructor

* Properly format translatePosMatrix comment

* Refactor globe-specific code outside projection classes, remove stencil-specific granularity settings

* Refactor granularity settings to be more readable

* Minor refactor of ProjectionErrorMeasurement

* Refactor draw_raster.ts

* Move globe utility functions to utils.ts, use easeCubicInOut instead of smoothStep

* Simplify imports in globe.ts

* globe.ts refactor

* Move ProjectionErrorMeasurement to a separate file

* Refactor ProjectionErrorMeasurement

Change parseRGBA8float to a private static function, use isWebGL2 function instead of instanceof

* Refactor draw_raster.ts

* Refactor globe projection error measurement to not use Painter

* Painter.clearStencil creates custom ProjectionData instead of calling getProjectionData(null, null)

* Remove "deduplicateWrapped" functionality from source_cache.ts

* Globe projection no longer requires a map instance

* Painter doesn't pass `this` to `updateGPUdependent`

* isRenderingDirty is now a function

* Rename ProjectionBase to Projection

* Replace globeView property with setGlobeViewAllowed

* Add mercator and globe projection unit tests

* Remove tests that test for exact clipping planes

* Update build test with new bundle size

* isRenderingDirty is now a function

* Fill, fill-extrusion, line layers, subdivision: Import changes from kubapelc/globe-vector branch

* Fix unit tests

* Subdivision: ensure consistent triangle winding order, fix unit tests

* Fix terrain

* Fix fill extrusion not working with terrain

* Fix typos

* Fix line gradient bug

* Subdivision: fix line ring handling

* Subdivision: fix unit test expecting an invalid line segment

* Fix fill-extrusion ring handling

* Fill-extrusion refactor and fix failing test

* Update terrain fill extrusion test expected image

* Render tests for fill, line and fill-extrusion for globe

* Move fillArrays function into a separate file

* Add vector globe example

* Remove changes for line and fill-extrusion layers to make the PR smaller

* Add unit tests for fillArrays()

* fillArrays unit test has better segment size limits

* Update build test build size

* Fix html example description

* Fix missing docs for granularity settings

* Rename globe fill render test tile source layer to "vector_tiles"

* Fix classifyRings comment format

* Move subdivisionGranularitySettingsNoSubdivision constant to a static readonly field, shorten the name

* Use `import type` for SubdivisionGranularitySetting where possible

* Fix typo

* Revert fill_attributes back to default exports

* Improve comment for scanline subdivision

* Subdivision: break up scanline subdivision function into more functions

* Move SubdivisionGranularitySetting into its own file

* Unit tests: use mock of MercatorProjection instead of the full class

* Add SegmentVector unit tests

* Subdivision: unit tests for poles, ring triangulation, fix bug in ring triangulation

* Subdivision: more pole unit tests

* Subdivision: fix wireframe generation, add unit test for wireframe

* Rename subdivisionGranularitySettings.ts to subdivision_granularity_settings.ts

* Move granularity settings registration to subdivision

* Update build size

* Rename `fillArrays` to `fillLargeMeshArrays`

* Move virtual buffers to a test util file

* Better warning for segments.ts vertex overflow

* Better comment for projection subdivision granularity

* Clarify mesh comparison in fill_large_mesh_arrays.test.ts

* Move mesh creating functions into a separate file, add tests for mesh comparison and grid creation

* Refactor and add better doc comment for `fillLargeMeshArrays`

* Refactor fill_large_mesh_arrays by removing duplicated code

* Move debug functions to mesh_utils.ts

* Unit tests: use StructArrays instead of VirtualVertexBuffer, etc.

* Subdivision: refactor

* Subdivision: rename subdivideFill to subdividePolygon, remove wireframe function

* Subdivision: throw when a vertex is outside int16 range

* Subdivision: refactor generatePoleQuad into a proper function

* Subdivision: add subdivision benchmark

* Subdivision: split scanline subdivision into smaller functions

* Remove wireframe generation function

* Subdivision: better doc comments for scanline subdivision

* Fix 'as any' in segment.ts

* Reuse condition in fill_large_arrays

* Deduplicate code in fill_large_arrays

* Subdivision: remove redundant function in tests

* Subdivision: improve scanline subdivision comments

* Subdivision: benchmark is not async

* Rename SegmentVector's invalidateLast to forceNewSegmentOnTextPrepare

* More tests for segment.ts

* Fix typo in forceNewSegmentOnNextPrepare

* Subdivision: more tests for fillLargeMeshArrays

* Subdivision: better comment in fillLargeMeshArrays
* Fix merge

* Import line layer changes from kubapelc/globe-vector

* Lines: shorten line_bucket.test.ts subdivision settings

* Lines: minor refactor

* Lines: update build size

* Lines: minor refactor
@HarelM HarelM marked this pull request as ready for review April 10, 2024 17:36
@HarelM
Copy link
Member Author

HarelM commented Apr 10, 2024

This is still in draft, but I wanted coverage support, which doesn't work well for some reason...

@codecov-commenter
Copy link

codecov-commenter commented Apr 10, 2024

Codecov Report

Attention: Patch coverage is 86.59287% with 327 lines in your changes are missing coverage. Please review.

Project coverage is 86.97%. Comparing base (d7cd701) to head (518041f).

✅ All tests successful. No failed tests found.

Files Patch % Lines
src/data/bucket/fill_extrusion_bucket.ts 9.09% 98 Missing and 2 partials ⚠️
src/render/subdivision.ts 91.39% 35 Missing and 16 partials ⚠️
src/geo/projection/globe.ts 88.20% 37 Missing and 9 partials ⚠️
src/geo/projection/mercator.ts 75.43% 32 Missing and 10 partials ⚠️
src/render/painter.ts 79.82% 20 Missing and 3 partials ⚠️
...o/projection/globe_projection_error_measurement.ts 88.69% 15 Missing and 4 partials ⚠️
src/data/bucket/circle_bucket.ts 70.58% 13 Missing and 2 partials ⚠️
src/symbol/symbol_layout.ts 29.41% 11 Missing and 1 partial ⚠️
src/render/draw_hillshade.ts 85.41% 4 Missing and 3 partials ⚠️
src/data/bucket/line_bucket.ts 70.00% 2 Missing and 1 partial ⚠️
... and 6 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3963      +/-   ##
==========================================
+ Coverage   86.88%   86.97%   +0.09%     
==========================================
  Files         242      250       +8     
  Lines       33044    34872    +1828     
  Branches     2011     2118     +107     
==========================================
+ Hits        28709    30329    +1620     
- Misses       3374     3542     +168     
- Partials      961     1001      +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@HarelM
Copy link
Member Author

HarelM commented Apr 10, 2024

@kubapelc I've rigged codecov to work on this PR, feel free to see which classes are covered well and which need more work.
I'll keep this PR open so that we can get up-to-date coverage information on every PR merge.

HarelM and others added 4 commits April 11, 2024 08:43
* Import changes for fill-extrusion from main vector globe branch

* Fill extrusion: refactor

* Fill extrusion: indent shader ifdefs

* Fill extrusion: add example

* Fill extrusion: update build size

* Move globe specific projection methods to projection interface

* Fix failing unit test

* Use vec3.clone() instead of manually copying vector components
* Import background layer changes from main vector globe branch

* Import hillshade layer changes from main vector globe branch

* Subdivision: explicit types

* Fix single-pixel seams in the oceans

* Add render test for background pattern on globe

* Refactor drawBackground

* Refactor drawHillshade

* Update build size

* Update globe background-pattern render test with results from CI

* Hillshade: refactor prepareHillshade

* Add a render test for fill layer seams fix
@HarelM HarelM marked this pull request as draft April 17, 2024 06:54
HarelM and others added 3 commits April 18, 2024 09:01
* Import changes for circle and heatmap layers from the main vector globe branch

* Minor refactors

* Update build size

* Use "/ 8.0" in shader instead of "* 0.125"

* Update shader comments

* Use a thin type instead of full Transform in projection

* Only import types in projection.ts

* getPixelScale and getCircleRadiusCorrection only need map center as argument

* Only import types where possible in projection classes

* Smaller refactors

* Fix failing unit test

* Add heatmap render test

* More explicit types in projection interface

* Globe plane equation is a vec4

* Fix wrong args in projection functions
@HarelM
Copy link
Member Author

HarelM commented May 15, 2024

@kubapelc, I've merged main into this branch.
I've left some "HM TODO" in places where I replaced some logic that was needed.
Please check this thoroughly.

@HarelM
Copy link
Member Author

HarelM commented May 15, 2024

Bahhh.... I messed this one up.
Let me know if you need me to do anything to resolve this, or if you are taking care of it...

HarelM and others added 3 commits May 15, 2024 16:16
* Import changes from main vector globe branch

* Fix import

* Remove unused code

* Remove unused imports

* Update build size test

* Remove unused function

* Add render test results for Debian

* Add another Debian render test variant

* Add more render test variants

* Hide collision boxes on the backfacing side of the globe

* Fix pitch-aligned texts getting hidden when their anchor is beyond horizon

* Update build size

* Fix merge

* Better comment in draw_collision_debug

* Update build size

The 10 kb size increase seems to come from the main branch

* Minor refactor

* Use explicit types, even for unused parameters

* Refactor screenspace path projection

* Refactor imports for projection.ts and collision_index.ts

* Fix import in collision_index.ts
@HarelM
Copy link
Member Author

HarelM commented May 20, 2024

@kubapelc what still missing for this branch besides the atmosphere and the style spec definition for globe?

@HarelM
Copy link
Member Author

HarelM commented May 20, 2024

Note that there were examples that were added, which will appear in the docs site.
For those to appear as expected there's a need to generate images for them using the generate image script.
The following are the warning from the build generation code:

WARNING -  Doc file 'examples/index.md' contains a link '../assets/examples/globe-fill-extrusion.png', but the target 'assets/examples/globe-fill-extrusion.png' is not found among documentation files.
WARNING -  Doc file 'examples/index.md' contains a link '../assets/examples/globe-vectortiles.png', but the target 'assets/examples/globe-vectortiles.png' is not found among documentation files.
WARNING -  Doc file 'examples/index.md' contains a link '../assets/examples/globe.png', but the target 'assets/examples/globe.png' is not found among documentation files.

See here:
https://github.com/maplibre/maplibre-gl-js/blob/main/docs/README.md#writing-examples

@kubapelc
Copy link
Contributor

what still missing for this branch besides the atmosphere and the style spec definition for globe?

Everything rendering related on my part is basically done and merged, what remains is adapting the rest of MapLibre to work with globe projection. Most notably controls need to be adapted for globe, then animations such as flyTo, placing custom HTML at a location, custom layer API for globe and for placing 3D objects onto the globe (as mentioned in the globe discussion), and in general everything that projects screen pixels to coordinates or vice versa. I'm working on it, it's a long list, but I think (hope) it should be much less complex than the rendering part...

Also #3887 should be implemented for the globe to be usable.

@kubapelc
Copy link
Contributor

I'll fix the globe examples.

@HarelM
Copy link
Member Author

HarelM commented May 20, 2024

Thanks for the info!
If you happen to resolve the following enhancement it will be great:

@HarelM
Copy link
Member Author

HarelM commented May 20, 2024

Please also take a look at the coverage report.
I think some unit tests will be helpful here to cover more of the code scenarios as render test are good but are not suitable for edge cases.
I would recommend aiming to 80-90% for each file so that the overall coverage would stay around 80-90%. Or less than 100 lines not covered as part of this PR.

kubapelc and others added 3 commits May 20, 2024 14:14
* Add example images

* Add "-" into example name

* Remove basic globe example
* HiSilicon fix: enable face culling whereever possible

(cherry picked from commit fe439a5)

* Improve circle layer performance by discarding empty pixels

(cherry picked from commit 266897d)

* HiSilicon fix: software clipping of polygon outlines

(cherry picked from commit 98167ba)

* HiSilicon fix: software clipping for line layer

(cherry picked from commit d521e95)

* HiSilicon fix: circle software clipping

(cherry picked from commit f2ed744)

* HiSilicon fix: enable backface culling for symbols

(cherry picked from commit 54e3632)

* Update build test

* Fix terrain using a mirrored projection matrix

* Fix typos

* Fix terrain coord textures being flipped vertically

* Update build size

* Fix rendering of images with face culling, fix image rendering near pole regions

* Add render test for images on a globe

* Update comment in circle.vertex.glsl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants