Add initial COPC and LAS 1.4 support

[connormanning]

This PR adds support for COPC and LAS (and LAZ) 1.4 data, including EPT data which uses LAS 1.4 as its data storage. Some sample COPC data can be found in the Data tool here. Some key things in the code are as follows.

This work was accomplished by Hobu Inc during the OSGeo Community Code Sprint 2023.

Examples

After npm start, some very small embedded test datasets are here:

• COPC: http://localhost:1234/examples/copc.html
• EPT with LAZ 1.4 encoding: http://localhost:1234/examples/ept_laz_14.html

And you can link out to more substantial data:

• SoFI Stadium COPC (360M point COPC, ~2GB)
• Hobbs NM EPT (24B point EPT)

copc.js

Lots of functionality has been offloaded to the copc.js library which has utilities for reading LAS/LAZ/COPC/EPT data and is fairly well unit-tested (PRs graciously accepted). Parsing LAS headers, loading hierarchies, decompressing point data, and extracting the attributes via views are all accomplished via copc.js. Since copc.js is distributed as an ES module, I used copc-umd to generate a UMD bundle for use by this project.

Code

Geometry

The COPC/EPT geometries are quite similar, and the identical functionality has been extracted into a BaseGeometry. The COPC/EPT geometries parse metadata into a common format and create their point data loaders, and specify the mechanism for loading a hierarchy page: these are just slightly different methods from copc.js.

The GeometryNode for both formats is the same since the differing functionality is encapsulated in the above. Perhaps it should have a better name since it applies to both formats.

Loader

The EptLaszipLoader has been updated to use the most recent laz-perf/WASM (via copc.js). The decompression of the actual laszip data has also been moved out of the loader (previously it was here) and into the web worker. The CopcLaszipLoader is quite similar except that it passes a compressed range of data from a COPC file rather than a full laszip file.

Worker

The web worker where laszip data is decompressed and extracted into buffers looks very different. However this wasn't purely for stylistic reasons: the attributes are pulled out more generically to allow seamless support for extra-byte attributes (compare with previous code which has a strict set of attributes). However this code is not currently adding extra-byte data to the output at the moment, though they could trivially be added to the map and extracted in the loop the same way as the existing attributes.

Bug?

I may have introduced a bug where nodes are destroyed with dispose(), but the renderer still tries to render them. This would cause a crash because the node is missing its geometry after disposal. I made some attempts to fix it and now I'm having trouble reproducing it, but I'm not fully convinced it's fixed. I've added this console.log, which is triggered immediately before crashing. Usually this would occur when zooming out, and in my experience it has always been on a level 1 node like 1-1-0-1 or similar. So I'm not positive that this PR is production ready, but should be 99% of the way there.

See also

• Completes Add COPC Support #1065
• Relevant to Potree 2.0 considerations #756

[jo-chemla]

Incredible to see these two pieces come together! The Potree viewer being one of the most used implementation for tiled pointclouds, and COPC being the open-standard made to store and stream massive tiled pointclouds.
Note @m-schuetz this might be a great direction when you mentioned supporting extra per-point attributes for Potree-Next.

Also note: PotreeConverter v2 is also the fastest pointcloud tiler I've come across - would be interesting to redo the benchmark. Would also be great if pdal copc conversion could use the same kind of optimization - or if potreeconverter could have an option to export to COPC.

[m-schuetz]

Awesome, thanks for this work! It will take me a couple of days to dig through it. Too bad I wasn't able to be at the code sprint, in my home-town of all places.

[m-kuhn]

Great work, thank you so much for this @connormanning, I didn't actually realize you were working on this when we talked back at the codesprint.

I just tested it on a 27GB COPC and it worked perfectly fine.

[m-kuhn]

Note: seems it works if I load it with npm start but not if served as static files from a server, in this case it will just try to download the complete file apparently. Note: It may also be me doing something wrong..

[Rdataflow]

@m-kuhn just guessing, maybe those commands didn't run successfully on your box?

npm run build
npm run postinstall

nb. on my box static files seems to work

[m-schuetz]

Nice work, haven't encountered any issues so far so I'd be ready to merge. Just wondering about one thing:

The COPC example you've linked loads with 4 concurrent http requests, while the EPT example only loads with 1 http request at a time. Is this a difference in some settings in the code? Or could it be due to server configs? I think I've had cases where loading data from a server from a different domain limited the amount of concurrent requests to 1.

SoFI Stadium COPC (360M point COPC, ~2GB)

Hobbs NM EPT (24B point EPT)

[Rdataflow]

The COPC example you've linked loads with 4 concurrent http requests, while the EPT example only loads with 1 http request at a time. Is this a difference in some settings in the code? Or could it be due to server configs? I think I've had cases where loading data from a server from a different domain limited the amount of concurrent requests to 1.

just guessing: could be related to server providing http/2 connection (multiplexing)

[m-kuhn]

Indeed I had not run the npm run postinstall, thank you.

It loads and shows fine, but I still experience an "infinite load" setting, where data continues to be downloaded (visible in the developer tab).

[Rdataflow]

It loads and shows fine, but I still experience an "infinite load" setting, where data continues to be downloaded (visible in the developer tab).

odd to observe in your implementation the request has Range: bytes=0- 👀 ... thus consequently it will load the entire file

[Rdataflow]

It loads and shows fine, but I still experience an "infinite load" setting, where data continues to be downloaded (visible in the developer tab).

odd to observe in your implementation the request has Range: bytes=0- 👀 ... thus consequently it will load the entire file

true, I can reproduce it (most of the times). @connormanning this looks to me like a bug

[connormanning]

Thanks for the feedback, all. I will take a look at:

• single-threaded EPT
• full file download request when run from the compiled version, rather than the dev version

Neither of these are the expected behavior so they both sound like bugs to me.

[Rdataflow]

The COPC example you've linked loads with 4 concurrent http requests, while the EPT example only loads with 1 http request at a time. Is this a difference in some settings in the code? Or could it be due to server configs? I think I've had cases where loading data from a server from a different domain limited the amount of concurrent requests to 1.

Situation

• https://s3.amazonaws.com/hobu-lidar/sofi.copc.laz is served via HTTP/1.1 (handles 1 request per connection)
• https://na-c.entwine.io/coh/ept.json is served via HTTP/2 (allows multiplexing multiple requests through 1 single connection)

Investigation

• Adding a HTTP/2 capable reverse proxy in front of the former file shows a single connection pattern too (as expected for HTTP/2 multiplexing)
  

Steps to reproduce
run HTTP/2 capable nginx server with this snippet

    location /hobu-lidar/sofi.copc.laz {
        proxy_pass https://s3.amazonaws.com/hobu-lidar/sofi.copc.laz;
    }

Conclusion

• single or multi connection pattern is only related to HTTP/1.1 or HTTP/2 protocol support on serverside

cc @connormanning

[Rdataflow]

It loads and shows fine, but I still experience an "infinite load" setting, where data continues to be downloaded (visible in the developer tab).

This reproduces in both: dev mode (npm start) as well as in static served versions.
The infinite loading is observed in chromium based browsers (chrome, edge) with range: bytes=0- but is being blocked in firefox trying range: bytes=0--1
It seems to occur only with a single COPC prepared by @m-kuhn - maybe an unexpected file structure not being caught?

cc @connormanning

[m-schuetz]

single or multi connection pattern is only related to HTTP/1.1 or HTTP/2 protocol support on serverside

I've once had the same data set on the same server serve with 1 or 6 parallel requests.

• 6 parallel requests if the potree page was on the same domain as the data
• 1 parallel request if the potree page was on my local PC and loading from that server.
• 6 parallel requests if the potree page was on my local PC and loading from a local webserver.

So I was wondering if your results may be due to the configuration of the proxy limiting CORS requests, rather than the difference between http 1 or 2?

[connormanning]

Yes, this must be something to do with the CloudFront proxy. If you use the direct S3 link to the same data here, the pattern looks as expected.

One note: even with the proxied version, I think the data is being fetched concurrently. It shows up as a single line in the overview, but when I enabled network throttling so the requests were very slow, I could see that requests were actively downloading data concurrently. Both while the requests were occurring, and also afterward I could see that the timestamps for the download portion of various requests were overlapping.

So I don't think there's an issue here. I'll look at the "whole file download" issue next.

[connormanning]

Fixed here. COPC allows point data nodes to be empty, but the code was not handling this. When this happens, the node info looks like:

pointCount: 0
pointDataOffset: 0
pointDataLength: 0

Of course, in this case we do not want to try to fetch/decompress the point data which does not exist, but the previous code was computing a range request from 0 to -1 in this case since range requests are inclusive so we must subtract 1 from the end of the range.

So both issues mentioned here by users should be resolved - feel free to post if anything else comes up.

[Rdataflow]

I can confirm the whole file download is gone with your fix 👍

[klakar]

Got this to work on a local web server. "Old" ept.json and .copc.laz files work the same without major issues.
Sometimes the GUI loads without point cloud, but a page refresh usually solves that.

On mobile (Android) however, I can't get any point clouds to show... (not with COPC or Ept)
I've tried on phone and tablet, but on the same web server, so it might be the server that has some issues?
"Stock" Potree with ept works fine on mobile...

(On iOS it shows, but no colors. Chrome on Android will show point cloud in VR but not in normal view...)

[m-schuetz]

Gave it a try on my Android Phone (Galaxy S20FE; Opera Browser) and it works, so I'll merge the PR!

Thanks @connormanning and @hobu !