Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test] Progressive rendering for big dataset #5

Open
wants to merge 36 commits into
base: master
Choose a base branch
from

Conversation

lixun910
Copy link
Member

@lixun910 lixun910 commented Nov 3, 2023

Test progressive rendering based on the batches returned from loaders.gl. This could be used to load a really big dataset.
Base on loaders.gl v4 and geoarrow work.

  • no impact on small or medium-sized dataset
  • instantaneous rendering of the first batch
  • rendering while loading more batches

Note:

  • Arrow: batch size can be set by -lco BATCH_SIZE=65536 (rows)

Loading ~1 million polygons
incremental-render

Loading ~11 million polygons with 1 million batch size
incremental-render-ca-1mi

Signed-off-by: Xun Li <lixun910@gmail.com>
Signed-off-by: Xun Li <lixun910@gmail.com>
Signed-off-by: Xun Li <lixun910@gmail.com>
Signed-off-by: Xun Li <lixun910@gmail.com>
Signed-off-by: Xun Li <lixun910@gmail.com>
@lixun910 lixun910 changed the base branch from upstream-xli-geoarrow to master November 15, 2023 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant