Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to run supercluster as server side clustering #124

Closed
jpnathannew opened this issue May 23, 2019 · 13 comments
Closed

How to run supercluster as server side clustering #124

jpnathannew opened this issue May 23, 2019 · 13 comments

Comments

@jpnathannew
Copy link

jpnathannew commented May 23, 2019

I am creating server side clustering for my map. When I use PHP and Mysql for create the clustering it is taking more time. So I am planning to use this project
I have few doubts

How to run this project as server?
If I pass My lat and lang points into this project this will convert everything to cluster?
How to expose API for markers to cluster?

Please help me. I am waiting for your response

@mourner
Copy link
Member

mourner commented May 23, 2019

Duplicate of #79

@mourner mourner marked this as a duplicate of #79 May 23, 2019
@mourner mourner closed this as completed May 23, 2019
@jpnathannew
Copy link
Author

Duplicate of #79

It is not a duplicate of #79. Please guide me to use this supercluster

@xlcrr
Copy link

xlcrr commented Oct 15, 2020

hi @jpnathannew did you make any progress? I am working on the same problem. Maybe we can help each other?

@kityan
Copy link

kityan commented Oct 16, 2020

hi @xlcrr! What is the problem you trying to solve? You can follow these steps with nodejs app:

  • load points (lat, lng) from source (database, file, etc)
  • convert data to a suitable format (GeoJSON Point)
  • build index with new Supercluster(options).load(points)
    When I was developing such application, I've wanted to store clusters, but the best solution was to serve directly from nodejs app using (see getClusters method).

@xlcrr
Copy link

xlcrr commented Oct 16, 2020

Hi @kityan thanks for the message

I am using Laravel PHP framework on the backend.

Once the client has the data, processing it with supercluster is incredibly fast (100's ms). The slowest part seems to be getting the data from the table into the browser (9-10s for 100,000 points and growing on virtual machine) therefore I would like to save the clusters to a table and update this with a script on the backend.

I am still trying to figure out how to do this but I am thinking about passing the data into a javascript file and execute it on the backend with $response = exec('node my_script.js'), and store this data to a table to allow for faster lookup, or maybe using a microservice. (Both of which I am new to).

@kityan
Copy link

kityan commented Oct 16, 2020

@xlcrr the main idea is not to send 100k points to client for clusterization. For my project I have splitted front end code: for some zoom levels (e.g. 9-18) it requests backend for points (here you can use directly your php backend). For other zoom levels (e.g. 1-8) it requests backend for CLUSTERS not for POINTS. And here nodejs app with Supercluster comes in. You can proxy with web-server (nginx, etc) or via your php code.
Of course loading 100k points from db may be slow, but it will be done rarely. You nodejs app (with Supercluster) should do this operation (rebuild index) on some events in db: e.g. point add, point remove, point update. So, you need to make some kind of communication between your php code (here are methods for point add/remove/update) and nodejs app. Or if you use MongoDB or any other db with watch features, you can place such watchers directly in nodejs app (handler function will get all points again and rebuild index)

@kityan
Copy link

kityan commented Oct 16, 2020

@xlcrr My advice: do not store clusters index to db (I am not really sure that Supercluster's index is serializable).
Build a standalone microservice using nodejs+Supercluster and let it store and rebuild clusters index in memory.
You will be able easily request this microservice to give clusters for BBox (see getClusters method)
Yes, you should be familiar with long-live server side (nodejs vs php). But it's worth it.

In this case your $response = exec('node my_script.js') is not suitable. You should have standalone nodejs app and communicate with it (directly from web front, or via php code (here you can use sockets, websockets, etc))

P.S. Moreover, you should expect that number of clusters on higher zooms can be very close to number of points (depends on cluster radius). So you could have a lot of data. It is also a reason no to try storing clusters to database on each clusters index rebuild but to serve them from nodejs app.

@xlcrr
Copy link

xlcrr commented Oct 16, 2020

@kityan thank you very much for the reply. I will do some research and report back

@xlcrr
Copy link

xlcrr commented Oct 21, 2020

@kityan I decided to save zoom levels to DB

see https://openlittermap.com/global :-)

@kityan
Copy link

kityan commented Oct 22, 2020

@xlcrr, well done! I don't think that it's the best solution (to store clusters to DB and not to serve them from standalone app), especially if you have to reindex clusters frequently. But, anyway. My backend is always nodejs, so it's more convenient to me.
P.S. I've remembered your project from angel.co, we've chatted there. If you will need any advice on my stack (see the link in github profile) — let me know.

@kityan
Copy link

kityan commented Oct 22, 2020

@xlcrr P.S. A little more about pros and cons. When I use standalone app to handle frontend requests I can use getClusterExpansionZoom() to improve UX. I recieve bbox and zoom level from frontend. Then I get clusters with getClusters() and then patch data with info from getClusterExpansionZoom() for each cluster. Also you can decrease response size. GeoJSON is a bit verbose and if you communicate only with your own frontend, you can simplify json.
But I think you can patch data also before storing in database. Anyway, as a result you can make clusters clickable (jump to zoom level the cluster expands). It's much more user friendly.

@xlcrr
Copy link

xlcrr commented Oct 24, 2020

@kityan I have recently launched OpenLitterMap as open source. You can see my backend PHP implementation here. Can't believe we chatted before that's great! Sounds like an awesome implementation, well done! We will see what the emerging OLM community can recommend as refactoring this atm is currently outside my timeframe

@yudeqang
Copy link

yudeqang commented Aug 30, 2023

Have any postgis implementation?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants