Scaling Leshan v2.x to 10,000 Clients, General Performance Query.

[HarrisonPace]

Question

Hello Leshan Developers,

I would like to use Leshan in a development environment to simulate around 10,000 LwM2M clients.
All of these clients will be connected to an external (load-balanced) LwM2M Server (on an IoT Platform).

These clients will need to update resources as though they are real devices in the field (i.e. produce simulated measurements).
The hardware simulating these clients is fairly powerful (think enterprise server: 10g NIC, lots of RAM: etc.)

I would like to create a minimal viable product. As this isn't to be used in production, I was hoping to use the included Test Server and integrated API. (I understand the pitfalls outlined here: #789 (comment) )

Before I embark on this project, I wanted understand if this was viable using Leshan and the best way to simulate a large number of LwM2M 1.0 Devices. I was worried scaling may be an issue. Ideally I want to minimize development of my own LwM2M server and get something operational fairly quickly:

Goals:

• Simulate a large number of LwM2M 1.0 Clients
• Update Resources of each client as though it was simulating measurements.
• Produce Observe Notifications as required by the Server.

I would appreciate any feedback based on this plan and any pitfalls, you would expect to see,

Regards

[sbernard31]

Hi,

Just to be sure I get you :

• your goal is to develop LWM2M client simulator based on Leshan ?
• using *leshan-server-demo is just a first step to test this simulator?

It's hard to me to know if you will be able to simulate 10.000 clients but I can list you known limitation :

1.) You are talking of LWM2M v1.0 clients, so you should read : #1244
2.) Number of thread used by Leshan clients : #491, eclipse-californium/californium#1203

You can also look at this POC : https://github.com/sbernard31/benchmark-clients

HTH

[HarrisonPace]

Hi,

Just to be sure I get you :

• your goal is to develop LWM2M client simulator based on Leshan ?
• using *leshan-server-demo is just a first step to test this simulator?

It's hard to me to know if you will be able to simulate 10.000 clients but I can list you known limitation :

1.) You are talking of LWM2M v1.0 clients, so you should read : #1244 2.) Number of thread used by Leshan clients : #491, eclipse-californium/californium#1203

You can also look at this tool : https://github.com/sbernard31/benchmark-clients (written very quickly and not so tested)

HTH

Thanks,

1). Both 1.0 and 1.1 are supported by the server so either version could be used in our context.

2). We don't plan to develop anything at the moment, we really just want to use the demo to simulate 10,000 devices and test the IoT platform we are using. Long Term we may develop our own frontend / API, if limitations require us to do so.

[sbernard31]

We don't plan to develop anything at the moment, we really just want to use the demo to simulate 10,000 devices

It's hard to me to imagine that just running 10.000 leshan-client-demo on same machine will fit your need because :

1. this will use lot of threads. (explained by issue above ☝️ ) (note there is some OS limitation about number of thread but you can generally tweak your OS to increase it)
2. you said "These clients will need to update resources as though they are real devices in the field (i.e. produce simulated measurements)" and it will be surprising that current leshan-client-demo did exactly what you expect.

But if with inputs I give you previously, you even think this could work. I think you should just test it. If you don't plan to develop anything this will be quickly tested.

Let us know about your result 🙏

[jvermillard]

To complement what Simon already said,
On a modern Linux machine, I ran the benchmark client up to 35k clients (35k thread). The limit was the number of local ports to bind more UDP sockets.

The overall Leshan server performance is excellent; the limitation often comes from the security-store backend you use. It needs to be fast to keep the DTLS handshake quick.

Scaling the Leshan server to 10k on a single machine with active/passive failover is usually something not complicated. For +100k, you need to start looking at clustering your server setup.

[allaismail]

Hello All,
I would like to implement a cluster of leshan servers in kubernetes and a set up a load-balancer to balance the lwm2m trafics to the cluster, but i dont know where i can start, And what will be the final solution.
I would appreciate any feedback based on this plan and any pitfalls, you would expect to see, 
Regards

[sbernard31]

@allaismail I'm not sure this is totally related to this issue.

Anyway, some inputs about leshan in a cluster :

• https://github.com/eclipse/leshan/wiki/Using-Leshan-server-in-a-cluster
• https://github.com/eclipse/leshan/wiki/Server-Failover

You could also have a look at this californium resources (but I didn't know so much about it) :

• https://github.com/eclipse-californium/californium/tree/main/demo-apps/cf-extplugtest-server#built-in-dtls-connection-id-load-balancer-cluster

[allaismail]

thank you very much; this will help me to start !!

[sbernard31]

Should we close this issue ?