use single curl session for Volkszaehler

[mbehr1]

To reduce load if multiple channels towards the same middleware are in use I added some handling to use a synchronized curl session between the logging threads. In other words: the logging threads share a single curl session if they are communicating with the same middleware. Thus the http requests gets synchronized instead of being started in parallel. Currently only activated for Volkszaehler api but could easily be added for MySmartGrid as well.
E.g. on my system I have 4 channels from one meter towards the same middleware. I constantly have a load of >4 as there are 4 apache2 services active. Using this change the load gets <2.

[andig]

Not entirely sure what this does: execute requests sequentially? What happens if things start queuing up? How much latency does it add (thinking about realtime logging here)?

The real load of middleware.php comes from the fact that ORM, DB connection and Entities are initialized with every single request. If load is a real concern then it would make more sense- if we have requests that need queuing- to bunch them into a single request instead of queuing them as two requests. middleware.php would need a fix for that but that could easily be done.

[mbehr1]

The problem I tried to fix (and I think did fix ;-) with this PR is the following:

in my setup I have two meter (d0 and ocr) and 6 channels (4 from d0 each sec., 2 from ocr each minute) pointing to the middleware running on the same raspberry pi.

The current threading model does the following:

• generate one "reading thread" for d0 (ok)
• generate one "reading thread" for ocr (ok)
• generate 4 logging threads for each d0 channel (~)
• generate 2 logging threads for each ocr channel (~)

The 6 logging threads each use a separate curl connection to connect to the middleware. This causes 6 parallel connections for the local apache2 server performing 6 parallel connections to the mysql server,… with a good impact on cpu load and especially RAM usage (on a 256MB raspi quite important).

My PR changes this in the following way:

• no change for defined/started threads
• each logging thread now shares one curl session for each middleware. In my case: middleware is always the same -> one session.

So instead of each logging thread having it’s own curl session they share one (as they all want to talk to the same middleware instance). So instead of firing 6 parallel requests to the apache/middleware/mysql they basically get send sequentially, using the same curl session so talking to the same apache instance (as curl seems to keep the session open for some time and reuses it).
If one connection get’s stuck the other would wait for it. But I think this shouldn’t be a problem as they are all talking to the same middleware (which most likely then has a general problem anyhow). Channel to different middleware instances still use parallel/independent sessions and don’t get blocked/serialized.
The logging thread code handles delays pretty well already. If more data from the reading thread is available it simply get’s bundled into one request/data packet to the middleware. So this serializing should not be any problem and just provide cpu and mainly ram benefits.

Am 03.02.2015 um 09:31 schrieb andig notifications@github.com:

Not entirely sure what this does: execute requests sequentially? What happens if things start queuing up? How much latency does it add (thinking about realtime logging here)?

The real load of middleware.php comes from the fact that ORM, DB connection and Entities are initialized with every single request. If load is a real concern then it would make more sense- if we have requests that need queuing- to bunch them into a single request instead of queuing them as two requests. middleware.php would need a fix for that but that could easily be done.

This could be done anyhow and would be quite nice. I thought about this but would be major change in the channel/logging-thread logic and the middleware.But I still think that having a lot of parallel apache2/mysql connections on small embedded systems should be avoided.

Gruß
Matthias=

[andig]

So this serializing should not be any problem and just provide cpu and mainly ram benefits.

Agreed. It's the same I have done in the frontend by batching the requests into a single one that is sequentially evaluated.

This could be done anyhow and would be quite nice. I thought about this but would be major change in the channel/logging-thread logic and the middleware

Let me know when and if so I can prepare the data context for what you need.

[mbehr1]

Thanks! Will do.

Am 10.02.2015 um 20:45 schrieb andig notifications@github.com:

So this serializing should not be any problem and just provide cpu and mainly ram benefits.

Agreed. It's the same I have done in the frontend by batching the requests into a single one that is sequentially evaluated.

This could be done anyhow and would be quite nice. I thought about this but would be major change in the channel/logging-thread logic and the middleware

Let me know when and if so I can prepare the data context for what you need.

—
Reply to this email directly or view it on GitHub #117 (comment).

Gruß

Matthias