Redis prefix in service-runner

[borpin]

emoncms/scripts/services/service-runner/service-runner.py

Line 45 in e6da270

if server.exists('service-runner'):

As discussed here, it was found that service-runner was not picking up the command from the backup module as service-runner was simply looking for service-runner and my settings had a redis prefix of emoncms.

When the python code was changed to look for emoncms:service-runner the backup module worked as expected.

[pb66]

Also see https://community.openenergymonitor.org/t/emonpi-service-runner-now-uses-redis-core-wifi-sync-postprocess-module-updates-required/7601.

There is a slight conflict in the fact that each emoncms instance uses a prefix to seperate them and only one service-runner can be running ordinarily, therefore just one queue gets checked.

Perhaps emoncms needs to make a exception for the service-runner entries and use a common queue where the instance specific path is passed in the command so that when a service-runner flag is issued from instanceA it is only instanceA that gets backed up or updated etc etc.

[edit - Or can the service-runner (redis?) handle wildcards? ie the service-runner watches the*:service-runner queue(s)]

[greeebs]

I've had a quick play around with redis and scan and I think I can easily enough look for *:service-runner. If someone wants to assign this issue to me, I'll put a PR together.

The main thing to consider is that scan always does a scan through all keys and the pattern matching is done at the client side (yes, even if you use the match parameter). On my emonPi the redis queue never has more than a hundred odd keys and it takes no time to retrieve them. I don't know what the performance impact will be of continually scanning through all keys...

Can others run redis-cli scan 0 count 1000 and advise how many items were returned on their multi-instance installs?

Note: If the first item (labelled "1)") returned by scan is "0", there are no more items to retrieve and the second item (labelled "2)") contains the numbered list of keys. If the first item isn't "0", run the command again replacing "0" with the number returned in the first item.

[borpin]

Just ran it, 6109! Lots of..

   6088) "emoncms:input:5847"
   6089) "emoncms:input:lastvalue:603"
   6090) "emoncms:input:lastvalue:5208"
   6091) "emoncms:input:lastvalue:5218"
   6092) "emoncms:input:lastvalue:4874"
   6093) "emoncms:input:lastvalue:2602"
   6094) "emoncms:input:lastvalue:3647"
   6095) "emoncms:input:lastvalue:3396"
   6096) "emoncms:input:lastvalue:4586"
   6097) "emoncms:input:lastvalue:5706"
   6098) "emoncms:input:lastvalue:3133"
   6099) "emoncms:input:lastvalue:5687"
   6100) "emoncms:input:lastvalue:3473"
   6101) "emoncms:input:lastvalue:3862"
   6102) "emoncms:input:lastvalue:2005"
   6103) "emoncms:input:lastvalue:4949"
   6104) "emoncms:input:lastvalue:1432"
   6105) "emoncms:input:lastvalue:2230"
   6106) "emoncms:input:lastvalue:2540"
   6107) "emoncms:input:lastvalue:1847"
   6108) "emoncms:input:lastvalue:3713"
   6109) "emoncms:input:lastvalue:148"

so I wonder if something is not quite right.

Does match the admin screen info (which I tried to paste here but it didn't like the formatting).

I flushed the buffer and it seems to have settled at 235 keys. I'll keep an eye on it.

1) "0"
2)   1) "emoncms:input:5862"
     2) "emoncms:input:5866"
     3) "emoncms:feed:14"
     4) "emoncms:input:lastvalue:5865"
     5) "emoncms:feed:32"
     6) "emoncms:input:5897"
     7) "emoncms:input:5879"
     8) "emoncms:feed:36"
     9) "emoncms:input:lastvalue:8"
    10) "emoncms:input:30"
    11) "emoncms:input:lastvalue:10"
    12) "emoncms:feed:10:bufferstatus"
    13) "emoncms:input:5883"
    14) "emoncms:feed:20:buffer"
    15) "emoncms:input:5856"
    16) "emoncms:feed:37:bufferstatus"
    17) "emoncms:feed:15:bufferstatus"
    18) "emoncms:input:lastvalue:30"
    19) "emoncms:input:lastvalue:5879"
    20) "emoncms:input:5884"
    21) "emoncms:input:lastvalue:5909"
    22) "emoncms:feed:28"
    23) "emoncms:input:10"
    24) "emoncms:input:lastvalue:5900"
    25) "emoncms:user:inputs:1"
    26) "emoncms:feed:10:buffer"
    27) "emoncms:input:lastvalue:5871"

[borpin]

Could emoncms fire in a setup key that service-runner could scan for once (perhaps) and then service-runner will know what the prefix to use is?

[greeebs]

Could emoncms fire in a setup key that service-runner could scan for once (perhaps) and then service-runner will know what the prefix to use is

I think @pb66 had a better idea - to not use the prefix when sending stuff to service-runner.
@TrystanLea, @glynhudson what are your thoughts on this? Is there much work to remove the instance prefix when pushing values into redis with the service-runner name?

[borpin]

I think @pb66 had a better idea - to not use the prefix when sending stuff to service-runner.

I suspect that isn't easy. Look at the code in the backup controller, it just pulls in $redis as a global object and uses it without specifying the prefix. The prefix must be done when it is instantiated.

Out of interest I did a search on "new Redis" in GitHub. Clearly some modules create new connections rather than using the global one (not that I understand why). I wonder then, if it is possible for any module etc that want to use service-runner, to create its own connection to redis and use that? Could service-runner have its own prefix?

https://github.com/emoncms/backup/blob/e8110759c46a62aaa86f70756c169465ac7acf16/backup/backup_controller.php#L38

[greeebs]

Could service-runner have its own prefix?

Or more ideally - not have any prefix. Then service-runner won't need changing ;)

[borpin]

Or more ideally - not have any prefix. Then service-runner won't need changing ;)

I think it was originally added as someone had an instance where they ran redis for a different purpose as well.

I also think that getting those Modules that need service-runner to create their own connection and thus use a predefined prefix might be the best solution.

In the short term, could an 'or' be added to the python code so it looks for both a key with and without the emoncms prefix as that is the default for non-pi installations?

[greeebs]

In the short term, could an 'or' be added to the python code so it looks for both a key with and without the emoncms prefix as that is the default for non-pi installations?

That's certainly a pragmatic interim solution. I'll put something together in the next few days and submit a PR.

[TrystanLea]

That sounds like a good interim solution to me, happy to accept a pull request for that.

[greeebs]

Apologies for not actually getting to that PR (and many thanks to @borpin for his!), too much paying work on just at the moment.

I did have another thought today about how we could make this more generic without bogging it down too much, but it does complicate the service a fair bit.
Given Brian's PR solves the immediate problem, but not the generic case, I think we leave this issue open and when I do get some time in the next few weeks, I'll submit a PR with more significant changes to get this working better in the "generic prefix", or multiple instance cases.