Script-Engine: jsr223 script core to allow different scripting engines

[smerschjohann]

Hi,

I have implemented another script/rule engine to support different script languages. Those script-engines have to support jsr223. There a quiet a lot of choices: Jython, JRuby, Nashorn, etc.) (https://java.net/projects/scripting/sources/svn/show/trunk/engines)

The Scripts need to be located in "configurations/jsr_scripts", the path "jsr_scripts" is currently hardcoded. I had to implement my own FileWatcher as the FileObserver was "integrated" with the models. My FileWatcher is based on the FileWatcher-API provided by Java7 and therefore supports the underlying OS functionality.

I've added jython.jar to the lib-folder of this plugin. There might be a better way. I'm open for any suggestions (I'm not that familiar with the OSGI-architecture)

Script Implementation

Currently I've only checked python support as it is my main focus. I have added a test script as well, which shows how the scripts have to be written.

Rule-Class

Each rule is basically a class in the given scripting language. It needs to implement three functions:

• getName:
  • for internal reference (timer trigger)
• getEventTrigger
  • returns an array of EventTrigger
• execute
  • gets as first argument the reason why it was called

Supported triggers:

• ChangedEventTrigger (for updates and changed states)
• CommandEventTrigger
• ShutdownTrigger
• StartUpTrigger
• TimerTrigger

class TestRule(Rule):
    def getName(self):
        return "TestRule"

    def getEventTrigger(self):
        return [
            StartupTrigger(),
            ChangedEventTrigger("Heating_FF_Child", None, None),
            TimerTrigger("0/50 * * * * ?")
        ] 

    def execute(self, event):
        oh.logDebug(self.getName(), "event received " + str(event))
        oh.logInfo(self.getName(), str(ItemRegistry.getItem("Heating_GF_Corridor")))
        action = oh.getActions() 
        oh.logInfo(self.getName(), "available actions: " + str(action.keySet()))
        ping = oh.getAction("Ping")
        oh.logInfo(self.getName(), "internet reachable: " + ("yes" if ping.checkVitality("google.com", 80, 100) else "no"))

Output

21:47:06.975 [DEBUG] [.openhab.model.jsr232.TestRule:60   ] - event received Event [triggerType=STARTUP, item=null, oldState=null, newState=null, command=null]
21:47:06.977 [INFO ] [.openhab.model.jsr232.TestRule:75   ] - Heating_GF_Corridor (Type=SwitchItem, State=OFF)
21:47:06.979 [INFO ] [.openhab.model.jsr232.TestRule:75   ] - available actions: [Exec, Transformation, Ping, HTTP, Audio]
21:47:07.003 [INFO ] [.openhab.model.jsr232.TestRule:75   ] - internet reachable: yes

Interaction with openhab

In general all interaction is done throw the oh object. It has support for the following

logging

• oh.logDebug(logger_name, format, arg0,....)
• oh.logInfo(logger_name, format, arg0,....)
• oh.logWarn(logger_name, format, arg0,....)
• oh.logError(logger_name, format, arg0,....)

BusEvent

• oh.sendCommand([Item] item, [String] commandString)
• oh.sendCommand([Item] item, [Numer] number)
• oh.sendCommand([String] itemName, [String] commandString)
• oh.sendCommand([Item] item, [Command] command)
• oh.postUpdate([Item] item, [String] stateAsString)
• oh.postUpdate([String] itemName, [String] stateAsString)
• oh.postUpdate([String] itemName, [State] state)
• oh.storeStates([Item[]] items)
• oh.restoreStates([Map<Item, State>] statesMap)

ItemRegistry

• ItemRegistry.getItem(itemName)
• ItemRegistry.getItemByPattern(String name)
• ItemRegistry.getItems()
• ItemRegistry.getItems(String pattern)
• ItemRegistry.isValidItemName(String itemName)

Accessing actions:

To access the old known methods (all actions) the openhab interface supports getActions() and getAction(name_of_action_provider)

oh.getActions()
ping = oh.getAction("Ping")
ping.checkVitality("google.com", 80, 100)

Global variables/classes

• RuleSet
• Rule
• State
• Command
• ChangedEventTrigger
• CommandEventTrigger
• Event
• EventTrigger
• ShutdownTrigger
• StartupTrigger
• TimerTrigger
• TriggerType
• ItemRegistry <-- this allows direct access to items ( ItemRegistry.getItem(itemName) )
• oh

[buildhive]

openhab » openhab #2637 SUCCESS
This pull request looks good
(what's this?)

[buildhive]

openhab » openhab #2650 SUCCESS
This pull request looks good
(what's this?)

[kaikreuzer]

Hi,

This is really cool stuff, thanks for it! I was also considering jsr223 support, but couldn't really see how it could fit in best. Your way to deal with actions etc. looks pretty elegant.

Your PR brings me in some trouble though: As you are hopefully aware, the evolution of the core takes place at Eclipse SmartHome now, which builds the base for openHAB 2. At Eclipse SmartHome, there is work for a new rule engine framework going on, which is also meant to allow other scripting languages - so your work fits in here nicely.

I planned to make the current xtext-based rule syntax compatible with this new rule framework, so that there is no real need for migration for users - their rules could be kept (with maybe only slight modifications). If we now all of a sudden have not only the xtext-based rules, but many other possible syntaxes, I am out of the game with this approach though...

That's why I would like to check with you, what you think is the best approach to handle this situation. I mainly see two options:

• Merge this PR, but make the bundle a part of the addons and not of the core runtime. On the wiki for this addon, we should clearly mention the risk that scripts created for this might not work on openHAB 2 anymore. To reduce the risk, we could actually check, if your bundle works with the openHAB 2 compatibility layer, which mimics the openHAB 1 APIs - we could actually get it working there.
• See how jsr223 can be fitted into the new rule framework at Eclipse SmartHome and directly adapt the rule syntax to allow alternative scripting languages (it would imho suffice to allow other languages between the "then" and the "end" keywords of the rule files - this would remove the need to deal with the trigger definition there as well).
  Writing this, I think I personally would even opt to follow up on both options. What is your opinion?

Regards,
Kai

[smerschjohann]

To be honest, I've not looked at smarthome / openhab2 so far, but it might the right time to do so now.

I've read the thread on the eclipse forum and as I see it, it does not differ that much from the current possibilities, only the way of describing them is changed. The same can also be said about the jsr223 implementation.

So a "rule" or "automation" can be seen as a sum of triggers (on + if combination) and the execution part, this is nearly the same as I'm currently doing it with jsr223. I already thought about "custom" triggers in this code, but for fast development purposes kept it simple and only used the triggers already available by the Rule system and do not allow additional conditions in them right now. But it would be possible to do this:

A trigger consists of:

1. a type (UDATE, CHANGE, STARTUP, ....) (in the json format the "on"-block) with basic evaluation logic already (which item, which previous state, which new state) (the "old" trigger mechanic)
2. a set of conditions in the form of a evaluation function. The triggers can then be generated in such a way that they combine different Condition classes for evaluation.

So basically what I want to tell here is the following: If we see the smarthome json-defintions as "construction"-code for underlying classes, we could do the following:

Rule-"Ingredients"

1. Establish a Trigger-Factory/Builder
   1. Define easy predefined condition-classes (like mentioned in the json examples) e.g. Compare
   2. Allow adding more condition-evaluation-classes to this factory (this would allow custom java code or even scripting languages to make their own Trigger-Conditions available

Usage example

trigger = TriggerBuilder.createTrigger("triggerName", TriggerType.CHANGE, "itemName", null, null)
                                                      .withCondition("Compare", config)
                                                      .withCondition("SuperSpecialCondition", config2)
                                                      .build();

1. Establish a Execution-Factory

execution = ExecutionFactory.createExecution("exec1", config);

Rule generation "brewing"

The Rule generation can be done by different code-basis:

1. json-description-parser
2. direct Java code
3. scripting languages

These Rules will then be added to the Rule-Engine.

The JSON-Description-Parser would do the following:

1. use the Trigger-Builder to generate needed conditions for the rule which needs to be met for the execution part
2. use the Execution-Factory to generate the execution part
3. combine them into a Rule object and add them to the Rule Engine, to summerize a rule will consist of

• a set of trigger objects
• one execution object

finally:
let the Rule-Engine evaluate the the triggers and call the Rule.execute(event, trigger) method. The Rule class calls the underlying execution object in case it is a json generated rule.

To illustrate the usage I quickly prototype it. It is not that different from the way I did the script engine right now):

interface Condition {
     boolean evaluate(Event event);
     void setConfiguration(HashMap<String, Object> config);
}

interface Trigger {
    String getName();
     TriggerType getTriggerType();
     boolean evaluate(Event event);
}

interface Execution {
   String getName();
   void setConfiguration(HashMap<String, Object> config);
   void execute(Event event, Trigger trigger);
}

interface Rule {
    List<Trigger> getEventTriggers();
    void execute(Event event, Trigger trigger);
}

class JsonGeneratedRule implements Rule {
    JsonGeneratedRule(List<Trigger> triggers, Execution execution) {
       this........
    }

    List<Trigger> getEventTriggers() { return this.triggers; }
    void execute(...) { execution.execute(event, trigger);
}

This would allow to make the execution part interchangeable, but does not require to do so.

This would therefore allow defining the rules, triggers and conditions in Java, Scripting languages and with the json description "language".
It would even not be too hard to transform the already existing xtend scripts into these class structure. I think this would be a good solution for smarthome.

In openhab the current definition of the interface could nearly be the same as it is right now. I can already define the interface like that and extend the rule engine a little bit. This engine could than be used in openhab 1 and 2.

And last but not least: The scripting engine(s) need the ability to add classes to those factories (triggers, conditions, execution-parts), to be usable in the json-descriptions

I would also change it in such a way that not a method "getRules" is called, but rather let the scripts call something like "oh.addRule", "oh.addCondition", "oh.addTrigger", ("oh.addBinding"?, "oh.addAction"?)

---

In my opinion it would be no good idea, to reduce the functionality of how a rule can be defined as they are really powerful in this way and for example one can even generate different triggers based on a supplied configuration if wanted. One of my rules can be seen here

[smerschjohann]

Any thoughts on my suggestion? Otherwise I would start with it the next days.

[buildhive]

openhab » openhab #2729 SUCCESS
This pull request looks good
(what's this?)

[kaikreuzer]

Any thoughts on my suggestion? Otherwise I would start with it the next days.

Sorry, I didn't find the time to read through your proposal yet, but will hopefully provide you feedback by the end of this weekend - thanks for your patience!

[kaikreuzer]

Ok, I read through your ideas, but I think I did not fully get your suggestion yet - what you describe, is that meant to be your suggested solution for openHAB 1 in order to be as compatible as possible with the new Eclipse SmartHome concept? If this is the case, I somehow feel that it would mean quite some duplicate implementation efforts wrt JSON parsing, trigger and engine instantiations etc. - this is all work that will be done for Eclipse SmartHome as well.

One of my other concerns is that the "script-based rule brewing" will be hard to get compatible - you will need all those interfaces and factories to be available within the scripting languages. Is that a feature of jsr223 that makes this always possible easily? Aren't there any conceptional mismatches for some scripting languages?
Why do you think it is important to have the "brewing" completely available within scripting? Wouldn't it be good enough to have the "frame" given through JSON (or a DSL) and only the execution logic as a script?

[smerschjohann]

At the time of writing I did not know, that a first draft of the classes in smarthome were already done, so therefore I did not use them.
Nevertheless, what I meant here was more over that even smarthome does not need to rely on the json descriptive language. It could more likely be designed in such a way, that the classes/objects are generated by the json parser in the same way as any other "implementation technique" can generate them. So the json description is just one way of using and defining rules in smarthome - but each possible implementation would be compatible to each other as they use the same interface definitions.

I looked at the smarthome classes and think that they are too complicated for a scripting language (and even for the java implementation) if all the boilerplate code has to be defined by hand. It would be advisable to create helper classes which help generating them (Builder classes).

Furthermore there could be "comfort" methods which take as argument the "basic" rule engine interface (e.g just isSatisfied in case of ConditionHandler) and in addition names/values/configs which are really mandatory to make them available in a modular fashion. The rest is handled by a default implementation. Or maybe use the new "default implementation" feature of interfaces in java?

There are two possibilities to proceed here:

1. create a compatiblity layer in smarthome which generates "new" style(/smarthome) objects from the currently available classes in jsr223 implementation
2. (which I prefered and meant by my writing above) start implementing the new rule engine and directly integrate it with jsr223 in openhab 1. This openhab1 rule engine would use the interface without the focus on modularisation. There could just be a fixed range of preexisting modules (ConditionHandler etc.).

Question: [..] you will need all those interfaces and factories to be available within the scripting languages. Is that a feature of jsr223 that makes this always possible easily?
Answer: Yes, jsr223 makes this extremly easy. But as described above, it would be nice if there exist comfort methods/builder which allow to write less boilerplate code.

Question: Aren't there any conceptional mismatches for some scripting languages?
Answer: probably. Nashorn is not anything near as comfortable as Jython when it comes to jsr223. But I guess it would even be possible to use the whole set of interfaces in Nashorn, but with a lot of inconvenience - true.
Solution: Again: comfort methods/classes: Those could be written preferably on the java side, but if a particular script language needs something special: Write wrappers in this language which abstract those details away.

Question: Why do you think it is important to have the "brewing" completely available within scripting?
Answer: Even in the smarthome approach, the final product which would get parsed is very static. If you look at my example you can see that I generate the rule trigger part by the available items that are configured at that point of time. In this particular case it might not be needed as the triggers could be changed in such a way that one does not need to generate them dynamically.
Nevertheless there might be cases where this becomes very handy and if its not possible make things extremely inconvenient. Moreover I don't see any reason to not support the rule generation in scripting languages (and in java).
This does not mean that it has to be used - it should just be possible. Nothing stops one from allowing scriptings in the json description. Allowing that would not be too hard.

[kaikreuzer]

Hi @smerschjohann,

You are right, besides a JSON reader (be it to read it from within OSGi bundles, from the file system or through the REST API), there can be potentially other sources for rules.

But please note that with the new modular concept, the focus is on moving from rules to rule templates. So a rule should actually look like "rule2" in https://github.com/eclipse/smarthome/blob/master/bundles/automation/org.eclipse.smarthome.automation.json/src/main/resources/RuleDefinitions.json#L57. This means it is merely a configuration to instantiate a rule from a given template. The idea of this is to make it much easier to share and exchange rules (that is: rule templates) between users. And then again, these templates are simply a set of modules and the idea is actually to have different modules available for different scripting languages.

(which I prefered and meant by my writing above) start implementing the new rule engine and directly integrate it with jsr223 in openhab 1. This openhab1 rule engine would use the interface without the focus on modularisation. There could just be a fixed range of preexisting modules (ConditionHandler etc.).

I doubt that this will lead to something that is compatible with openHAB 2 in the end - and this is my most important goal. Any new kind of concepts in openHAB 1 will make it harder to provide a migration path to openHAB 2 for users. This is why we actually declared the openHAB 1 runtime to be feature frozen a while ago and only want to accept bug fixes for it. New concepts should be implemented in Eclipse SmartHome (and therefore for openHAB 2).
My suggestion would therefore to continue the discussion at https://www.eclipse.org/forums/index.php/t/860533/ - am am absolutely open to all your suggestions and use cases and hence we should work on making them available on the long run. We could even start with a small implementation of these APIs based on your jsr223 work and see how the concepts match and what might be missing and needs to be adjusted on the APIs. What do you think?

[smerschjohann]

What you write would also mean that I stop providing jsr223 for openhab 1.
But I really think most users would not really care if they have to change their rules (a little bit) when migrating to openhab 2. I would think some would rather prefer a working python/ruby rule engine than using the existing Xtend engine. At least that was the main reason for me developing this "replacement".

Maybe the best is to stop working on jsr223 for openhab 1. It just keeps the way I have it right now. I could move the package to another namespace and integrate it as an addon. If someone wants to use it, they can - as an addon. No direct migration path will be given for this addon and if something breaks (which will probably happen), they were warned. I guess someone using this addon at this stage would have the competence to migrate the rules later on.

And yes we can discuss about the rule engine for smarthome and possible integration points for script engines on the forum.

[kaikreuzer]

Yes, this was exactly my suggestion from above! (See the two bullet points on #2378 (comment)) :-)

[smerschjohann]

Hm, I am currently looking for a package name but do not really know which would fit here.
I think core is the right namespace. Maybe I should just remove the autostart parameters?

[kaikreuzer]

Yes, agreed, the package can stay. So my original suggestion (#2378 (comment)) would still apply, if you agree:

Merge this PR, but make the bundle a part of the addons and not of the core runtime. On the wiki for this addon, we should clearly mention the risk that scripts created for this might not work on openHAB 2 anymore. To reduce the risk, we could actually check, if your bundle works with the openHAB 2 compatibility layer, which mimics the openHAB 1 APIs - we could actually get it working there.

Would you have a chance to check whether it's working with openHAB 2 and what we might have to do on the compat layer?

[smerschjohann]

I try to check OpenHAB 2 compatiblity when I have a bit spare time. As I do not have a running OpenHAB2 configuration right now, this could take some time (as I don't have much time currently).

I just moved the js223 core to the addon package. Hope I didn't miss a spot. I will add a wiki page entry soon.

[kaikreuzer]

Ok, thanks. @teichsta, would you schedule this for the 1.7 release?

[buildhive]

openhab » openhab #2932 FAILURE
Looks like there's a problem with this pull request
(what's this?)

[buildhive]

openhab » openhab #2943 FAILURE
Looks like there's a problem with this pull request
(what's this?)

[teichsta]

Ok, thanks. @teichsta, would you schedule this for the 1.7 release?

yes, i will! Thanks @smerschjohann!

[teichsta]

this entry can be removed

[teichsta]

remove version (unless you really need this specific version)

[teichsta]

Hi @smerschjohann, thanks again for this great contribution which looks quite good already. I've merely had a look at the ScriptXXX classes since many other classes seem to be copied from the existing ScriptEngine. Please find my (very few) review comments inline. Please also squash your commits into one once you've incorporated the feedback. Best, Thomas E.-E.

[smerschjohann]

yes they were merely copied and modified for my needs. I would even have used them when they were not referencing directly to XText etc. As I did not want to change the current Script-Engine this seemed to be the only choice ;)

I have integrated your remarks. Also at https://github.com/openhab/openhab/wiki/Addon%3A-Jsr223-Script-Engine the Script-Engine is documented. If you look at Jython-Installation you see that I modified my local start-script of openhab.

Do you have a better idea how to directly integrate it? - And for that matter, make it easier for a new user to use jython with this addon?

I will test my modifications during the next days and squash the commits afterwards.

[buildhive]

openhab » openhab #3010 SUCCESS
This pull request looks good
(what's this?)

[teichsta]

Do you have a better idea how to directly integrate it? - And for that matter, make it easier for a new user to use jython with this addon?

hmm … in order to keep the general configuration clean i don't see any other solution than documenting the change to the start.xxx files like you did already.

I will test my modifications during the next days and squash the commits afterwards.

great, thanks!

[smerschjohann]

looks good, still working as expected. Can be merged now.

[buildhive]

openhab » openhab #3029 FAILURE
Looks like there's a problem with this pull request
(what's this?)