Reloading or swapping rulesets at runtime: is it possible?

[flan]

Note: my preferred working environment is Python, but it's still early enough in my project's development cycle to change to something else.

Reading through other issues, I've seen some attempts at removing or installing additional rules by interacting directly with the rules module, as in #96 and #136, but I can't seem to find documentation for this, leading me to think it's unsupported. (The C code is easy to follow as a form of living documentation if I'm wrong, though)

What I'm hoping to do is slightly different from what I've seen before, though: create a full ruleset at runtime and replace the running ones, as a way of supporting operational exceptions without encoding all of the possible conditions into antecedents ahead of time. But it's not clear how to go about doing this: there doesn't seem to be a way to stop or pause rules execution after run_all() is called, and there are no locks around the rulesets list in the Python bindings, making me think that attempting to replace elements is likely not threadsafe.

Is replacing rulesets at runtime supported? If not, is there any simple way to transfer in-Redis context to a new client process so the system can restart with minimal downtime and state-loss?

[flan]

After some more exploration, it looks like #2 and testdynamic.py demonstrate how to manipulate rulesets at runtime, but I'm not clear on how to generate a JSON-formatted ruleset.

It looks like creating a new ruleset object always adds it to the global list and, while it would be easy to ensure external accesses to this space don't overlap, it's not clear to me whether this is the right way to proceed. Assuming memory is effectively a non-issue and the total number of newly added rulesets wouldn't be in the hundreds before a restart, is it okay to keep adding to this list indefinitely?

[flan]

It looks like what I'm trying to do is very close to #121, which has a suggestion for half of the process, but not how to create the new ruleset properly.

From the syntax examples I've found, I'm not sure how the ruleset identifies the script-side function to be called after matching.

[aravindnc-olt]

@flan As per my understanding if you are using JSON ruleset then you cannot invoke a server side function dynamically. You have to call another api to check the event state and invoke the function based on its output/state. If you need parameters, you can pass it along with the first API call itself.

[flan]

@aravindnc-olt That could be my problem, then: I wasn't aware that JSON rulesets were different from in-script-defined rulesets and have been trying to figure out how execution happens in script-side logic after beta nodes are evaluated.

Might I ask how you went about learning how the pieces of durable_rules fit together beyond what's in the (very good) script-side interface documentation? It might be immediately obvious to me how (if) what I want to do can be done, given a suitable starting point.

[aravindnc]

@flan This is how I had done it,

1. Create a JSON ruleset something like below, API (/rulename/definition)
   rulename$state: { cond1: { all: { $m: { $gt: { amount: 0 } } }, run: "valid", to: "valid" } }, valid: {}
2. Then if you pass {amount:100,sId:1} to this, it will change its state to valid. (API: /rulename/event)
3. To get the current state you have call another api (/rulename/events/1). This will return a JSON which you can evaluate current state as valid. So based on this state you can call you server side function. So if you want additional data then you can pass it along with the request in step 2.

This is based on my understanding. @jruizgit might have a better understanding.

[jruizgit]

Hi, thanks for posting the question. @aravindnc, thanks for your reply. As @aravindnc said, when defining a ruleset using JSON you cannot inline a function. The JSON definition is purely declarative. In the testdynamic.py example you can see how the Host::get_action function is overriden. durable_rules gives you the opportunity to resolve a rule function based on the action_name defined in the rule definition run: attribute.

I haven't documented the JSON format nor the host extensibility points because I didn't consider them as first class. Turns out the JSON dynamic rule scenario is much more useful than what I originally thought. Let me put some thought on how to make it first class.

class Host(engine.Host):

    def __init__(self):
        super(Host, self).__init__()
        self._redis_server = redis.Redis()

    def get_action(self, action_name):
        def print_lamba(c):
            c.s.label = action_name
            print('Executing {0}'.format(action_name))

        return print_lamba;

    def load_ruleset(self, ruleset_name):
        ruleset_definition = self._redis_server.hget('rulesets', ruleset_name)
        if ruleset_definition:
            return json.loads(ruleset_definition)
        else:
            raise Exception('Ruleset with name {0} not found'.format(ruleset_name))

    def save_ruleset(self, ruleset_name, ruleset_definition):
        print('save ' + json.dumps(ruleset_definition))
        self._redis_server.hset('rulesets', ruleset_name, json.dumps(ruleset_definition))

[flan]

Thanks, @jruizgit, @aravindnc

So, in this case, does get_action() get invoked as a callback whenever a terminal node is reached, or would it still be necessary to poll the API endpoint mentioned before, /<rulename>/events/1?

[aravindnc]

@flan For me, get_action is not get fired every time (not sure what is the issue), so I'm doing it via API and it is working fine.

[alexhung]

@jruizgit I've just came across this repo today, in the course of researching rule-based engine for my upcoming project. The ability to load the rules from either JSON file or a database will be of importance to the project. This is so the rules can be maintained externally by either dev or users. I'm looking forward to documentation and examples on the best way to implement.

[jruizgit]

@alexhung, yes I will work on getting the format and extensibility points documented.

One clarifying note: get_action is called once for every distinct "run" name in the ruleset definition. get_action should return a function of the form fn(c) the function returned will be called every time a rule leading to the "run" name is resolved. In the example below, get_action will be called two times, one for "suspect" and one for "fraud". The function returned for "suspect" is going to be called every time the state machine moves to the "suspect" state. The function returned for "fraud" will be called every time the state machine moves to the "fraud" state.

{
    "start": {
        "trigger": {
            "all":[{
                "m": {"$gt":{"amount":100}}
            }],
            "to": "suspect",
            "run": "suspect"
         }
    },
    "suspect": {
        "trigger": {
            "all":[{
                "m": {"$gt":{"amount":1000}}
            }],
            "to": "fraud",
            "run": "fraud"
        }
    },
    "fraud":{}
}

[alexhung]

@jruizgit Since you will/are working on the documentation, another thing I would love to learn how to do is to define the consequent. In my use case, I'd like to assert a fact for that rule so that it can be retracted. I haven't dive into the code to figure that out yet. I suspect I will be able to figure it out but documentation will help a lot.

[alexhung]

@jruizgit I noticed your commits for the JSON documentation! Thanks!

I'm running into error when I try it out, however. Here is the snippet of my node.js code.

const express = require('express');
const bodyParser = require('body-parser');
const app = express();

app.use(bodyParser.json());

const redis = require('redis');
const database = redis.createClient({
 ...
});
const rules = require('./rules');

const durableEngine = require('./node_modules/durable/libjs/durableEngine');
const host = durableEngine.host();

host.getAction = function(actionName) {
  return function(c) {
    c.s.label = actionName;
    console.log('Executing ' + actionName);
  };
};

host.loadRuleset = function(rulesetName, complete) {
  database.hget('rulesets', rulesetName, function(err, result) {
    if (err) {
      complete(err);
    } else {
      complete(null, JSON.parse(result));
    }
  });
};

host.saveRuleset = function(rulesetName, rulesetDefinition, complete) {
  database.hset('rulesets', rulesetName, JSON.stringify(rulesetDefinition), complete);
};

host.setRuleset('match_1', rules, (err, result) => {
  console.log(err);
  console.log(result);
});

Whereas rules.json contains:

{
  "rule_0": {
    "all": [
      {
        "first": {
          "$and": [
            {
              "customerType": "institution"
            },
            {
              "location": "AM"
            },
            {
              "groupType": "bar"
            }
          ]
        }
      }
    ],
    "run": "foo"
  },
}

When the node server is reloaded, host.setRuleset('match_1', ...) is called and this error is returned:

TypeError: Cannot read property 'continueWith' of undefined
    at ruleset (/home/app/rules/node_modules/durable/libjs/durableEngine.js:768:33)
    at createRulesets (/home/app/rules/node_modules/durable/libjs/durableEngine.js:1105:38)
    at Object.that.registerRulesets (/home/app/rules/node_modules/durable/libjs/durableEngine.js:1171:28)
    at Object.that.setRuleset (/home/app/rules/node_modules/durable/libjs/durableEngine.js:1163:22)

The run value in the ruleset is a string ("foo") so I am unsure why the error is occurring.

Any help will be highly appreciated!

[jruizgit]

Hi, thanks for trying the documentation, actually the rules need to be wrapped in a ruleset object. Also you need to use the setRuleset method. I will fix the documentation, in the meantime, would you mind trying the example below?

var r = { 
  "ruleset": {
    "rule_0": {
      "all": [
        {
          "first": {
            "$and": [
              {
                "customerType": "institution"
              },
              {
                "location": "AM"
              },
              {
                "groupType": "bar"
              }
            ]
          }
        }
      ],
      "run": "foo"
    },
  }
};      

var h = d.host();
h.getAction = function(actionName) {
    if (actionName === 'foo') {
        return function(c) {
            console.log('foo ' + c.m.customerType);
        }
    }

    return null;
}

h.setRuleset('ruleset', r, function (err, result) {
    if (err) {
        console.log(err)
    }
});

var a = d.application(h);
a.run();