is there any way to reduce the load(build) time of the rules from a resource ?

[jinagamvasubabu]

Grules is performing very well with executing of a fact against 1000's of rules so far in my project and i am very much satisfied with it. But loading rules is taking some time (~10's of seconds) for 2000 complicated rules. I know loading is one time activity which i do whenever we add/edit a new rule or container startup and not every time.

But Is there any way to make Grules building rules from resource better just like executing a fact ? instead of taking seconds ?

[niallnsec]

I have the same issue, I have a pretty large and complex ruleset which takes about 2min 30s to load on an intel i9. Once loaded speed is great but the slow start is frustrating. I have been trying to create a way of pre-compiling rules and loading them in one go, but haven't managed to get it to work properly yet.

[niallnsec]

Yeah thats what I was thinking, similar to the way that tools like YARA allow you to precompile large rulesets. As far as I can tell, the majority of the slow down in parsing my ruleset is the deduplication of conditions and validations on format.
So in theory all we need to do is run the loading/deduplication process and export the generated maps. The problem I ran into is that some of the structures are more complex, requiring pointers which would have to be stored and then resolved somehow, which starts to get a bit more complicated.

[newm4n]

Well, you see... ANTLR4 is reading the GRL to construct the AST Graph.

Seems this parsing (to build the AST graph) took a long time. Its more likely the ANTLR4 parsing issue that's slow, if not the RETE indexing issue that scans all AST node to ensure no duplication of Expression/Expression Atoms. The RETE is important, for execution speed reason.

One solution that I can think of, is to save the KnowledgeLibrary (that contains the KnowledgeBases) into a binary file.
The binary file is a representation of AST graph, that loading from it is straight forward.

All we need to do now, is to save/load the KnowledgeLibrary into/from file, that would not require ANTLR4 parsing and RETE reindexing.

[newm4n]

Yeah thats what I was thinking, similar to the way that tools like YARA allow you to precompile large rulesets. As far as I can tell, the majority of the slow down in parsing my ruleset is the deduplication of conditions and validations on format.
So in theory all we need to do is run the loading/deduplication process and export the generated maps. The problem I ran into is that some of the structures are more complex, requiring pointers which would have to be stored and then resolved somehow, which starts to get a bit more complicated.

I will create a new improvement issue for this, Kind of marshaller/un-marshaller of KnowledgeLibrary to byte array, as POC.

[niallnsec]

When experimenting, the main problem I had was that when you follow the structures in the KnowledgeLibrary you eventually get to the Expression struct. Its a pointer type and (please correct me if I am wrong here) a single Expression struct is often referenced by multiple rules. Since it contains contextual information it has to be a single instance, so when you serialise you have to encode it in such a way as to preserve the internal pointer references. As you go deeper those references get more complex. It can certainly be done but I think it's going to be tougher than it looks on the surface.

[newm4n]

Yeah, I'm well aware of that. That's what the RETE indexing is doing. Making sure that expression are reused by multiple rule, expressions. Its not a simple object-graph, it could be a collection of object-referencing-object chaos. :D

[newm4n]

A new PR #182 is created for this. I have not tested it on Huge rule set, as I don't have Huge set.
But I am confident that it should load much faster to load rule set from their binary form, compared to their GRL script form.
So, the work flow is

1. To build the binary, you load your GRL into KnowledgeLibrary as normal.
2. Once built. you can use func (lib *KnowledgeLibrary) StoreKnowledgeBaseToWriter(writer io.Writer, name, version string) error. The writer could be a plain io.Writer or a wrapped archive/zip Writer for smaller binary size.
3. Next time, you don't have to load the rule from GRL, you can straight use func (lib *KnowledgeLibrary) LoadKnowledgeBaseFromReader(reader io.Reader, overwrite bool) (retKb *KnowledgeBase, retErr error) function to load from your binary rule set, straight away from KnowledgeLibrary, no need Builders.

Or, you can make your server to start and only read rule set from a binary file. While your development PC can be used to build/produce the binary for later deployed to the server.

[jinagamvasubabu]

in the meanwhile i will give a try with the rules which we created for benchmark tests and see if i can store it as binary and load the same.

[newm4n]

@jinagamvasubabu , what do you mean by "couldn't get the access to all the rules from production(data set)" ?

[jinagamvasubabu]

actually i am trying to test this in my local machine with the production data, but for some reason i am not able to get the rules dump from production DB.

[newm4n]

Can you load all of your rules in normal way ? I mean, using GRL ?

[jinagamvasubabu]

sure @newm4n, will try tomorrow

in the meanwhile i have tried to check with benchmark rule sets (999 duplicated rules and 1 unique rule) and it looks good

please check this draft PR - #184

[newm4n]

I made a test.

Created 5000 dummy rules... like

rule ThemselvesMachineEvery "1 Real Vote" salience 79 {
	when
		prove.All == four.Music() && 
		shake.Value <= legal.Contain()
	then
		set.Language();
		model.Mouth = church.Beautiful();
}

rule RunAroundSecurity "2 Education Field" salience 66 {
	when
		people.Very == personal.Eight() && 
		customer.Style <= dark.According()
	then
		itself.Better();
		bad.Computer = blue.Describe();
}

... and 4998 more ...

From this dummy rules, I got the following result.

    GRBPerformance_test.go:75: Load GRL duration   : 63722 ms
    GRBPerformance_test.go:76: Saving GRB duration : 7433 ms
    GRBPerformance_test.go:77: Load GRB duration   : 5669 ms

• Load GRL duration : To load a GRL script containing 5000 dummy rules. Took 64 seconds, I tried this few times with max 71 seconds.
• Saving GRB duration : To save the content of a KnowledgeBase into binary file.
• Load GRB duration : To load all rules stored in binary file back into KnowledgeLibrary. Took 5.6 second. I tried this few times with max 7 seconds.

The content of knowledge base loaded from GRL Script, compared to loaded from Binary is identical.

I will add my test code into main repo ASAP.

[newm4n]

There you go
https://github.com/hyperjumptech/grule-rule-engine/blob/b802e3067522e8d4bcab7b345e7e46cba2ca2714/examples/benchmark/GRBPerformance_test.go