Support Event Culling via Experiment Engine

[DavidBerdik]

Event culling is currently only available when using JGAAP via the user interface. Support for event culling should be added to the CLI-based experiment engine as well.

Warning: This change will break compatibility with CSV files built for use of the experiment engine up to this point because it will introduce the need for an additional entry to the CSV.

[DavidBerdik]

Useful starting point for researching how to implement this: https://github.com/evllabs/JGAAP/blob/master/src/com/jgaap/backend/ExperimentEngine.java#L183

[mvryan]

It is supported in experiment engine

[mvryan]

JGAAP/src/com/jgaap/backend/ExperimentEngine.java

Line 212 in c4b1180

for (String event : events) {

As you can see here, you can include canonicizers and cullers on individual events. Canonicizers prefixed with @ and cullers prefixed with #. There was documentation outlining all of this but it must have been lost with the wiki.

[mvryan]

I started a branch that uses much clearer and expensive JSON for experiments if you’re looking for something fun to build on.
Here is an example experiment in that framework. It also adds validation as a first class setting.
https://github.com/evllabs/JGAAP/blob/643f677f5fc5173a99099ba82b29381922c181e3/src/test/resources/experiment.json

[DavidBerdik]

JGAAP/src/com/jgaap/backend/ExperimentEngine.java

Line 212 in c4b1180

for (String event : events) {

As you can see here, you can include canonicizers and cullers on individual events. Canonicizers prefixed with @ and cullers prefixed with #. There was documentation outlining all of this but it must have been lost with the wiki.

Oh wow! I did not know about this. Even looking at an archived page of the wiki, it looks like that information is not available: http://web.archive.org/web/20100810065633/http://server8.mathcomp.duq.edu/jgaap/w/index.php/Command_line

Thank you for pointing this out so I didn't have to go hunting myself. This is a better implementation than what I was going to do, actually. I was going to have the canonicizers work the same way as they do in the UI, where they are applied to all events.

I started a branch that uses much clearer and expensive JSON for experiments if you’re looking for something fun to build on.
Here is an example experiment in that framework. It also adds validation as a first class setting.
https://github.com/evllabs/JGAAP/blob/643f677f5fc5173a99099ba82b29381922c181e3/src/test/resources/experiment.json

Ah I did not know about this either. How significant is the performance hit? I think that we should consider replacing the CSV approach with this or at least making it available in addition to the CSV format. It's far cleaner and more user-friendly.

[mvryan]

https://web.archive.org/web/20160527010715/http://evllabs.com/jgaap/w/index.php/Experiment_Engine

…

On Mon, Sep 20, 2021 at 22:50 David Berdik ***@***.***> wrote: https://github.com/evllabs/JGAAP/blob/c4b118061388f132859021b7a7d9dd8e57506893/src/com/jgaap/backend/ExperimentEngine.java#L212 As you can see here, you can include canonicizers and cullers on individual events. Canonicizers prefixed with @ and cullers prefixed with #. There was documentation outlining all of this but it must have been lost with the wiki. Oh wow! I did not know about this. Even looking at an archived page of the wiki, it looks like that information is not available: http://web.archive.org/web/20100810065633/http://server8.mathcomp.duq.edu/jgaap/w/index.php/Command_line Thank you for pointing this out so I didn't have to go hunting myself. This is a better implementation than what I was going to do, actually. I was going to have the canonicizers work the same way as they do in the UI, where they are applied to all events. I started a branch that uses much clearer and expensive JSON for experiments if you’re looking for something fun to build on. Here is an example experiment in that framework. It also adds validation as a first class setting. https://github.com/evllabs/JGAAP/blob/643f677f5fc5173a99099ba82b29381922c181e3/src/test/resources/experiment.json Ah I did not know about this either. How significant is the performance hit? I think that we should consider replacing the CSV approach with this or at least making it available in addition to the CSV format. It's far cleaner and more user-friendly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH3ZGH32ADLQ2EUQ443633UC7XG5ANCNFSM5ELA5H4A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[DavidBerdik]

https://web.archive.org/web/20160527010715/http://evllabs.com/jgaap/w/index.php/Experiment_Engine

Thanks for sharing! I did not realize that there was a more up-to-date version available out there. I'm going to scrape it and put recreating it and updating it where necessary on our list of things to do at some point.

Also, what do you think of my suggestion?

How significant is the performance hit? I think that we should consider replacing the CSV approach with this or at least making it available in addition to the CSV format. It's far cleaner and more user-friendly.

[mvryan]

Yes it takes a bit of a performance hit but it’s not too bad. It is the way to go in my opinion because it enforces best practices better and has more readable results.

If you start here you can see it’s part of an effort to rebuild around Spring https://github.com/evllabs/JGAAP/blob/643f677f5fc5173a99099ba82b29381922c181e3/src/main/java/com/jgaap/rest/JGAAPApplication.java

My thought is if you have this rest based ee you can spin up a bunch ok jgaaps and have them work like a cluster / micro service

[mvryan]

Here you can see we build a confusion matrix that’s way more informative than the hard to parse text files. https://github.com/evllabs/JGAAP/blob/643f677f5fc5173a99099ba82b29381922c181e3/src/main/java/com/jgaap/generics/Experiment.java

…

On Tue, Sep 21, 2021 at 17:06 David Berdik ***@***.***> wrote: https://web.archive.org/web/20160527010715/http://evllabs.com/jgaap/w/index.php/Experiment_Engine Thanks for sharing! I did not realize that there was a more up-to-date version available out there. I'm going to scrape it and put recreating it and updating it where necessary on our list of things to do at some point. Also, what do you think of my suggestion? How significant is the performance hit? I think that we should consider replacing the CSV approach with this or at least making it available in addition to the CSV format. It's far cleaner and more user-friendly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH3ZGHPKXCTVKPYFAUOF43UDDXT3ANCNFSM5ELA5H4A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[DavidBerdik]

Nice! I like the idea of a Spring-based JGAAP. Being able to do it like this would be much nicer than the hacky methods that I have used to do large-scale experimenting in the past. (Populate a database table with experiment configurations and use some wrapper code to pull experiment configs from the table.)

It's not something that I can play with right now (too many other, higher priorities), but I definitely would be interested in exploring this further if you do not move forward with it yourself.

Here you can see we build a confusion matrix that’s way more informative than the hard to parse text files.

Cool! I think if the plan is to phase out the hard-to-parse text files, it would still be worth making the results available in a JSON format. Whenever I have a large set of results from an experiment, I use a set of Python scripts to read them. The thing is, those scripts, while functional, have to do awkward string manipulations to work and sometimes require slight modifications to work depending on the scenario. It would be great to have that output available in a JSON format so that if necessary, these kinds of tools could be used but would not require awkward string manipulation.