Plugin architecture overhaul

[hsoft]

Plugins came very late in moneyGuru's design and, even then, they were implemented in a "cobble something together and see if it holds".

@brownnrl in #26 is building something that involves new plugins and the shortcomings of the system as a whole are becoming unbearable. It's time to design the thing a little better.

Here's a list of problems with the current system:

All activated by default

As soon as a plugin is present in the plugins folder, it's activated. It's not always a desirable situation and one has to manually remove/put back files in there to manage plugins.

We need a new view that allows us to list plugins and activate them from within moneyGuru.

No versioning

We have no way of managing plugin updates and we don't know for which version of moneyGuru plugins were designed. The second part doesn't matter much yet because the plugin API hasn't moved (it hasn't been used!), but if we're serious about having a proper plugin API, we might as well handle versions properly.

The "plugin update" part is already a problem though with plugin examples. The plugin folder, when it doesn't exist, is created and then automatically populated with the examples bundled in moneyGuru. If the folder already exists however, we don't touch it. The result of that is that updates to plugin examples aren't effective until the user wipes his plugin folder.

[brownnrl]

Plugin versioning and enabling are two excellent points to consider. I have some other areas that need thought put in before I think we'd see plugins become useful.

Input Panes

We have to provide plugin authors with an platform agnostic method for collection input parameters for their plugins. These could be single pane forms with a set number of input widget types.

Example : Let's say we're developing a mortgage calculator plugin. I would need a way to collect some simple information similar to an online form (rate, term, extra payments, etc).

"Recommended" Storage API

The plugin will have whatever permissions to the file system that the application has, but it would be nice to simplify for the developer a platform neutral location for them to store their plugin data. It would also give the developers the ability to say: Go to my data folder under the moneyguru plugins directory, where you'll find file X that contains information Y.

Example : For our mortgage calculator, I write the functionality to fetch common interest rates and I want to cache them somewhere. Or for the case of the (#26) import autofill plugin we want to store a cache of what the user inputs were correlated to import items.

Plugin specific area for file format

The plugin may need to store some specific information pertaining to the current document and it's accounts. This is where the versioning you mentioned is critical.

Example : See the next section.

Break out the core functionality into nice interfaces

Scheduled transactions : There should be some interface to make a custom scheduled transaction that does not have a constant amount.

Example : For our mortgage calculator, we create a liability for mortgage principle and an expense of mortgage interest. We create two custom schedules using the schedule API for each that split the payment across these accounts. This implies that we'd need to store the information about the schedule being generated by the API and a plugin in the file format "somehow."

Budgets : There should be some interface to make a custom budget that behaves differently depending on how the plugin developer intends.

Example : Several -

• Some budgets I have are fixed in that I don't want to spend more than X in a given time period, and I don't want the balance to roll over.
• Some budgets I have are rolling, but the rolling date may be weekly, monthly, annually, or bi-annually.
• Some budgets I have "build up" where they behave like a phantom account. My money is in cash / savings, but I secretly have a desire to earmark it for that new computer. I don't have a "new computer asset account", it's just that some of the money in my account is painted with my "new computer color".
• Some budgets I have "wind down" where they behave like a decreasing phantom account. For instance, I like to do some recreational flying but darned if it isn't very expensive. So a set aside X dollars annually and draw from that whenever the weather is nice.

Graphs and Reports : There should be a method to show a graph that is simplified for the purposes of plugin development. Additionally, when #399 (which should probably be titled "Reportlab integration", but I definitely like that idea, I will try to attack it at some point after my own queue empties if no one else has), the process should be targeted that plugins can contribute to the reports.

[hsoft]

These are all good ideas, but they're more in the realm of "expanding plugins" with new APIs and capabilities rather than refactor them. Is it bad to keep these ideas here? bah, I guess not (maybe we can update the title of the ticket), but we shouldn't try implementing all of this at once, we'll never release anything.

For the short term, I was more thinking in line of "whatever is needed to get #26 out of the door".

[hsoft]

Nothing stops us from considering this ticket a "brainstorm" ticket and spawn individual tickets from it.

[brownnrl]

So, minimally, for #26, we should focus on the items you mentioned: versioning and a configuration pane for plugins. There should probably be a core.gui window (maybe an offshoot table from preferences?) that enables/disables plugins and displays their name and versions.

From the few things I mentioned, we'll need a storage API. To start out can just be a method that returns a moneyguru recommended paths for temporary and local data storage directories. We'll need to store user contributions to imports, and configuration data.

We may also want to experiment with some input widget panes for plugins as well. We may want to configure how the autofill plugin will work or how the user would like it to behave. I have some code I developed from another project that might work out well for this purpose. It is a sort of dynamic form generator from string labels and dictionaries that is PyQt-specific, but maybe it could be adapted to the cocoa side as well.

[brownnrl]

Wanted to get this thought down: using a python egg file as a file format. The idea is related to #302 where attachments such as receipts or cancelled check images are desired to be saved and #378 where the suggestion of a zip file format for saving transactions in chunks of time for lazy loading.

What is proposed would be akin to macros in a word processing document. Some plugin logic may be specifically targeted to a workflow inside a single moneyguru file (or set of accounts and transactions). The user may just want to do something very specific with their accounts that couldn't or shouldn't be generalized to a plugin enabled on all moneyguru files (which would be kept in the local data as is done currently).

So you would include the native XML file (or series of json files or what have you ala #378) storing data on transactions / schedules / budgets / accounts, image and file attachments, minimal code for egg file format, and custom plugins.

There would be a security consideration, where the user would select whether python code should be loaded or if the XML and image data should be just read as a zip file. If it's a trusted file, then you can import the contents and enable the embedded plugins.

The last part of the idea is a little more far fetched: It would be interesting if you could include the core module code (less the gui module which wouldn't be necessary) along with some helper or standardized API that any python3+ interpreter could import the egg file, load it's contents, make changes or perform analysis, import/export through that API, and save changes to the native data format. They would still have to have core dependencies such as lxml, etc. A little bit of a wild idea, but it would be interesting.

[hsoft]

Unless there was an overwhelmingly great use case for it, I would be against adding any executable code in a .moneyguru file. Python code cannot be sandboxed and people don't expect such a file to possibly be a threat. I would be very sad if moneyGuru came to be a malware vector.

[brownnrl]

Ok, taking the egg file idea off the table.

To throw some ideas out and speaking in terms of pure-data I have a couple of thoughts (in semi-question form).

• Leaving an XML element open for plugins to store file-specific data? Name and version attributes could tell moneyguru where to send the plugin data. Something along the lines of:

<moneyguru-file>
... accounts, transactions, recurrences, budgets
<plugins>
 <plugin name="example-name" version="XXX">
  (plugins control their data here)
 </plugin>
 <plugin name="import-regex-mapping" version="0.1">
   <rule>
     <regex>TO SHARE 002</regex>
     <transfer>bank 1 checking</transfer>
   </rule>
 </plugin>
</plugins>

• Conversion to moneyguru in a zip file format and giving the plugins a folder to store file-specific data? Inside the zip folder

|- example_file.moneyguru
|-- Data (See #378)
   |-- accounts.json
   |-- 2012.json
   |-- etc.
|-- Attachments (See #302)
   |-- XXXXX1.png
   |-- XXXXX2.png
   |-- XXXXX3.png
   |-- etc.
|-- Plugins
   |-- ExamplePlugin
      |-- version.txt
      |-- (other files up to the plugin)
   |-- ImportRegexRulesPlugin
      |-- version.txt
      |-- rules.txt

See, as an example, a plugin someone developed for gnucash which had a rules.txt file.

• Adding identifiers to transactions, recurrences, budgets, and any other "core" data structure so plugins can reference them? For example, if image attachments were implemented as a plugin instead of a core concept. In XML, maybe this would look like:

<moneyguru-file>
<transaction uuid="676b452e-0665-41ae-b259-f16d415774f0" date="2017-06-24" mtime="1389753346">
<split account="bank 1 checking" amount="USD -18.03"/>
<split account="mortgage interest" amount="USD 18.03"/>
</transaction>
<plugins>
  <plugin name="image-attachments" version="0.1" tags=">
    <attachments transaction-uuid="676b452e-0665-41ae-b259-f16d415774f0">
      <attachment>(let's say a base64 encoded PNG image akin to IPYTHON image representations</attachment>
    </attachments>
  </plugin>
</plugins>
</moneyguru-file>

Or in the zip file, the plugin would have it's own xml or json file for mapping uuid's to file names in it's own sub-directory of the zip file.

[hsoft]

These are all good ideas (but please, let's not implement all of this at once!).

For the UUID thing, we'll have to think about it, but I'd tend to think that it's overkill compared to a more ad-hoc method of cross-referencing (date+seq, for example, which would have the advantage of being more easily findable in eventual yearXXXX.json files without the need for an index file). The feature could be transparent to plugins and we could make things so that all the plugins have to do is to simply reference a Transaction instance and we take care of the referencing in core.saver.native. This way, we wouldn't need to expose any implementation detail in the developer documentation.

[brownnrl]

hmm. Have to think about a reasonable api for referencing using the instances then.

Yeah, I was just putting ideas on paper. No intention of starting down any roads until my plate is clean again, or their is a dependent need from the current work.

[brownnrl]

How would you feel about using a zope.interface style system (or exactly that) for plugins? Where functionality isn't indicated by an isinstance call, but by a InterfaceName.implementedBy(klass) type of check?

Then, you would have plugins that looked like...

class MyPlugin(Plugin):
    implements(IImportAction)

    def can_perform_action(...):
        [...]

    def perform_action(...):
        [...]

Not sure how it would all work with current inheritance. But an aspect to consider for plugin architecture.

[hsoft]

It's an interesting possibility if we end up with a lot of plugin types. But if we were to take that road, I'd be more inclined to use Python's ABC rather than zope.interface.

[hsoft]

Slowly but surely walking the path towards pluginification. I plan to implement #451 for the next release (2.10)

[hsoft]

I'll scale back on this. Nobody ever wrote an external plugin, it's not worth maintaining an API. I'll slowly close the API and integrate existing plugin directly in moneyGuru's code.

Also, for having recently worked within the newly designed import structure, it's way too complicated now and much of this complexity is simply to allow arbitrary plugins to plug arbitrary features in. Let's stop this.

[hsoft]

By the way, sorry @brownnrl I know you worked a lot on this plugin design, but the complexity it brought to the code was just too much for plugins to be worth it. As I strip the thing, I see the code simplify and it's a breath of fresh air.

Had plugins been used, it might have been worth it, but since it wasn't, this complexity is just a weight holding the project down.

[brownnrl]

I completely understand. In truth, I’m glad you are not only still developing moneyguru but taking it in a consolidating direction. Choosing a select set of features and not having multiple supported platforms (beyond the feature sets already provided by Qt) will probably increase adoption.

I’ve been using moneyguru twice a month with full solid coverage of my family’s finances since mid-way through 2016. I feel guilty my contributions dropped off completely a few years ago, work and home life picked up and I found my free hours dwindling. With the import plugin I just started doing way more development work than actually importing transactions, and the trade off didn’t seem worth it.

But I would like to help again where I can. I do have a few bugs to note, and I echo @tuxlifan in thinking that an envelop / zero sum budget system would be a killer feature.

Thank you again very much for keeping moneyguru alive.