Native modules

[jgaehring]

Just had another pie-in-the-sky idea...

I wonder if it's possible to distribute native "modules" over a CDN or similar host.

Since the native/client app is a single page application, with JavaScript primarily responsible for most of the rendering, I wonder if it could be possible to load those scripts from another source. Those would be our modules, and they could correspond to farmOS modules. The native app then would just ship as a "core", with a few common modules, perhaps, then query the farmOS server to find out what additional modules it needed. After it got some addresses from that query, it could hit the CDN and run those scripts, potentially storing them on its file system for later use so it doesn't have to run a http request every time it needs them.

This would require some sort of Drupal module added to farmOS, too, since it would have to control and respond to those requests with address(es) for where to find the right JS files the native app needed for, say, the native Observations or Livestock modules. The Drupal module could possibly handle other tasks the server needs to perform in order to support the native app, like installing the restws module and other dependencies (less sure of how that works). It could also run some sort of query of what modules were currently being used in that installation of farmOS, and accordingly what native modules can be "installed" (ie, requested from the CDN) that would support those farmOS modules.

The main value proposition would be a more fluid user experience, for those who are using the native app along with farmOS community modules. It could also/especially be useful while developing native support for more and more farmOS modules incrementally, and perhaps allowing other developers to contribute, without needing a monolithic release to the app store every time. I'm sure this would present all kinds of hurdles with CORS and security would have to be a top concern, but at a glance it seems feasible.

Curious what your thoughts are, @mstenta. Obviously not a high priority now, but an idea.

[mstenta]

Yes! This could be a great way to mimic the modular nature of farmOS in the native/client app.

Security should be a paramount concern with this, of course - because it implies downloading additional JS code into the native/client app - which opens up a possible attack surface.

One idea might be to package the JS code in the farmOS modules themselves (alongside the Drupal/PHP code). So it would always be coming from the farmOS site you're using (not from other sources/third-party URLs/etc). This would at least limit the security and trust to the farmOS server. If that gets compromised, all bets are off, but it feels more manageable than handling code from just anywhere.

The other benefit of that is that the code for each module can be self-contained in each repository. So there isn't a need for two repos for each module, for instance.

An example workflow:

1. User has a farmOS site.
2. User installs the Grazing module on their farmOS site.
3. User installs the native app on their phone.
4. The native app connects to the farmOS site, and asks for information about modules.
5. The farmOS site replies with a list of modules, including the machine name (farm_grazing), the location of the client/native JS file for it, and a signature of the file that can be verified by the client/native app (signed using the Drupal site's private key perhaps - need to also provide a pubkey for verification - need to look into that).
6. The client/native app downloads the JS file, verifies the signature, and installs it (whatever that means).

Another consideration: the client/native connects to a single farmOS site - but what happens if you want to change it to a different farmOS site, that has a different set of modules? Need to be able to toggle them on/off accordingly. Or just dump and re-download? Some things to figure out...

[jgaehring]

One idea might be to package the JS code in the farmOS modules themselves (alongside the Drupal/PHP code). So it would always be coming from the farmOS site you're using (not from other sources/third-party URLs/etc).

Yes. That absolutely makes the most sense, agreed.

The other benefit of that is that the code for each module can be self-contained in each repository. So there isn't a need for two repos for each module, for instance.

I've been thinking a lot about repository management lately, and what works best with a particular frontend development workflow. Having things broken up into smaller libraries is the way I want to go in the future, because that should make it easy to pull that library in for hosting on the production server, as well as running it in a test server on a local device running Webpack. Hopefully that will also make it more of a moot point whether that library lives in one large repository with all the other client libraries, or in the same repo as the Drupal module.

That example workflow is spot on. Love it!

Another consideration: the client/native connects to a single farmOS site - but what happens if you want to change it to a different farmOS site, that has a different set of modules?

Good thinking. I hadn't considered that, but yea, we'd definitely need to allow for that. I think some sort of toggle should be feasible. Vue is pretty good at conditionally loading specific components, which is how those modules would be represented within the framework. The client's core library then would just need to keep some sort of manifest of all the settings, modules, user info, etc, associated with each farmOS host site, and of course which site was currently being displayed and so which manifest to use.

[mstenta]

We could even try this approach out with the observation app piece as a start. That could be provided by the farmOS Log module. So really, then the native/client would only be the authentication code, ajax code, and module/component loading code (maybe some other stuff I'm forgetting). That could be a good next step after beta release, and first step towards this modular component approach...

[jgaehring]

Another thing this kind of module system could do, from the server, is decide what native modules the client needed based on which user's device was requesting it, what tasks/roles they were assigned and what kind of permissions they had. With quick forms, this level of personalization could be taken further than with regular modules, too; for instance, if the farm had a mechanic who had the app, they could get forms for asset movements and other specific tasks most pertinent to their responsibilities, rather than getting quick forms for, say, watering seedling trays; meanwhile the farm manager could get crop planning quick forms that others wouldn't need, etc, etc. Perhaps it would be possible to have some sort of decision tree in the controlling module at the server, which associates certain quick forms with the kinds of tasks assigned to the person, or some other logical branching to achieve this.

[mstenta]

Yes!!!

[jgaehring]

I just updated the section "A Path Towards a New Architecture" in my fork of the farmOS docs, to reflect some of these ideas about native modules, along with some ideas from #13.

@mstenta, when you're not too busy, I'd be curious to know what you think of the section Characteristics of the Client Architecture section, particularly where it concerns the "client module delivery" system, which is my first attempt to describe these native modules in a little more technical detail.

[mstenta]

Sorry this slipped between the cracks. Yea that looks like a good outline! I wonder if this kind of use-case has been handled anywhere else? Or if this is a new idea?

[jgaehring]

I imagine there are some other examples of that kinda thing out in the wild, yea. Possibly some of the other frameworks built on top of Cordova, though I'm not aware of anything that's an exact match. Plus I'm not sure how much of those frameworks are open source, were we to try forking something.

As the idea has germinated I have wondered if it would be possible to adapt something like Cycle.js to the purpose, or if something like Next/Nuxt could have some ideas I could borrow from, even if they weren't really intended for the kind server-to-native UI rendering we'd want to do. I'm sure there are probably even closer parallels out there, so I should definitely try to do more research.

Or maybe I'm just trying to reinvent server-client architecture? :)

[mstenta]

Yea, it is a reinvention of client-server architecture in some ways. What makes this case unique is that we want the client to be able to operate without the server. Sort of like putting on a scuba tank. It takes what it needs so that when it goes under (offline) it still works.

... whoa ... I decided to Google "scuba.js" and: http://mydea.github.io/scubajs/

Edit: not suggesting that library - just funny that it exists, and is named based on the same idea it seems. :-)

[mstenta]

Do you think this sort of architecture would be strictly part of the "native" portion of this project? Or could it be useful in other contexts? Ie: If you could run the client JS app locally in a browser, you would want it to be able to pull in modules from the farmOS server as well, right?

[jgaehring]

Do you think this sort of architecture would be strictly part of the "native" portion of this project? Or could it be useful in other contexts? Ie: If you could run the client JS app locally in a browser, you would want it to be able to pull in modules from the farmOS server as well, right?

Yea that's what service workers would do; that sort of architecture is already baked-in to how they work, which is really nice. It's actually what triggered this whole line of thinking for me. And of course the web is purpose-built for pulling in new documents and scripts ad hoc. So it might be overkill or reinventing the wheel for the web part, but I think it would still be nice to have some shared module, the "client module delivery system" as I think I called it, that can control what modules the native client gets, as well as any other client apps, like the offline module.

Haha, funny that scuba.js is actually somewhat relevant though.

[jgaehring]

If we were using RN, wonder if this would serve the function: https://github.com/Microsoft/react-native-code-push

[jgaehring]

Closing this in favor of #217