Embedded XForms Server for mobile and offline clients #1221

Open
ebruchez opened this Issue Aug 27, 2013 · 7 comments

Projects

None yet

2 participants

@ebruchez
Collaborator
ebruchez commented Aug 27, 2013 edited

Status

The purpose of this RFE is to gather ideas and track progress of an Orbeon Labs investigation project. The purpose of the project is to make the Orbeon Forms forms engine available on clients, including online and offline web browsers and mobile applications.

As of October 2016, there has been some progress on this project, including:

  • conversion of our DOM library to Scala
  • build modularization
  • experimentation with XPath 2 in the browser

We expect to continue progress on this project incrementally.

Subprojects and dependencies

  • Web XForms: XPath 2 API #2127
  • XPath: replace reflective calls with native functions #2214
  • Don't use dom4j XPath APIs #2164
  • Migrate to a minimal version of dom4j #2165
  • Complete conversion of XForms engine to Scala #2825
  • Mobile/offline Form Runner UI

Historical note

With Orbeon Forms 3.7 (back in 2009!), we had an implementation which wasn't bad in concept, but was too limited and based on Google Gears (the precursor to HTML5 local storage among other things). Due to those limitations, Google abandoning Gears and lack of interest from customers at that time, we decided not to maintain it anymore starting Orbeon Forms 3.8. See:

Rationale

The idea is to bring our forms engine to Web apps and "native" mobile apps. This would serve multiple purposes:

  1. supporting offline form management and data entry with a Web app
  2. supporting offline form management and data entry from with a native phone/tablet app
  3. generally having more of the form processing handled on the client, as client capabilities improve

The general idea is to make the code which currently runs on the server run on the client, whether within a Web browser directly or as a native application on a mobile device.

We need to abstract and modularize the XForms engine to make it as small as possible and as independent as possible from other things including the servlet API.

The user-interface part would remain browser technology-based. We don't want to change this at all at first and we wants to leverage either an actual Web browser or a Web view within a native app.

One components that we need to have is some kind of abstraction for the client/server communication. Right now that communication is handled via Ajax in the JavaScript code and we have the XForms server on the server. What we would like to do here is allowing the option to have something local like at simple queue. That doesn't seem to be extremely hard to do.

Platforms

General considerations

  • We plan to compile our code to JavaScript with Scala.js.
    • it is no longer experimental as of 2015
    • it can run in a WKWebView as of iOS 8
  • We have a mix of Java and Scala code and we plan to migrate the Java code that is needed to Scala. (#2825)
  • We are ruling out the use of GWT, which is a tool chain made obsolete (for us) by Scala.js.

Web app

For a web application we have to compile our code to JavaScript. For offline support we can use modern browser APIs.

Native iOS app

The current plan is to use JavaScript as well. In the future, and if it matures and gains the appropriate bindings, Scala Native could become an option as well.

HIstorical considerations

We also considered these options, but have opted not to use them:

  1. Scala -> scalac -> IKVM -> Xamarin.iOS -> ARM x86 assembly
    • iKVM is stable
    • MonoTouch is stable (?)
    • uses optimizing compiler (I think)
    • drawback: painful toolchain
    • benefit: might be the best performance possible as of 2013
    • opens the door of Microsoft phones/tablets as well
  2. Avian VM
    • promising (2013-08-30 update: Scala runs fully)
    • JIT/AOT compilers and GC performance still poor

Abandoned projects, for reference:

  1. RoboVM compiles Java bytecode to native code (project/product is dead)
  2. ScalaGWT
    • run in WebView
    • no JIT in WebView probably
  3. LLVM backend
  4. J2ObjC from Google
    • compiles Java source to Objective-C
    • see also Going under the hood of Inbox
    • would not work for us almost half of our server-side code is in Scala

Native Android application

Android already supports Java bytecode. This means we have two options:

  1. Compile our Java/Scala code to Android.
  2. Compile our Scala code to JavaScript as in the case of the browser and iOS.

We would need to use a WebView probably and then we would need to interact between the application and the WebView. On Android the JavaScript side of a WebView is able to call a Java object that has been exposed.

Modularization effort

The Orbeon code needs to be modularized and abstracted. In particular:

  • client-server connection
  • temporary files
  • form compilation vs. form execution
  • HTTP client for submissions
  • etc.

Subsequent steps

Once a prototype is developed, we needs to think about the general architecture of the mobile application, including lifecycle, how to load forms and so on. Form Runner needs to be included, and local storage at least needs to be used.

@ebruchez ebruchez added this to the Review milestone Jun 26, 2014
@avernet avernet removed this from the Review milestone Oct 17, 2014
@ebruchez ebruchez removed this from the Review milestone Oct 17, 2014
@ebruchez
Collaborator
ebruchez commented Jan 13, 2015 edited

Steps for a prototype

Strategy

As of 2015-01-13, the plan to use Scala.js. This way a single compilation can be used:

  • in standalone web browsers
  • in native apps via web views

This also allows us to master Scala.js, which we want to use more for Form Builder (and later Form Runner).

First goals

We are trying to define reasonable chunks of work which lead us somewhere. First goals:

  1. run some of our code in the browser
  2. run an XForms hello world prototype in the browser

It is ok to hack to get to these goals. The purpose is to get familiar with the tool chain, potential issues, and learn more about what remaining work will be needed.

Known challenges

  • Java code/libraries don't compile with Scala.js right now
  • handling of uploads will be different
  • handling of submissions will be different
    • 2016-09-16: Noting RosHTTP which is 100% Scala and works on the JVM, Node.js and browsers.
  • modularize/refactor
    • we only want to pull in a small subset of our code

Things the client must not need

  • XSLT 2
    • later dynamic XBL might require a solution, preferably NOT using XSLT
  • XPL and processors
  • eXist, iText, Flying Saucer
  • most other XML processing code (MSV, XML Schema validator)
    • XML Schema validation could be added at a later time
  • Java libraries: httpclient, ehcache
  • Java logging libraries (need some kind of wrapper)
  • at first: no HTML handlers
    • client receives always same HTML "template" from server
    • so no full updates at first

General architecture

  • XForms
    • what we really want to run
      • XFormsContainingDocument, model, control tree, control comparator, and related
      • so basically the core XForms engine
    • client takes
      • HTML page produced by server
      • serialized version of static state
        • could be XML or JSON or anything that can rebuild ElementAnalysis trees
        • this way complex XBL logic, XSLT etc. remains on server
    • client initialization
      • all controls are non-relevant
      • initial show does a diff with fully non-relevant tree (null)
    • client operation
      • state handling is changed
        • static state
          • explicitly kept, as it represents the "form definition taken offline"
        • dynamic state
          • app is not multi-user
          • there could be only one such dynamic state at any given time
          • later, we could support switching between form sessions, keeping a number of them
  • Form Runner
    • persistence
      • use some kind of local storage/db on the phone

Q&A

  • Q: Why wouldn't we compile the form on the client too? A: Form compilation right now requires XML, SAX, and XSLT 2, in particular for some key XBL components. We can't rule this out in the future, but for now it doesn't seem unreasonable to say that forms are compiled on the server, and that the client gets HTML + serialized data structures that it needs.

Concrete next steps

1. Initial XForms engine extraction

Goals:

  • must compile with scalac only (no Java code)
  • must only use Scala code/libraries
  • exclude XSLT, XPL, HTTP client, etc. (everything not core XForms)
  • get a mostly extracted XForms engine
  • get a list of remaining library dependencies if any
  • get a list of further tasks that might come up
  • if possible: do this as refactoring which can be merge on master branch

Non-goals:

  • remove all Java dependencies
  • compile with Scala.js

Steps:

  • move things around until they compile ;)
  • write interfaces & stubs where needed, hacking is ok

2. First compilation with Scala.js

Goals:

  • reasonable subset of XForms engine compiles with Scala.js
    • XPath dependency analysis is explicitly excluded
    • use DumbXPathDependencies
  • clarify next steps

Non-goals:

  • code doesn't need to run or do anything useful yet
  • having a functional XForms engine

Steps:

  • if needed, stub out XPath engine, BUT could also hookup Darius XPath

3. Simple client-side Hello World

Goals:

  • run Hello World in the browser

Non-goals:

  • fully functional XForms engine
  • process XForms source on the client

Steps:

  • client/server infrastructure
    • serialize/deserialize static state
    • generate initial HTML with serialized states
    • hookup local Ajax server
  • hookup XPath processor (Darius XPath)
  • interface with DOM for the UI

4. Demo iOS app

Goal:

  • run the above in a simple iOS app
  • must show basic XForms, interactive, in web view

Steps:

  • TODO

5. Solid XForms support

Goals:

  • go beyond Hello World
  • most XForms must work
  • complete XPath support
    • Java/Scala bridge
    • typed values
    • hookup XPath function library
  • complete XForms support that makes sense
    • submissions
    • TODO

Steps:

  • TODO
@ebruchez
Collaborator

form-runner

@ebruchez ebruchez added Mobile and removed Mobile labels Jul 20, 2015
@ebruchez
Collaborator
@ebruchez
Collaborator
ebruchez commented Oct 5, 2016
@ebruchez ebruchez added the Top RFE label Oct 5, 2016
@ebruchez
Collaborator

Immediate concrete next steps:

  • create an XForms module in the build (we have progressed a lot on build modularization already)
  • convert at least P1 classes to Scala (#2825)
  • after that: determine next step for further modularizing XForms engine
@ebruchez
Collaborator
ebruchez commented Nov 4, 2016

+1 from evaluator, with a preference for a web app using browser storage, as native apps can be more challenging to deploy.

@ebruchez
Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment