Proposal: External secondary instances (as in CommCare)

[MartijnR]

Also described here from an ODK Collect perspective and in an old wider discussion here.

As in https://dimagi.github.io/xform-spec/#secondary-instances---external, this feature just adds support for src attribute on a secondary instance and a jr://file connector for the src value.

<instance src="jr://file/countries.xml" />

I propose to start with only supporting the jr://file connector for now, and then build upon this feature in the future (e.g. by adding CSV support). This means the same resource retrieval method as jr://image, jr://audio and jr://video can be used.

Until now there is no proper agreement to add external data. This addition would not require re-inventing the wheel and as a bonus brings us closer to CommCare, raising the prospect for future compatibility.

[lognaturel]

The one thing that makes me uncomfortable about this is the name of the protocol (jr://). It implies some connection to JavaRosa but of course this specification aims to be implementation-agnostic. orx:// would probably be a better choice. I talked to @MartijnR about this and the big value to having the name be jr:// is that it matches the Dimagi spec which is also important.

[MartijnR]

Sorry, I didn't see your response!

it matches the Dimagi spec which is also important

In addition, in this case jr: is not used as a namespace prefix but a URI Scheme (yeah, I just looked that up - I used to call it protocol), so it doesn't actually refer to http://javarosa.org/xforms and it doesn't change if you change the https://javarosa.org/xforms prefix from jr to something else in an XForm document (which is a totally valid thing to do). It's just unfortunately and 'coincidentally' exactly the same as the most commonly used namespace prefix for the JavaRosa namespace. This makes its use a lot more palatable in my opinion.

The biggest reason to use it for external data is to make it consistent with jr://image, jr://audio and jr://video.

Definitely or:// or something like that (for all of them) would have been nicer in hindsight.

With respect to agreeing to add this to the spec, I just want to make sure we're all on the same page and are voting on the its worthiness and technical merit. It's not some sort of commitment for a particular time when it is implemented in the various tools. Normally, the spec is ahead of the implementation, and should be.

[lognaturel]

Right, right, it's unfortunate in the jr://image case, too. But so be it, inconsistency would be even more confusing. Let's add a little bit about the URI Scheme somewhere in the spec to provide context.

getodk/collect#275 also has a good discussion related to this including Dimagi's experience.

This all seems like a good idea from an XForms / spec design standpoint but the more I learn, the more I find it difficult to divorce this question of how external data should be handled from some discussion of implementation and from how people are writing forms today.

[chrislrobert]

I have to admit, this idea of allowing the spec to drift off to become some "should have been" version of reality, divorced from implementation, makes me nervous. It then becomes more an aspirational thing -- or, worse, a description of how some particular group of generous spec contributors would like things to be -- as opposed to how things are. I can definitely get the arguments in favor of that approach, but I'm just saying that it makes me nervous. It might also create demand for a "shadow spec" (or "spec as implemented") that better matches current implementation.

…

On Thu, Jan 19, 2017 at 1:03 AM, Hélène Martin ***@***.***> wrote: Right, right, it's unfortunate in the jr://image case, too. But so be it, inconsistency would be even more confusing. Let's add a little bit about the URI Scheme somewhere in the spec to provide context. opendatakit/collect/#275 also has a good discussion related to this including Dimagi's experience <getodk/collect#275 (comment)> . This all seems like a good idea from an XForms / spec design standpoint but the more I learn, the more I find it difficult to divorce this question of how external data should be handled from some discussion of implementation and from how people are writing forms today. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#50 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIO0Hqjo_6BStukqJHjRUIpElOXjRFprks5rTvy0gaJpZM4LK1-6> .

[MartijnR]

find it difficult to divorce this question of how external data should be handled from some discussion of implementation and from how people are writing forms today

I understand. In that respect I think we're lucky that XLSForm is so popular, as we could change things behind the scenes if we wanted when it's safe to do so.

However, I think we can really leave that as a separate discussion and take it one step at a time. The important thing to realize, also in relation to @chrislrobert's comment, is that there should be no implication to break existing XForms or XLSForms at all if they are currently working in a particular client.

This is about providing a flexible path forward for extensions to external data that doesn't involve going further off the deep end (re. search() appearance and query attribute). We need external data in this spec. An organization such as Enketo wouldn't want to follow a spec that didn't have such a crucial feature.

If the concern is with actually creating external XML documents, I think that's a concern that can be dealt with by the form/data servers. E.g. the real power of external data documents comes from the ability to dynamically create them (e.g. from previous surveys). I believe some are already doing this. They could be created in any format. Querying this data can be done in the same way. E.g. it's easy to use pulldata() for these documents without backwards-compatibility issues.

I am trying to be careful not to increase the scope of this issue, but if there is a specific concern, I think there may be some ideas to address them satisfactorily. Fyi, this is the future CSV support, I was referring to as a follow-up.

[chrislrobert]

Okay, makes sense. Just one other point on the idea of pulling everything into XML/XForms: that strategy may just not work for the JavaRosa implementation. One reason that we went with the renegade approach to search() and pulldata() was because JavaRosa couldn't handle massive lists of XML/XForms choice options without having crippling memory and performance problems. With our implementation, you can take 2,000 choice options, pull them out of the XForms XML and put them into an attached .csv (which we translate into a SQLite DB on the client side), and it starts performing like a dream. I may be misunderstanding your "the future CSV support <https://enketo.org/xforms/#csv-support>" implementation, but if it does something pyxform-ish to generate all that XML, JavaRosa will choke and die, plain and simple. But perhaps the "transformation approach" can be implemented such that external instances can be queried dynamically from some kind of local DB and the actual options don't all need to be loaded into memory? I'm betraying my ignorance of XForms implementations here: I don't know how much can be essentially "virtual" vs. how much relies on actually loading everything up into memory. This could be another area where our interests and users simply diverge from others': we've built our company around long, complex surveys that pushed at the limits of ODK 1.x, and we pushed those limits outward, bit by bit. We now have thousands of users with very long, very complex field surveys, and that places further constraints on the kinds of designs that we adopt or support. The traditional external-itemset and cascading-select implementations just didn't scale enough for us, so part of what we were trying to do was break out of those XForms limitations at run-time...

…

On Mon, Jan 23, 2017 at 1:14 PM, Martijn van de Rijdt < ***@***.***> wrote: find it difficult to divorce this question of how external data should be handled from some discussion of implementation and from how people are writing forms today I understand. In that respect I think we're lucky that XLSForm is so popular, as we could change things behind the scenes if we wanted when it's safe to do so. However, I think we can really leave that as a separate discussion and take it one step at a time. The important thing to realize, also in relation to @chrislrobert <https://github.com/chrislrobert>'s comment, is that there should be no implication to break existing XForms or XLSForms *at all* if they are currently working in a particular client. This is about providing a flexible path forward for extensions to external data that doesn't involve going further off the deep end (re. search() appearance and query attribute). We need external data in this spec. An organization such as Enketo wouldn't want to follow a spec that didn't have such a crucial feature. If the concern is with actually creating external XML documents, I think that's a concern that can be dealt with by the form/data servers. E.g. the real power of external data documents comes from the ability to dynamically create them (e.g. from previous surveys). I believe some are already doing this. They could be created in any format. Querying this data can be done in the same way. E.g. it's easy to use pulldata() for these documents without backwards-compatibility issues. I am trying to be careful not to increase the scope of this issue, but if there is a specific concern, I think there may be some ideas to address them satisfactorily. Fyi, this <https://enketo.org/xforms/#csv-support> is the future CSV support, I was referring to as a follow-up. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIO0HsSMlWne4fGgasE4lgMD3Z9UmwF6ks5rVO4JgaJpZM4LK1-6> .

[MartijnR]

can be implemented such that external instances can be queried dynamically from
some kind of local DB and the actual options don't all need to be loaded
into memory

Exactly. If I understand correctly (though they lost me when they tried to explain), that is exactly what CommCare has done. So the language is XPath, but the underlying implementation could do whatever they want such as storing the data in an a db and transforming an XPath query into a db query (and simpler, converting a pulldata function call to a db query). That's no concern of the specification.

[chrislrobert]

Amazing! If somebody's written a flexible XPath->SQL transformation for expressions, that would be incredible! I'm a bit in awe if somebody pulled that off in a way that's not too limiting..!

…

On Mon, Jan 23, 2017 at 1:48 PM, Martijn van de Rijdt < ***@***.***> wrote: can be implemented such that external instances can be queried dynamically from some kind of local DB and the actual options don't all need to be loaded into memory Exactly. If I understand correctly (though they lost me when they tried to explain), that is exactly what CommCare has done. So the language is XPath, but the underlying implementation could do whatever they want such as storing the data in an a db and transforming an XPath query into a db query (and simpler, converting a pulldata function call to query a db). That's no concern of the specification. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIO0HvUqNR0Y9Tee50f2Zyy6t8Y8HnLFks5rVPYTgaJpZM4LK1-6> .

[MartijnR]

I think they do something more clever than a one-to-one transformation of XPath into SQL. Something involving virtual DOM (fragments?) and static analysis of the expressions and indexing to keep lookups very fast. They definitely don't deal with a huge XML documents in memory. Would be worth digging into their code I think.

So ready for a PR for this issue? (and a new proposal for CSV support)

[lognaturel]

I had to give this some more thought but I am now sold. As @MartijnR says, it's a syntax that we can all agree on and that can be a path to greater interoperability. It also enables nice things like an alternative to preloads for session data. There's likely a way to match the current Collect "external itemset" functionality to this syntax and even if that's not done for a while, it's fine as long as we document the differences somewhere (I'll create a separate issue for that on the Collect side).

I had a fun adventure through the CommCare code. And then I realized that unless I'm massively missing something, there's already some support for external secondary instances in ODK JavaRosa. Additionally, it is a part of the original OpenRosa XForms spec that I believe this spec is based off of. This is kind of a separate question but it'd be nice to know how/why this spec differs from that one.

This is where secondary instances are identified in the XForm. In ODK's JavaRosa fork, they are stored in the FormDef in a map from instance name to FormInstance objects here. In Dimagi's fork, they are represented as ExternalDataInstance objects and stored in a map from instance ID to ExternalDataInstances also in the FormDef. In both, the EvaluationContext is set based on the main instance and the secondary instances. XPath expressions are evaluated within that context.

Dimagi has done a lot of (good) refactoring so it's time consuming to compare the two javarosa forks. I believe the way that they've gotten some better performance is by using that ExternalDataInstance class. In ODK javarosa, non-main instances are parsed and their data is loaded in XFormParser. On the other hand, I believe the Dimagi implementation does some selective parsing. @ctsims, I'm seeing your name on a lot of that Dimagi-specific code. Could you please let me know if I'm sort of on the right track?

I thought I could use the syntax described in the OpenRosa XForms spec and at least print something from an external instance in ODK Collect but I wasn't able to get anything working (it just fails silently). I'm not sure whether that's because something is missing in the implementation or just because I'm building the form wrong (I don't really know where to put the external xml file, for example).

I also see that this is actually already in XLSForm. It seems really strange to me that it would be there and not here. Certainly not to say that @MartijnR did the wrong thing at the time since this spec has been neglected but I'm wondering whether we should try to move things here first in the future (I don't know).

[MartijnR]

Great! I'll create a PR and a new proposal for CSV data.

Funny you found some code ODK Javarosa already. Indeed there is no inventing involved for this feature.

should try to move things here first in the future

I agree. I'm glad we're having great cross-organization commitment to managing an XForms spec now. I'm very optimistic we will completely avoid (all) these rogue pyxform/xforms additions now!