Custom config fields/"form metadata"

[smoyte]

The new "Form Metadata" activity described in #311 is awesome, and even more awesome would be the ability to define custom fields there so that data could be entered once at the app level and then included in all submissions. Examples could be a team name, region code, etc. This would prevent enumerators from having to fill out the same data over and over, thus reducing errors.

Things defined in this way would be accessible in forms via preloading, e.g. for a custom field with key teamname:

<bind nodeset="/widgets/teamname" type="string" jr:preload="property" jr:preloadParams="teamname"/>

Additional notes:

• Keys should be forced to be unique
• Custom fields should be included in existing "bulk configuration" functionality

[lognaturel]

@MartijnR I bet you would say external secondary instances would be a great way to address this...

[MartijnR]

Yes, absolutely. :) It's a perfect use case for it and the value can be accessed with instance('session')/path/to/node. The cool thing is that these external XML instances could be dynamic and vary per project or user (if the server supports that).

[smoyte]

I read the linked docs but I don't quite get the idea here.

Are you just suggesting this as a way to implement the custom metadata feature? i.e. as the user defines the metadata we would store it in a special external instance that is local to the phone? If so, I'll trust that that makes sense as I don't have much experience with JR internals.

We definitely want the ability to edit these values in the ODK settings. We already have the ability to make such things editable on the ELMO server at form design time just by using an internal instance on the rendered form. But then the value is frozen once the form is downloaded. That may work for some cases but not all.

[lognaturel]

Yes, @hooverlunch, some of us talking about the ODK XForms spec here have been talking about deprecating preloads since they're kind of a hack. So I'm not sure that'd be the way to go.

@MartijnR, there's nothing that would prevent the XML file from being something that's just local and generated when the phone's owner writes some settings, right? Does one of the specs (ODK XForms or XPath) specify what happens if either the external instance or a specific node is missing? I'd want to make sure that we can provide useful feedback.

[lognaturel]

@hooverlunch It's a little fuzzy in my mind, too, which is why I'm pulling in @MartijnR who already supports external XML instances in Enketo and understands XPath a bit lot better.

What I'm imagining is that there'd be some standard name for the file generated from user settings (custom-user-metadata.xml or something, in fact that could also be a setting to reduce the magic or perhaps it could match what CommCare does with sessions which might be what @MartijnR was thinking). You'd have the option to directly write that file to the phone or have each user go to the settings section to write out their own values (or both -- load some defaults and have the user fill out the rest). Then in your form you'd just write XPath expressions to refer to the different nodes in that XML file -- instance('mySweetCustomUserData')/user/teamname.

That's nicely extensible and also standards-compliant.

[smoyte]

I get it. Then we'd use calculate to load the value in instead of jr:preload?

[MartijnR]

yes, this could be done in conjunction with revamping existing preload items, or be a separate feature (a separate XML document).

@MartijnR, there's nothing that would prevent the XML file from being something that's just local and generated when the phone's owner writes some settings, right?

No, but I think the existing spec(s) cater(s) to either an internal secondary instance (without src) or to an external instance that is published in the xFormsManifest. To support 'local external' src attribute values we should probably consider adding a new URI scheme. Edit: I don't know, maybe not. We can perhaps do it in the connector part that comes after the jr:\\ scheme which is what CommCare is doing with sessions.

Does one of the specs (ODK XForms or XPath) specify what happens if either the external instance or a specific node is missing? I'd want to make sure that we can provide useful feedback.

If a stringValue is requested for a non-existing /path/to/node this should return '' (and NOT crash!), I believe.

[lognaturel]

I'm really liking this general approach. Of course, the challenge for getting this into Collect is that no one really seems to know what the status of external secondary instance support in JR is. See this for fun times.

I would be in favor of doing this in conjunction with revamping the existing preload functionality and including these custom fields in the same artificial instance. We could match roughly what CommCare does. In fact, I think this is the kind of data they put in session/user/data if I'm understanding correctly. @hooverlunch what do you think?

@MartijnR are you interested in doing this in Enketo eventually? Should we move this conversation to the ODK XForms spec repo?

Good to know that there's no crash when referring to a non-existing node. There will probably have to be some thought given to how these values could be marked as required and how that would be indicated to the user. That might also require a spec addition but maybe it can be done with existing constructs, I'm not sure.

[smoyte]

This is all a bit over my head at this point but it sounds good to me. I'm all for refactoring/revamping as we go, within reason.

[MartijnR]

I think whether it belongs with preload items (and thus a possible future session instance) or as a separate secondary instance, may depend on where these custom values are defined, either in the app or at the server.

In the latter case, it's already supported according to our current XForms spec. All the work is done by the OpenRosa server by creating the XML document, and (by server or form builder) dynamically creating/updating the XForms definition. This seems to be the approach to would fit well with servers that have complicated user/project/organization roles (such as Ona/KoboToolbox).

In the former case, I'm not sure if it will make sense for a web application, such as Enketo, to support it, but nevertheless would be good to move it to the XForms Spec repo. It will make sense for other mobile clients. (I imagine you have a technique to centrally configure multiple ODK Collect devices?)

[smoyte]

I think this sounds good to me. Our use case is the former one since, as I mentioned earlier, the client wants to be able to enter things like a team ID and/or other things at the phone level, once, and have them copied into all forms.

I should say though that I'm considering just kind of hacking it by using one of the new form metadata fields like email or username for this purpose as it would be simpler. So for now, this is not as high priority for me as #452...

[smoyte]

@lognaturel @MartijnR Curious what the status of this one is. Existing preload stuff has been revamped. What would need to happen to get custom metadata support? How can I help?

[lognaturel]

Fill out the survey on ODK management structure and volunteer to be on the technical steering committee? 😁

I think this has to be really carefully considered in the context of what ODK's story for loading data external to forms should be. There already are a number of ways to load external data through CSVs with more interesting proposals for extensions or workflows out there (see here).

I think a good next step would be to describe the scenario/feature on the forum so we can look at it alongside the others. It would be helpful to get a clear sense of if/why these values need to be updated on the devices themselves. Could config files be pushed instead? Would enumerators generally need to modify these values? Would something like a concept of a "form template" that is partially filled help? How about something like this?

[smoyte]

Fill out the survey on ODK management structure and volunteer to be on the technical steering committee? 😁

Already done!

I think a good next step would be to describe the scenario/feature on the forum so we can look at it alongside the others.

Will do.

[smoyte]

Here it is: https://forum.opendatakit.org/t/custom-form-metadata/9496