Prefill with last-entered value

[smoyte]

Hello!

We are designing for cases where enumerators are going from house to house and want to copy forward the same cluster number and/or keep an incrementing tally of houses visited (assigning a number to each one).

I propose we could handle both of these needs by giving the ability to pre-fill a question with the value for the same question on the most recently saved (but not necessarily finalized) form.

I'm proposing this be defined by jr:preload="previnstance" and optionally jr:preloadParams="increment".

increment would always just attempt to increment the value by one on int questions only. Blank would not be incremented, just copied forward as blank.

Either of these params on a non int question would be ignored.

Thoughts?

[lognaturel]

This is related to and perhaps a duplicate of #389. What do you think, @hooverlunch, @ratikanta-sdrc, are you trying to achieve the same thing?

[lognaturel]

Hey, actually, I think this could also be achieved with full external secondary instance support, right @MartijnR? I think there might have to be something additional to identify the last instance for the current form.

[MartijnR]

In this case it could be internal as well perhaps (since it may not be submitted), i.e. the client dynamically adds the last instance to the XForm and it can be accessed by instance("__last")/path/to/node e.g. I'm assuming this is really something for a custom fork, right?

[lognaturel]

Thanks, @MartijnR! Does dynamically adding the last instance to the XForm mean literally writing all of the contents of one XML file (the one for the last instance) to another (the latest instance)?

It actually sounds like a fair number of people want something related to accessing values from previous form instances. I think it's somewhat related to the concept of case management?

@hooverlunch now that I think about it more deeply, I think there are some problems with trying to use the last instance on the phone to determine numbering. What if I fill out a form for house 5, finalize and send it, delete it and then fill out another form?

[smoyte]

@MartijnR definitely no custom fork! We are on the band wagon and want to stay there :)

@lognaturel Good point.

Perhaps we keep a simple, local cache for each pair (form, question) where question is a question that has jr:preload="previnstance" defined on it. Each time a new instance is filled out, we overwrite that value in the cache.

How that gets stored and injected into the instance on the current form is up to you ODK nerds. Thoughts?

But the fact is that if we don't support this, folks will end up writing these values on their hand or a napkin or something, which is a bummer. We should be able to support something like this as it's a real need. The data is passing through our app and people need it. Let's help them out!

[lognaturel]

Let's take a step back. You mentioned two cases -- copying forward a cluster number and keeping a tally. For the cluster number, how often does this value change? Could something like what you're describing in #443 be used?

For the tally, how would it be used? Does the enumerator actually need to know the count? Is being able to see how many forms were sent sufficient for example? Is it more that they need a unique identifier?

[MartijnR]

Hmm. Maybe good to move this to the spec repo. I find accessing the last saved instance a rather weird feature.

The way I understand case management is that it allows tracking an entity (e.g. over time) for longitudinal surveys. This would be a awesome to support, and I think it would require the ability to download previously submitted instances for that entity and form.

edit: our messages crossed

[MartijnR]

Actually, yes it does seem caseDb is for locally stored data, so I guess you could create an XPath query for last locally stored item. Maybe it's a synced database. So yes, seems totally right @lognaturel.

[smoyte]

@lognaturel here is some more detail on the scenario:

1. The cluster number is roughly analogous to a village. They don't want to enter it in the metadata screen because then folks will forget to change it when they move to the next village. They want folks to see and manually confirm the previously entered value (by swiping to the next question) each time they fill out a form.
2. The household number increments each time, and resets when you go to a new cluster. So a total count of the forms sent would not work as it wouldn't handle the reset. Instead we would prefill the previous value and they would see it and say "Oh, we're not on household 15 because we just changed clusters. I'll change it to 1". And then for the rest of the cluster the auto incremented value would be correct.

More generally speaking, I understand that instances are supposed to stand on their own, etc. Perhaps why you are thinking this is "weird" Martijn.

But I am not intending this as a fundamental change to the ODK data model assumptions. Instead, think of it as a simple, relatively dumb convenience feature implemented as a layer on top of the ODK core. It's local to the current device and it's analogous to keeping some jot notes in the margin, which, again, is what folks will end up doing, and that seems silly/weird to me.

The question about unique ID is one I've asked them several times. I proposed just using a random value but they like having things in sequence, and I get it. The enumerators go in sequence from house to house and sometimes refer back to the forms by remembering "Oh, that was the second house in that village", etc. You lose that if you do everything randomly generated.

The folks we're building this for previously went out and had a student team build a custom app for them which they ultimately ended up abandoning. They're now moving to ODK/ELMO which is much more sensible. But one of the things they liked about it vs ODK is that it was able to keep this simple count for them. And I'm still not seeing why we can't do this here...

Thanks for the good discussion!

[MartijnR]

@hooverlunch Apologies for using 'weird' to describe it. Your use case is obviously valid (because it exists). Thanks for elaborating on it. I meant, if it were suggested as an Enketo feature or XForms spec addition, it would seem at first glance to be too domain-specific to include in the 'master' app/spec and I'd suggest forking. Implementing it in a fork just by attaching the last locally stored submission instance to the model, would be very clean and not create any merge headaches (in Enketo). And then use a calculate to retrieve values from it.

[smoyte]

@MartijnR thanks for the clarification. Just so I'm clear, you're saying 1) Enketo is a fork of the main ODK Collect, 2) you wouldn't want the pre-fill-with-last feature but 3) maybe I can convince the ODK Collect folks to add it and then 4) you could just not merge it in? That sounds fine to me.

So then the way we'd implement it is some kind of special external instance that is maintained by ODK Collect and that stores the last-entered instance for each form (perhaps whether the form has been deleted or not, see above). Then we could get at that by using calculate without needing to change the XForm spec.

Am I reading you?

So then @lognaturel how do we find out whether external instance support actually exists in ODK Collect? I'm on the edge of my seat here...

[lognaturel]

Oops, introductions may be in order. 😊 @MartijnR works on Enketo which is an ODK XForms-compatible web-based form engine. Martijn, please let me know if my wording is not sufficiently precise. He has been doing a lot of work with the ODK XForms spec and we've been having lots of discussions about how to make sure it helps us have a vibrant, standards-compliant ecosystem.

@hooverlunch works on ELMO which is an ODK-compliant backend (also let me know if I didn't phrase that properly).

[MartijnR]

Thanks @lognaturel :) I had just looked up Elmo! Nice!

Yes, so there are 3 100% unique data collection clients that follow the same XForms spec (of which Enketo is one), so we try to agree on spec additions and stay compatible. Nevertheless, our apps will of course have forks to meet organization-specific additional requirements that don't get accepted in the mother apps.

In this case, I don't want to discourage you from checking (maybe by creating an ODK XForms Spec issue) whether this is a use case that is widely (enough) required and is not considered too specific. We can pull in KoBoToolbox and Ona folks as well. The spec addition would be (just) for a new URI endpoint I think, e.g. jr://local-instances/last or something.

[lognaturel]

I think the big questions here are:

• can this be done with existing functionality
• if not, is it general enough to warrant changes to the ODK XForms spec and/or core ODK Collect

I think that the answer to the first question is unfortunately not really. But @yanokwa heard me muttering about this and made a very interesting suggestion which is to use an external app to get the desired behavior. That app could do interesting things like keep track of separate counters for each cluster number, allow the user to reset counters and things like that. I think that might be your best bet in the short term. It should be a very lightweight and quick app to write. Perhaps we should have a repo with examples of little apps like that.

What you described in your latest message @hooverlunch would still require changes to Collect. That's where the second question then comes in -- is it general enough to bring into trunk?

I think what you want in the short-term does seem pretty specific to your users although it would be interesting to hear more about @ratikanta-sdrc's use case to see if there's overlap. I think similar functionality would be enabled if we were to add something like case management. That's an idea we will want to explore but it's a much bigger conversation.

I believe what @MartijnR was saying regarding a fork is that it doesn't have to be a big thing to have a fork. If the changes are fairly isolated, you can keep pulling in changes from trunk and maintain your custom functionality until the community can come to an agreement on how to evolve standards or express the feature generally enough to add it to trunk.

[lognaturel]

@MartijnR Pardon my ignorance, what's the 3rd compatible engine?!

[MartijnR]

@lognaturel It is the current Survey123 mobile app :)

P.S. external app seems like a great idea too!

[lognaturel]

@MartijnR Ohhh, right, very cool. And of course, there are a bunch of form designers and other tools that also are ODK XForms compliant so it's a very important spec to keep clean and well-considered.

@hooverlunch part of the history here is that there've been things kind of duct taped to the spec and to Collect in the past and we've ended up paying a steep price for maintenance, reliability and/or extensibility. We are trying to be much more deliberate about thinking each feature through.

[smoyte]

FYI we've decided to go with the external app idea here. I got the breath counter thing working pretty easily and it's a good template to work from. Thanks for the idea!! I'll be sure to share it once it's done.

So I'd be fine with closing this issue out if you like.

[lognaturel]

Fantastic! I'm looking forward to seeing it and don't hesitate to ask if you have any questions on how to approach it.