Remembering previously entered values in Collect

[cooperka]

General goal

Give forms the ability to recall previously entered values when filling out the same base form multiple times on the same device. This behavior would be opt-in only.

Keywords: recall, autofill, autocomplete, preload, memory, template

Motivation

• As a census taker, I write the current street name on a napkin so I can remember it for each house I visit.

• As a natural disaster responder, I enter the exact same earthquake date every time I interview someone from the same area.

• As a forest surveyor, I type in my team name and choose a region code every time I log a new tree. Sometimes I make typos.

Resources

There have been numerous discussions in the past; I've tried to summarize and consolidate the conclusions here in the creation of this proposal. The "motivation" above is also inspired directly from these discussions.

• Remembering previously entered value in ODK Collect (the discussion prompting this proposal)
• View last submitted data at the time of filling up the form
• Prefill with last-entered value
• Automatic default selection
• Persistent Data on the Field
• Copy and edit forms (proposal for form duplication to address a similar need)
• Custom config fields/"form metadata" (similar need)
• Custom form metadata (similar need)

"Case Management" is related, but separate and much more complex. Linked here because there may be some overlap in ideas:

• Enable Case Management/Preloading
• Longitudinal data management

Proposed implementation

The cache is a simple SQLite or Shared Preferences key/value store that maps from the [form ID + question ref] to the [last saved value for that question].

The cache is updated whenever a form is saved. Any question in the form that is intended to be cached will be re-cached on save, whether or not that specific question has been edited since the last save.

When filling out a new form instance, the question value is loaded and set from the cache at a specific point in time, depending on which option is being considered (see options below).

The cache would be cleared if you clear app data via system settings, but not if you merely delete the form instance from your device.

To designate a question as being cached, there are multiple options:

Option A

XForms bind attribute: e.g. jr:preload="last-saved"

• Automatically loaded and set from cache on xforms-ready if no existing value
• See existing jr:preload options here
• Pros:
  • Works on all question types that can stringify their data
• Cons:
  • Requires a modification of the JavaRosa spec (though this may be easier than a modification to the XForms spec)

Option B

XPath function: e.g. lastSaved()

• Loaded and set from cache every time something changes, just like any other calculation
  • Will always be the most recently saved value
  • Combine with existing once() calculation for one-time updates
• Pros:
  • Works on all question types that can stringify their data
  • Allows things like auto-increment fields with e.g. once(lastSaved() + 1)
• Cons:
  • Users may not be comfortable understanding how this works, and may not know to use once()
  • Requires a modification to the XForms spec, which could affect other apps that implement the spec

Option C

XForms appearance: e.g. appearance="save-recent"

• Dropdown menu for single-line string entry fields
  • Appears when you focus the text input
  • Can be filtered by typing text (e.g. typing a will still include options for ant and cat but will filter out dog)
  • Options are always sorted by most recently used
  • Perhaps there can be an x next to each item to clear it from memory
• Pros:
  • Ability to choose from multiple previous choices rather than just 1 previous
• Cons:
  • Only applies to single-line text entry fields; would be hard to imagine a UI for this applying to date pickers or radio buttons, for example

Option D

setvalue action

• See Remembering previously entered values in Collect #30 (comment) below

Other options

Options that have previously been discussed but are insufficient to fully address these needs:

• Launching external apps to populate single fields
• Pre-loading CSV data
• Using external XML data
• The ability to mark an instance as a "template"
• The ability to duplicate instances

Option combinations

Options A/B/C are not mutually exclusive, but for the sake of simplicity I think we should keep the initial feature as minimal as possible.

I'm personally in favor of option A alone. It's simple, flexible, and hard to make mistakes when designing forms that utilize it (e.g. forgetting to use once() in option B could lead to major frustrations).

Names

Names I've considered for options A and B:

• lastSaved
• lastValue
• cached
• prevInstance

I personally like lastSaved best because it's clear and specific: assign the value from the form that was most recently saved (as opposed to any other value, such as last viewed, last changed, or last finalized).

Names I've considered for option C:

• save-recent
• show-recent
• recently-used
• autocomplete
• last-saved

I personally like save-recent as an appearance because it implies the two-way nature: answers are saved and shown in the future. Setting this option both enables caching and shows the recent values to the user.

Final thoughts

Again, I'm personally in favor of only implementing option A for now, but options B and C are also possibilities, and are not mutually exclusive. I'd love to hear your feedback.

[tiritea]

Couple quick initial comments (while I chew it over... great stuff BTW!):

"...filling out the same base form multiple times on the same device".

• I'd probably highlight the fact this is all about a particular device. This distinguishes/isolates this particular usecase to probably a single enumerator, and probably during a single 'assignment'. Which contrast with, say, prefilling instance data via form-specified defaults in the XForms XML instance data, or fetching previously submitted form results and using that to prefill a form (something we do with re-inspections). Some of the options you mention may or may not be applicable to the other types of "prefilling data" usecases, so it might help to eliminate them from current consideration (or not?).

"The cache would be cleared if you clear app data via system settings, but not if you merely delete the form instance from your device."

Given this is effectively a cache for a particular form on a particular device, it would seem to make sense that the cache should be flushed when said form is removed from the device. Not deleting a form's cache when a form is wiped off an device doesn't seem advantageous, and might well just make life complicated later... why not delete everything associated with the form, cache and all?

[tiritea]

• As a census taker, I write the current street name on a napkin so I can remember it for each house I visit.
• As a natural disaster responder, I enter the exact same earthquake date every time I interview someone from the same area.
• As a forest surveyor, I type in my team name and choose a region code every time I log a new tree. Sometimes I make typos.

These specific usecases suggest that the individual enumerator is deciding they want to reuse certain (arbitrary?) fields in a form, whilst out in the field; that is, they simply want to add (or override) default literal values specified in the form's instance XML; ie the form defaults that the form writer explicitly added when creating the form.

Rather than having to introduce new features for the form definition language itself (new XForm binding attributes, new XForm appearances, new XPath functions, ...) it would seem this could be more easily accomplished, and more applicable generally, by simply initially populating the form with a previously user-defined instance XML template, rather than the instance XML from the form definition. This could be entirely accomplished (optionally) on the XForm client (ODK Collect or otherwise), with no modifications necessary to the ODK XForm spec itself nor any existing forms already deployed out there. By adding a "Save as Template" button which would save the current instance XML contents of the form you are filling onto the device, and correspondingly a "Fill Template Form" button as an alternative to "Fill Blank Form", which would initialize the new form submission from the template instance XML instead of the form instnace XML. Whether we overwrite a single template per form, or allow users to save multiple templates under different names, can be a client-specific implementation detail; again, nothing needs to change in the spec.

At any time whilst out in the field, an enumerator could choose to save what they got as template to re-use again subsequently; eg start a form and fill in just the street (or earthquake date, or team name and region), save it, and reuse this to initialize new forms if they want to, or start a clean form from scratch if they dont. Or they are in the middle of filling in a household survey and suddenly decide they going to need to re-enter a bunch of data for everyone present, so save the partiallt filled form for first household member and use it as a starting point for everybody else. Again, the determination is entirely made by the enumerator, whilst out in the field, as immediate circumstances dictate; no prediction needs to be made by the form writer as to what can/not be reused (ie cached), it can be applied to any existing forms out there, and how or even if its supported (eg 1 or many templates, or none) is entirely client dependent.

FYI I do something very similar wrt re-inspections in our app; basically, basically the inspector determines whether this is a new inspection - and initialize with form instance XML if so - or a re-inspection, in which case the form is initialized with last submitted instance XML [our problem is being able to specify what questions to automatically clear on a re-inspections, eg signatures. But this isnt an issue for user-defined templates].

I would suggest that upon the enumerator deleting the form from the device, that any associates tempates are automatically deleted also (ie clear the cache).

This still leaves open the question of other usecases and potential mechanisms for prepopulating form date from outside sources (eg preload CSV's, extract default properties from accompanying XML files, etc) but at least for the usecase described - "filling out the same base form multiple times on the same device" - it may possible to accomplish this is in a fairly straight-forward way by initializing forms from a user-saved template instance XML data, rather than form instance XML data This could accommodate a wide variety of usecases without needing to change form definitions or add new features to the ODK spec; its merely a client-side implementation detail. At least for my re-inspections, this has turned out to be a very workable field solution that is both easy for our inspectors to understand and imposes no additional burden of our form creators.

[MartijnR]

it would seem this could be more easily accomplished, and more applicable generally, by simply initially populating the form with a previously user-defined instance XML template, rather than the instance XML from the form definition

This approach (purely client-side, no XForms syntax) seems the best to me. I think it is a very cool idea. Very logical, testable, no magic reference to a previous-record-on-same-device-for-same-form.

[cooperka]

Thanks for the feedback @tiritea and @MartijnR. I'm curious what you think of some past discussion about templates here. It was proposed and partially implemented, but ultimately abandoned.

I largely agree with @smoyth's concern in that same discussion:

I don't think many people would, on their own, think to create a template as a way of reusing a value for a single field (team name).

I think templates are a great idea in general, and they have their place, but in my experience they also tend to be messy, confusing, and frustrating when the user just wants a single field or two to recur based on the most recent value.

In the use cases here, the user would have to remember to do all of the following in order:

1. Open a blank form
2. Fill out a template with only the fields they want to be remembered
3. Make a copy of the template to be filled out (making sure never to accidentally fill out the template itself, because that would ruin it and they'd need to start over)
4. [Encounter something that causes the fields to change, such as a new street name]
5. Edit the template before starting the next real form
6. Go to step 3 and repeat

For comparison, with this proposal:

1. Form designer designates fields that are remembered
2. Forms are filled out as usual

Again, I think templates have their place, but I think they're too complex to fully meet the users' needs for these use cases.

[tiritea]

Thnx. Looks like there has been some good discussion around such 'templates' already... I've added a comment to that thread (rather than append here).

Overall, honestly, I'm still not convinced templates are too (more) complex; indeed most of these other approaches appear to be more complex to implement, and consequently could result in all sorts of unanticipated gotchas. Also, templates can be implemented today with no changes introduced to the ODK spec - so nothing we may regret wrt legacy support later - and it can be used for any forms in existance today. Why not implement templates, get some feedback from actual field use, and use that to identify what common usecases could be handled better, before actually committing to change the spec? But again, that's just my $0.02... I have no specific objections to the proposed Option A.

[yanokwa]

The heart of this feature is having a question populated with the last saved answer. You are essentially pre-filling the form and we do have a reasonable XForms mechanisms for doing that: secondary instances.

Now the mechanism doesn't have to be the UI! That is, we shouldn't have people make templates because I agree with @smoyth and @cooperka that it's way too complicated. I had a quick chat with @lognaturel about earlier conversations she's had with @MartijnR and here's the rough approach we came up with.

For questions that want the previously saved answer, use a setvalue that fires on xforms-ready. For the value, you can use XPath expressions and some agreed upon syntax to refer to the last filled instance of that form id (e.g., instance(last-finalized)/data/teamname).

That syntax will be a hint to the client to load that last instance as a secondary instance and grab the value. For bonus points, we can parameterize the syntax so we can get the last n values. This would be helpful as a magically historical auto-complete for text widgets (kinda like Option C). I think we have most of the pieces here to get this to work and the only new thing is the syntax for getting the last filled instance.

P.S. I don't think we should use preloads because those should probably be deprecated since we have actions

P.P.S. I think we shouldn't implement templates first because the UI changes to Collect are going to be disruptive and hard to roll back.

[tiritea]

Thank you for the detailed response. I guess I'm just loath to introduce anything new to the ODK XForms spec definition until we've exhausted and fully explored all other options, because it becomes (a) something that we can end up being forced to live with (ie legacy support of arguably 'mistakes', and AFAIK nothing has been officially deprecated yet), and (b) spec changes mean [quite selfishly] more work for other tooling attempting to be compatible with the ODK stack. :-)

[of course, I say that somewhat hypocritically, what with an aborted attempt to introduce an unnecessary numberOrZero() XPath function recently... ;-) ]

[MartijnR]

Apologies for not reading the template discussion in the forum thread.

Though a template solution is still technically far more attractive, I can see they may be too cumbersome for users of mobile clients (It’s different for a web client such as Enketo which can - and does - do this centrally by dynamically populate defaults via the URL).

For questions that want the previously saved answer, use a setvalue that fires on xforms-ready. For the value, you can use XPath expressions and some agreed upon syntax to refer to the last filled instance of that form id (e.g., instance(last-finalized)/data/teamname).

This setvalue approach with a secondary instance may be best the way to do this without a template (also worth looking into virtual endpoints like CommCare instead of instance(…) but that’s essentially the same approach). Clients that don't support the feature would just return an empty '' response because the (virtual) node doesn't exist.

[cooperka]

So it sounds like the way forward here is with Yaw's setvalue action with an XPath expression.

I think e.g. instance(last-saved)/data/foo makes sense for the expression (note the name change from last-finalized to last-saved here). It can optionally be parameterized as suggested, e.g. instance(last-saved)[2]/data/foo, but that may be a feature for later depending on how much complexity it adds.

@yanokwa are you okay with me moving forward with a PR? Any other concerns?

[yanokwa]

@cooperka I'm OK with you moving forward with this general approach, but it'd be good to get @ggalmazor and @dcbriccetti to chime in since they've been recently in that code.

@MartijnR I know nothing about virtual endpoints, but I suppose that doesn't prevent me from having an opinon! It looks like Dimagi also calls them artificial instances and they are designed to "load data from the application layer. This data must be served up as an abstract XML data structure which is handled and processed like a real XML document." In our case, we have a regular old XML document, so I think JR can use regular old binary endpoints. Agreed?

[ggalmazor]

Hi all!. Apologies for stalling the conversation for the last week!

It looks like the latest agreement is that we would be preloading the last saved value using setvalue and xforms-ready, which sounds fine to me.

Let me roughly describe what could be the simplest way to go about this with JR/Collect, which could probably translate to other platforms as well:

• Collect gives JR the path to the last saved submission's XML file when preparing for a new submission.
• JR parses it under a virtual (as in "it isn't declared in the blank form") last-saved secondary instance.
• If Collect doesn't provide the path to the last saved submission (maybe it's the first submission being filled), then JR uses a blank submission instead.

This mechanism would involve making changes to the code that handles the instantiation of new submissions only, which feels like a good balance between cost and feature value, with no important foreseeable tradeoffs.

I do think that we should provide some syntactic sugar for our users that would let them set this up more easily through any method that feels suitable, but we can think about this later.

On the other hand, remembering all the values saved on each field is a whole other kind of beast. Honestly, I can't suggest a good way to express this at the specs level or a smooth way to implement this within JR that doesn't feel like overkill. At this point, I would be more inclined to solve this externally with some client-side cache as @cooperka suggested.

In any case, I think that these are two different and complementary use cases: preloading the last saved value for a field vs. virtually making all the controls auto-completable with any previously used value for that field.

I think the latter would require much more discussion, starting from how this would actually affect the UI, and I'd suggest we punt on that while we can deliver the former and the users have a chance to play with it.

[aurdipas]

What if I've enabled the "autosend" and at the same time the "delete after send" on my collect?

[ggalmazor]

What if I've enabled the "autosend" and at the same time the "delete after send" on my collect?

Thanks for the question, @aurdipas!

Collect could still save a copy of the "last saved submission" somewhere safe.

[cooperka]

@ggalmazor thanks for the thorough example above. I've recently begun looking through the code to implement this, and I'd appreciate it if you could sanity-check my thinking so far (all I need is a nice big thumbs up 😸).

As best as I can tell, here's the series of steps taken starting from the user opening a blank form through to loading the formInstances used by that form (soon to include the new last-saved instance).

• FormChooserList#onItemClick (user opens a blank form)
• FormEntryActivity#loadFromIntent (called on new form)
• FormLoaderTask#doInBackground (start loading the form)
• FormLoaderTask#createFormDefFromCacheOrXml (returns FormDef with formInstances)

Case A: formdef is cached

• FormDefCache#readCache (foo.formdef => returns FormDef)

Case B: formdef not cached

• XFormUtils#getFormFromInputStream (foo.xml => returns FormDef)
• XFormParser#parseDoc (parse instance nodes)
• ExternalDataInstance#build (given an external src)
• FormDef#addNonMainInstance (every time it finds an internal/external <instance>)

What I want (i.e. your suggested implementation above) is very simple: add the last saved form instance to the formInstances list via FormDef#addNonMainInstance. This particular instance is referred to by the last-saved key (e.g. instance(last-saved)/data/foo in XML). Once that happens, everything else should work automatically.

Am I on the right path so far?

If so, it looks like I'll need to have the last saved instance (or null) in Collect's FormLoaderTask#createFormDefFromCacheOrXml and somehow pass that along to JavaRosa to be built along with any other external instances. This sounds like what you're suggesting. I'm going to keep forging ahead but I wanted to make sure I'm not totally off-course here.

[MartijnR]

In our case, we have a regular old XML document, so I think JR can use regular old binary endpoints. Agreed?

Seems like we can leave that open (some clients may use a relational db) as the URI is the same so something like this?

<instance id="myform">
    <data>
         <a/>
    </data>
</instance>
<instance id="__last-saved" src="jr://instance/last-saved"/>
...
<setvalue event="xforms-ready" ref="/data/a" value="instance('__last-saved')/data/a" />

I think the id attribute value is open and doesn't have to be defined in the XForms Spec (it is only referred to within the same form). May make sense for XLSForm to use something that reduces the chance of conflicts with user-added instances (e.g. by prepending double underscores). (And when we replace preloads, we could do the same for a jr://instance/session instance).

Something we may want to make sure is that xforms-ready doesn't fire when a draft record is loaded for editing (according to W3C). If so, we'd have to introduce another event that doesn't re-fire.

[ggalmazor]

@cooperka

If so, it looks like I'll need to have the last saved instance (or null) in Collect's FormLoaderTask#createFormDefFromCacheOrXml and somehow pass that along to JavaRosa to be built along with any other external instances. This sounds like what you're suggesting. I'm going to keep forging ahead but I wanted to make sure I'm not totally off-course here.

Everything you've said is looking good. Just let me note that if no last saved form is available to be provided to JR, JR should create a blank one and use it instead.

An optimization would be to search for any use of instance('last-saved') in the form to prevent having to parse it (or generate a blank one) in the first place.

I won't be able to be of much help in the Collect side, although I can review and try stuff as you progress (maybe you can create a WIP PR)

[ggalmazor]

@cooperka, have you thought about what @MartijnR's has commented? I'm taking some time to dig into that tomorrow and we can discuss it later ;)

[cooperka]

so something like this?

<instance id="whatever-you-want" src="jr://instance/last-saved"/>

That sounds great to me, the slightly modified syntax makes it more clear to the user where the data is coming from. It's less "magic" which is good.

Defaulting id to __last-saved when building forms seems reasonable, though that's separate from the changes I'll be making.

It's not too much of a difference in terms of implementation, so I'll move forward with the assumption that's how we'll format it, but the old way is just as easy if there are any objections.

[ggalmazor]

@cooperka, if I understand correctly, with @MartijnR's suggestions, Collect still has to provide the path to a last saved form, and then JR will use it only if the user declares an instance with a src="jr://instance/last-saved". Then, the user may refer to that instance using whatever id they choose.

If this is correct, I think it's an improvement because it's more explicit :)

[cooperka]

Exactly 👍 I have this working and will submit PRs soon.

[lognaturel]

Released in JavaRosa v2.14.0 and Collect v1.21.0.

Pending pyxform implementation: XLSForm/pyxform#337

[lognaturel]

Released in pyxform 1.0.0, XLSForm Online 2.0.0. Documentation: https://docs.opendatakit.org/form-logic/#values-from-the-last-saved-record