Remove online form cache #293

eyelidlessness · 2021-07-31T00:39:17Z

This change addresses problems introduced in #269:

The previous last-saved implementation introduced caching of forms in online mode, because the last-saved instance was stored along with the Survey. That behavior has been removed, instead storing last-saved records in a new lastSavedRecords store.
Fixes Form's default media references are removed after submission enketo#1063, Image labels are not rendered for new forms enketo#1062 regressions, ensuring binary defaults and media labels are once again resolved correctly (except for media labels with spaces in the jr: URL, which will be addressed separately to resolve Media files with spaces in their filename do not load enketo#1101)

eyelidlessness · 2021-08-02T17:44:41Z

public/js/src/enketo-webform.js

@@ -200,6 +200,7 @@ function _init( formParts ) {
        modelStr: formParts.model,
        instanceStr: _prepareInstance( formParts.model, settings.defaults ),
        external: formParts.externalData,
+        survey: formParts,


While still untested, the changes in this file restore almost exactly the original behavior prior to #269, the only difference is store.init does still need to be called to support last-saved functionality. The store is otherwise not in use in this code path, but it's worth calling out.

eyelidlessness · 2021-08-02T17:57:00Z

public/js/src/module/connection.js

+const uploadQueuedRecord = _uploadRecord;
+
+const uploadRecord = ( survey, record ) => (
+    setLastSavedRecord( survey, record )
+        .then( () => _uploadRecord( record ) )
+);
+


Separating these functions wasn't a strictly necessary change, but uploadRecord's signature has changed to require a survey to support last-saved functionality while online, and it felt like this separation was better than requiring that for offline mode's upload queue where last-saved should not be used anyway. An alternative approach would have been to call setLastSavedRecord in controller-webform.js, but that would have required backfilling a much more elaborate set of tests for that module.

eyelidlessness · 2021-08-02T17:58:31Z

public/js/src/module/connection.js

@@ -358,17 +365,11 @@ function getFormParts( props ) {
    /** @type {Survey} */
    let survey;

-    /** @type {XMLDocument} */


The changes below remove last-saved implementation details, which are now implemented in the new last-saved.js module instead.

eyelidlessness · 2021-08-02T18:02:51Z

public/js/src/module/controller-webform.js

@@ -14,6 +14,7 @@ import records from './records-queue';
 import $ from 'jquery';
 import encryptor from './encryptor';
 import formCache from './form-cache';
+import { getLastSavedRecord, populateLastSavedInstances } from './last-saved';


I tried to keep changes here as minimal as possible, as this module is still untested. The changes here are to support:

The signature change to uploadRecord

Continuing to ensure last-saved instances are populated when the form is reset after submitting/queueing/saving a draft record

eyelidlessness · 2021-08-02T18:04:09Z

public/js/src/module/form-cache.js

@@ -7,7 +7,10 @@ import events from './event';
 import settings from './settings';
 import connection from './connection';
 import assign from 'lodash/assign';
-import encryptor from './encryptor';


As with many of the changes in connection.js, the changes here remove last-saved implementation details, which are now implemented in last-saved.js.

eyelidlessness · 2021-08-02T18:10:44Z

public/js/src/module/last-saved.js

@@ -0,0 +1,152 @@
+import encryptor from './encryptor';


All of the last-saved implementation now lives here. These changes should mostly look familiar, as it's very similar to the previous implementation. The primary differences are:

Last-saved records are no longer stored in surveys, they're written to a new lastSavedRecords store

Some code style changes:

Named exports declared at the definition site; this improves the ability to follow references through the project, as well as makes it clearer at definition site whether a function is intended to be used outside the module

Arrow functions (they're just more idiomatic)

Not technically a change, but calling out that if a survey is already cached for offline mode, online mode will still update that cache's last-saved externalData instances when a survey changes. This ensures caches do not store stale last-saved data, particularly in cases where a last-saved instance may have been removed from the form.

eyelidlessness · 2021-08-02T18:11:34Z

public/js/src/module/records-queue.js

@@ -12,6 +12,7 @@ import exporter from './exporter';
 import { t } from './translator';
 import $ from 'jquery';
 import formCache from './form-cache';
+import { setLastSavedRecord } from './last-saved';


The changes here account for the API changes in connection.js.

eyelidlessness · 2021-08-02T18:13:24Z

public/js/src/module/store.js

@@ -945,5 +963,6 @@ export default {
    dynamicData: dataStore,
    record: recordStore,
    flush,
-    dump
+    dump,
+    get lastSavedRecords() { return server.lastSavedRecords; },


I tried to keep the changes to store.js to a minimum, keeping as much last-saved logic in last-saved.js as possible. These minor changes were necessary because the store's schema all has to be defined at initialization time. This breaks a bit with the pattern for other stores, as last-saved.js references the db.js interfaces directly rather than a wrapper around them.

eyelidlessness · 2021-08-02T18:19:52Z

test/client/config/karma.conf.js

+                        ) )
+                    )
+                ),
+            ],


This change, as well as the one in build.js, addresses another media regression (and potentially several other regressions 😬), this time caused by #289:

enketo-core defines and implements a set of aliased modules whose functionality it depends on

enketo-express also defines those aliases, but overrides those implementations.

When we switched to esbuild, it failed to account for those overrides, causing the enketo-express build to use enketo-core's behavior for those modules.

We should consider consolidating relevant portions of these configs, so the two builds don't diverge in the future.

eyelidlessness · 2021-08-02T18:21:33Z

test/client/form-cache.spec.js

-                } )
-                .then( done, done );
-        } );
-    } );


These tests are no longer relevant, because none of the modes access the form cache anymore.

MartijnR · 2021-08-13T18:44:00Z

Hi @eyelidlessness,

I had a quick glance and especially at the PR comments you made. Looks good. There is just a merge conflict atm.

I'd like to do some real-life testing in the next few days.

…ine mode

I committed these by mistake in a rush to get the draft PR open

Also brings in the remaining relevant update test from enketo#291

This resolves issues where media was being removed from a form when saving a record in offline mode

- Don't error when submitting an edit - Don't set/update a last-saved record when submitting an edit

eyelidlessness · 2021-08-17T17:45:17Z

@MartijnR sorry it took a bit to resolve conflicts. Rebase got pretty confusing 😬 but should be good to go now!

MartijnR

Thanks. Code looks good to me and I did some real-life testing with regular forms and forms that use the last-saved feature. Online-only views no longer cache the form itself. Database structure update seems to work as desired.

MartijnR · 2021-08-18T18:35:21Z

Since some other related issues need fixing before releasing a new version, I will just go ahead and merge.

MartijnR · 2021-08-18T18:35:49Z

In a short while if there are no objections raised....

lognaturel · 2021-08-18T19:48:55Z

We’ve got some quality assurance planned in the next few days but if you’re satisfied with your checks a release would be most appreciated!

MartijnR · 2021-08-18T20:20:35Z

Oh, that would be great. Now I am tempted to just wait... :)

lognaturel · 2021-08-20T20:36:30Z

That sounds fine to me! @eyelidlessness would waiting a few days make more conflicts accumulate? @getodk/testers are back from vacation on Tues of next week.

eyelidlessness · 2021-08-20T21:16:28Z

I hope not but I don't mind dealing with them if they do, if holding off is preferred!

eyelidlessness commented Aug 2, 2021

View reviewed changes

eyelidlessness marked this pull request as ready for review August 2, 2021 18:24

eyelidlessness requested a review from lognaturel August 2, 2021 18:24

eyelidlessness force-pushed the fix/remove-online-form-cache branch from 4c8cc15 to a367b71 Compare August 2, 2021 22:18

This was referenced Aug 3, 2021

Fix: resolve jr: URLs with spaces #291

Merged

Error displayed when submitting an edit enketo/enketo#1060

Closed

eyelidlessness added 6 commits August 17, 2021 10:34

Roll back changes in enketo-webform.js that introduced caching in onl…

62943a2

…ine mode

Remove online form cache

577df16

Remove testimages directly

73ec244

I committed these by mistake in a rush to get the draft PR open

Fix form cache update

dc52e78

Also brings in the remaining relevant update test from enketo#291

Resolve last-saved instances at runtime rather than storing in the cache

92f09ef

This resolves issues where media was being removed from a form when saving a record in offline mode

Fix edit mode submission

65a1080

- Don't error when submitting an edit - Don't set/update a last-saved record when submitting an edit

eyelidlessness force-pushed the fix/remove-online-form-cache branch from 20cd8ab to f09f101 Compare August 17, 2021 17:42

MartijnR self-requested a review August 18, 2021 18:31

MartijnR approved these changes Aug 18, 2021

View reviewed changes

eyelidlessness mentioned this pull request Aug 18, 2021

Backfilling tests of enketo-webform.js #304

Closed

9 tasks

Remove now-unused _updateFormData

d387d45

yanokwa force-pushed the fix/remove-online-form-cache branch from f09f101 to d387d45 Compare August 27, 2021 22:44

yanokwa self-requested a review September 10, 2021 17:26

yanokwa merged commit 8ef3b98 into enketo:master Sep 10, 2021

eyelidlessness mentioned this pull request Sep 10, 2021

Prepare for 2.8.1 #311

Merged

eyelidlessness mentioned this pull request Sep 21, 2021

Fix: rendering of binary defaults in online-only mode #316

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove online form cache #293

Remove online form cache #293

eyelidlessness commented Jul 31, 2021 •

edited

Loading

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

eyelidlessness Aug 2, 2021

MartijnR commented Aug 13, 2021

eyelidlessness commented Aug 17, 2021

MartijnR left a comment •

edited

Loading

MartijnR commented Aug 18, 2021

MartijnR commented Aug 18, 2021

lognaturel commented Aug 18, 2021

MartijnR commented Aug 18, 2021

lognaturel commented Aug 20, 2021

eyelidlessness commented Aug 20, 2021

Remove online form cache #293

Remove online form cache #293

Conversation

eyelidlessness commented Jul 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartijnR commented Aug 13, 2021

eyelidlessness commented Aug 17, 2021

MartijnR left a comment • edited Loading

Choose a reason for hiding this comment

MartijnR commented Aug 18, 2021

MartijnR commented Aug 18, 2021

lognaturel commented Aug 18, 2021

MartijnR commented Aug 18, 2021

lognaturel commented Aug 20, 2021

eyelidlessness commented Aug 20, 2021

eyelidlessness commented Jul 31, 2021 •

edited

Loading

MartijnR left a comment •

edited

Loading