Time-Use tab (Sebastian)

[sebastianbarry]

Got the basic time use tab working!

Now that it is working at a basic level, Next steps are to make changes to the time use tab to get it working how we want it.

A meeting is scheduled next Monday for this conversation with @asiripanich and @shankari

---

Testing:

[sebastianbarry]

The fix was these needed to be the same

[shankari]

@lgharib @TTalex @jf87 is there interest from the rest of the community on a manually entered time-use survey? Would you be interested in co-designing it, or should we go with @asiripanich's thoughts and plans for now?

[asiripanich]

@shankari, it would be best to co-design this time-use tab with the community.

Here is what we were trying to achieve with the time-use code designed by @MaliheTabasi :

Screen.Recording.2022-07-19.at.17.24.40.1.mp4

[sebastianbarry]

I am getting hung up on the state change between the Enketo Time-Use survey back to the Time-Use screen:

Screen.Recording.2022-12-12.at.10.58.36.AM.mov

Here is what the phonegap server log says when that "Save"/"Next" buttons are pressed:

---

@shankari how does the state change work for the enketo survey stuff? Is there an argument I should be passing to the EnketoSurveyLaunch.launch() which tells Enketo what to state change back to when it is finished?

I also looked and saw that there is an ng-click=enketoSurvey.onNext() which I am guessing is the function responsible for state-changing us to the next-state-in-queue (if the onNext() function is working off of a queue, not sure)

[sebastianbarry]

I tried adding an onNext argument for the EnketoSurveyLaunch.launch() (the function that gets called when we click the big "+" button to add a time-use entry) to be a $state.go('root.main.time-use') but it is giving errors:

[shankari]

@MaliheTabasi I have a couple of thoughts/questions about this design based on our experience with OpenPATH UI design so far.

• I would suggest not using icons for the modes in the UI. Using icons reduces our flexibility wrt modes - if we add a new mode, we need to add a new icon, which requires a full UI change. We have run into this with the diary screen and the GIS integration, the OSM-based GIS allows us to support modes such as FERRY, but then we need to support FERRY as a mode throughout the app and add a ferry icon etc (@PatGendre @TTalex). In this case, the mode comes from a survey displayed to the user and we want the survey to be flexible and independent of the UI. Using icons removes this flexibility.
• I also don't like the vertical line on the side of the UI. It makes it much harder to modularize the UI and make each trip or activity a separate directive. I'm not saying that it is impossible - we can try to incorporate a line in each trip - but it might be tricky to get it to line up properly. We had such line in the diary, and it was outside the list item. We have since removed the line in the label screen, which has been designed by a professional designer (from FabMob Quebec, @lgharib), and has generally been praised for looking much better. What is the rationale for including the line? I understand the Google Maps includes it, but I assume that Google Maps also has a 10 person team working only on the timeline UI.
• In general, please avoid using the diary as the basis for new work since we will be removing it going forward. Please focus on the label screen, which does not have the line, supports additional filtering and solves the problem of indicating which days actually have trips. It will also solve your design choice of "first location at the start of the day is the end of the previous day" which would be super hard to implement with the way the diary screen data is retrieved.
• finally, a question: will you still want this to be separate from the diary/label tab? Is the idea that the diary/label tab is trip focused and this is place/activity focused?

[shankari]

@sebastianbarry you have not yet changed the time use code to use the existing enketo directives. I'm not going to help debug obsolete code, and you should not waste your time on it either.

[TTalex]

@lgharib @TTalex @jf87 is there interest from the rest of the community on a manually entered time-use survey? Would you be interested in co-designing it, or should we go with @asiripanich's thoughts and plans for now?

Quick note on that subject: the feature doesn't match any current use case from our side. We're not, in fact, big survey users. Following @asiripanich plan sounds good !

[sebastianbarry]

In order to get familiar with the new Enketo directives, I commented out the <ion-content> from the time-use tab and placed in <enketo-demographics-survey>:

It displays correctly inline. It looks like interacting with the Next or back buttons freezes the app the same way it would with the Time-Use enketo survey, although the "Found previous survey response" indicates that the survey we are being provided with the <enketo-demographics-inline> in this view is the demographics survey, not our time-use survey (which makes sense, I used the demographics directive, there is no directive for the time-use survey right now)

Screen.Recording.2022-12-14.at.10.22.24.AM.mov

I believe I should make an enketo directive for the time-use survey. Probably something like <enketo-time-use-inline>, which would funciton like this: Once the user clicks the "+" button to add a time use entry, a funciton would get called which would launch the time-use survey by inject the <enketo-time-use-inline> directive I would have created into the time-use/list.html, which would then display over everything else, and once "Next" is pressed, the <enketo-time-use-inline> would be removed from the html again and so the page would display.

Pros and cons of doing it this way:

• This is the way i understand this to work, but I do not think this is the best or correct way of doing this
• While I do think creating a directive for this purpose would be useful (because we can put it into things like popups), I do not like the idea of having to inject and remove a directive from list.html. It seems like a sloppy way of displaying a new page, but I am not sure how to incorporate solely a directive to display the time-use survey, without needing to have some sort of javascript .launchSurvey function

[shankari]

• @sebastianbarry there is a big issue with the recording above, mainly that it is not actually displaying the survey. When you launch the demographic survey from the profile, does it work?
• the way that directives work is that the HTML and the javascript are both put into the new module. So the + button would be within the <enketo-time-use-inline>. There would not be a separate div or a launchSurvey function outside the directive.

[sebastianbarry]

• @sebastianbarry there is a big issue with the recording above, mainly that it is not actually displaying the survey. When you launch the demographic survey from the profile, does it work?

Good eye, it looks like the demographics survey is not working (I had not observed the demographics survey until you pointed this out, but I believe it hasn't been working on this branch yet)

Screen.Recording.2022-12-14.at.11.24.51.AM.mov

* the way that directives work is that the HTML and the javascript are both put into the new module. So the + button would be _within_ the `<enketo-time-use-inline>`. There would not be a separate `div` or a `launchSurvey` function outside the directive.

This makes more sense. So essentially I can create a <enketo-time-use-inline> directive and use it inside of our time-use/list.html, which would include the "+" button so that when clicked on (internal to the directive's scope), it would open up the time-use enketo survey?

[sebastianbarry]

I just reverted my changes and confirmed that the demographics survey is not working with this branch

[shankari]

This makes more sense. So essentially I can create a directive and use it inside of our time-use/list.js, which would include the "+" button so that when clicked on (internal to the directive's scope), it would open up the time-use enketo survey?

@sebastianbarry I don't think you still understand what directives do and might need to continue reading up on them You don't use the directive inside list.js. The directive is a new custom HTML tag. You just use it in the HTML.

I just reverted my changes and confirmed that the demographics survey is not working with this branch

what error do you see? You need to get it working with the branch before making any additional changes.

[sebastianbarry]

@sebastianbarry I don't think you still understand what directives do and might need to continue reading up on them You don't use the directive inside list.js. The directive is a new custom HTML tag. You just use it in the HTML.

My bad, typo. I meant to say list.html I edited my previous comment.

what error do you see? You need to get it working with the branch before making any additional changes.

Here is the error I'm seeing for the demographics survey

[shankari]

FYI, I just spoke to a deployer who wants to allow people to add "untaken trips" - aka trips that they were not able to take due to lack of mobility options. It sounds like we may be able to reuse the time-use survey to support that - it would just be a different survey popping up when the user clicks on the big green button.

[shankari]

Here is the error I'm seeing for the demographics survey

Right, so does that file exist? If not, you might need to pull from master again. Recall that there is no guarantee that this branch is up to date with master.

It looks like interacting with the Next or back buttons freezes the app the same way it would with the Time-Use enketo survey,

Look carefully at the <enketo-demographics-inline> directive, including the difference between your usage and the intro usage and figure what the difference is. I know, but you should figure it out.

[sebastianbarry]

I pushed my commit, the enketo survey has been successfully loaded and successfully comes up and goes away once the user clicks "next". The Demographics survey also works again:

Screen.Recording.2022-12-15.at.1.35.17.PM.mov

Next steps:

• Get the time-use screen to start displaying the output of the results of this survey

[shankari]

and goes away once the user clicks "next".

I don't think that this is the expected behavior. In general, the surveys should go to the next screen when the user clicks "Next", as you might remember from the demographic survey feedback. Please look through the existing surveys and carefully understand how surveys are supposed to behave (including Amarin's video above) before declaring that they work.

[shankari]

Please also look into the "next" button and its behavior.

[shankari]

might want to check with @asiripanich on whether we want to pull out some fields to display in the label for this as well (since it will show up in the diary).

[shankari]

Change comments.

[shankari]

Why is this a new file that is checked in and how is it different from the www/js/survey//external/time_insert.js

[sebastianbarry]

I found that the "next" button is displaying because the KoboToolbox survey is just being generated in a strange way - in my testing I was able to get it to display without the "Next" button without needing to change any code at all: I copied the KoboToolbox survey for the demographics survey to use as a template, and then changed/deleted the questions I needed

[MaliheTabasi]

Hi @shankari and @sebastianbarry, here is the PDF of the new time-use UI.

Time-use-designes-Maliheh-Tabasi-14-12-2022.drawio.pdf

[sebastianbarry]

Fixed the time-use enketo survey and any issues it had ("next" button is removed, and added spanish translations):

Screen.Recording.2022-12-20.at.11.12.45.AM.mov

Spanish

[shankari]

@JGreenlee so the way the config works right now, is that we have a config file for each study/program on GitHub
https://github.com/e-mission/nrel-openpath-deploy-configs

• @sebastianbarry when the user scans the QR code to join the study or program, we download the config associated with it, and we notify config listeners that we have read the config. We also save the config locally and load it on every app start.
  https://github.com/e-mission/e-mission-phone/blob/master_for_platform/www/js/config/dynamic_config.js

• at this point, we don't support config "refresh" by default. you would need to uninstall + reinstall to get the new config.
• shouldn't be hard to add a debug button to use it, but I'm not sure we want to require UI users to refresh in that way.
• we could also support auto-refresh where we re-read the config from the server when the app is launched but I have not done that due to lack of time

where should we store these survey links?

• right now, there is a survey config and each survey config item has a file associated with it
  https://github.com/e-mission/e-mission-phone/blob/master/www/json/enketoSurveyConfig.json.sample

So there is a trip confirm survey at json/trip-end-survey-multiple-select.json and so when we try to load the survey, we say "get me the trip confirm survey" and it loads it from there. But it should work to load it from a remote URL as well
since it just uses $http.get

e-mission-phone/www/js/survey/enketo/service.js

Line 194 in 6b3ea0a

.then(config => _state.formLocation = config[name].formPath)

Whether we want to keep this two level design (local config -> URL) or just specify the URL directly in the code so we can get it directly from the dynamic config is something we need to investigate.

[shankari]

Two big immediate tasks are:

• make all the directives (list, enhanced trip, place) with these two inputs
• figure out how we can/should load the dynamically specified surveys

@sebastianbarry and @JGreenlee will figure out who works on what and come back with a design tomorrow.

[sebastianbarry]

@shankari both me and @JGreenlee met after our meeting today to discuss which aspect of this project each of us will work on. We decided

• Jack will work on the directive for the <infinite-scroll-place-item> directive which will go after each <infinite-scroll-trip-item> directive
• Sebastian will work on the https://github.com/e-mission/nrel-openpath-deploy-configs to create a new config field for each config that will define the places-survey-link, and also work on the dynamic surveys part of it

@JGreenlee here is the new branch we can work off of together (I took only the important information from my time-use branch and put it into this new places-survey branch: https://github.com/sebastianbarry/e-mission-phone/tree/places-survey

[shankari]

@sebastianbarry you are right that there is a mapping between the survey result and the value to be displayed on the button once the survey is complete.
https://github.com/e-mission/e-mission-phone/blob/master/www/js/survey/enketo/answer.js#L39

We will have to see if we can include this extraction in the dynamic config file as well so that, again, we don't have to change the code if we need to support new surveys.

[asiripanich]

Hi @sebastianbarry, I guess you don't need our time-use kobotoolbox form anymore right? Please let me know if you still want that, I can share with you on Kobotoolbox.

[shankari]

for surveys going forward: I think we should remove json/enketoSurveyConfig.json (including the fallback to enketoSurveyConfig.json.sample if the json file is not present) and move that functionality into the dynamic config. We should also move the LABEL_FUNCTIONS functionality into the dynamic config because it relies on labels from the enketoSurveyConfig.json and that is going away.

So my suggestion is to do this in two stages:

1. each dynamic config survey (trip-level-additions, place-level-additions, trip-level-label) should have its own structure, that includes the existing fields:

    "formPath": "json/trip-end-survey-multiple-select.json",
    "version": 1.2,
    "compatibleWith": 1,
    "dataKey": "manual/trip_user_input"

But also include

    "not-filled-in-label": in `en, es...`
    "filled-in-label": in `en, es...`

• for the "time use" use case: the formPATH will be https://github..../data-.json (like @sebastianbarry created), the not-filled-in-label will be {"en": "Add Activity", "es": ....} and the "filled-in-label" can be blank
• for the "missing trip" use case: the formPATH will be https://github..../data-.json (like @sebastianbarry created), the not-filled-in-label will be {"en": "Trip not taken"}, and the "filled-in-label" can be blank
• for the existing trip survey use case: the formPATH will be https://github..../data-.json (like @sebastianbarry created), the not-filled-in-label will be {"en": "Add trip details", "es": } and the "filled-in-label" will be {"en": "Submitted"}

[shankari]

To try out the trip details survey, change all these locations from SurveyOptions.MULTILABEL to SurveyOptions.ENKETO
Until you change everything to be in the dynamic config, you may be able to test this only on a branch that still reads from the enketoSurveyConfig.json

$ grep -r MULTILABEL www/js
www/js/diary/infinite_scroll_detail.js:  $scope.surveyOpt = SurveyOptions.MULTILABEL;
www/js/diary/infinite_scroll_list.js:  $scope.surveyOpt = SurveyOptions.MULTILABEL;
www/js/diary/services.js:    const surveyOpt = SurveyOptions.MULTILABEL;
www/js/diary/infinite_scroll_trip_item.js:    $scope.surveyOpt = SurveyOptions.MULTILABEL;
www/js/diary/list.js:  $scope.surveyOpt = SurveyOptions.MULTILABEL;
www/js/diary/diary_list_item.js:    $scope.surveyOpt = SurveyOptions.MULTILABEL;
www/js/diary/detail.js:  $scope.surveyOpt = SurveyOptions.MULTILABEL;

[shankari]

Potential future step (2) for #915 (comment) is to actually put in the functionality from the LABEL_FUNCTIONS into the dynamic config.

[shankari]

So since we are making everything dynamic, we should also convert the linkedsurvey to the dynamic config as well

                    <div class="col-90">
                        <linkedsurvey element-tag="{{surveyOpt.elementTag}}" class="col" trip="trip"></linkedsurvey>
                    </div>
                    <verifycheck linkedtag="{{surveyOpt.elementTag}}" style="padding: 8px" class="col-10 diarycheckmark-container center-vert center-horiz"></verifycheck>

https://github.com/e-mission/nrel-openpath-deploy-configs/blob/main/docs/index.html#L62

[shankari]

we should test the trip confirm options and make them dynamic as well. This involves two changes:

1. we need to change the dynamic configs to have multiple options: @sebastianbarry
   • multilabel + time use
   • multilabel without time use
   • trip confirm + time use
   • trip confirm without time use

• create configs for all of them
• update https://github.com/e-mission/nrel-openpath-deploy-configs/blob/main/docs/index.html#L62 to include them in the generated HTML https://e-mission.github.io/nrel-openpath-deploy-configs/

2. instead of having the MULTILABEL vs. ENKETO spread across multiple files and hardcoded, it needs to be read from the dynamic config and hopefully in only three places (infinite_scroll_list.js, list.js and services.js) @JGreenlee

[shankari]

Consider the following use case:

• user has taken a trip today morning
• trip data was pushed at trip end, analysed, we now have confirmed trip
• user loads label screen, sees confirmed trip
• user labels confirmed trip, label is stored in local SQLite DB for future upload
• user label in SQLite DB is linked to timestamps, not trip ID

What could go wrong?

• user loads label screen, sees confirmed trip

• user labels confirmed trip, label is stored in local SQLite DB for future upload

• user re-loads label screen before label is uploaded to server

• we will re-retrieve the confirmed trip from the server

• this will not have the matched user label copied into it

• so user perception will be "label disappears"

to work:

• load from local database
• match them to the newly retrieved confirmed trip objects
• look at the most recent timestamps to see what labels have been put in

Question for you: Why not only find the most recent timestamp? Why do we have to load all local labels first, match and then find the most recent instead of the other way around?

• user labels trip1 with wrt1, trip2 with wrt2, trip3 with wrt3...
• user re-loads
• we retrieve the label with the most recent timestamp (wrt3)
• we match it to trip 3
• but labels disappear for trip 1 and trip 2

we don't just want to find the most recent label
we want to find the most recent label for each trip
so we need to read the labels
match them to the trips
and then find the most recent for that trip

[shankari]

to adapt this to time-use, I would largely use what we do for mode/purpose/trip-confirm, with the caveat that the matching may need to change slightly because:

• we need to match for places as well
• there can be multiple matches per trip or place

[sebastianbarry]

@shankari and @JGreenlee

I have added the the new surveys info/ survey information in the config files how I think they should work, (e-mission/nrel-openpath-deploy-configs#8) please take a look and let me know if there is anything else that should be added, such as:

• Is there a better way to convert the linkedsurvey using the dynamic config?
• Is there a better way to add the LINKED_FUNCTIONS functionality in the dynamic config?
  • Currently, my idea is that if there is a button named trip-labels, then we should display the TripConfirmSurvey instead of the MULTILABEL, otherwise if there is no button named trip-labels present, then display MULTILABEL

[shankari]

• SB + JG: understand this at the high level (Thu)
• SB + JG: make sure that you test and understand the trip confirm flow on the phone (Thu)
• SB + JG: By EOD Thursday: you need to have the data stored in the local SQLite database when it is filled in
  • you can test this by going to the profile -> developer zone -> email the sensed data to yourself and open it using sqlite3 from the command line
    • not sure it works on the iOS emulator, should work on the android emulator
    • on OSX, you can find it under /Users/<username>/Library/Developer/Xcode/DerivedData (and then follow the instructions here: https://stackoverflow.com/questions/11761695/where-does-the-iphone-simulator-save-the-sqllite-database)
    • find /Users/<username>/Library/Developer/DerivedData -name \*userCacheDB
  • I should be able to scan any of the QR codes that sebastian created
  • I should be able to fill out trip confirm and/or trip addition and/or place addition
  • I should be able to see the stored data for all three types of surveys in the SQLite database

==========================

• KS: I can make the server changes to accept this new type of data and to match it with trips and to store it in the server
  • KS: If all of your code is done and I can see the format on the server, I can add a hack to return fake data from the server for some subset of the server trips (EOD Mon)
  • SB + JG: review my changes so you can expand your understanding
• SB + JG: so then when you retrieve the confirmed trips from the server (after applying my changes), you should be able to see the list of trip additions in the confirmed trip if they have been processed (starting Tue)
• SB + JG: you should also be able to replicate the matching code on the phone (note that it will be subtly different than the current one because it matches an array instead of a single item)
• SB + JG: then you can read the unprocessed labels the same way as the existing trip confirm unprocessed labels and call the matching code on the phone (try for a day, fall back to KS if not possible)

[shankari]

For multi-label, the flow is:

• user fills out labels
• we store the labels to local SQLIte DB (aka BEMUserCache)
• we cache the labels in the trip object (in-memory)
• if the user re-loads the trips, then we retrieve labels from the SQLiteDB since the in-memory cached values will be deleted when the new trips are loaded.

[shankari]

@sebastianbarry @JGreenlee I found a non-empty userCache at

/Users/<username>/Library/Developer/CoreSimulator/Devices/5DECB15E-BADD-419C-B826-8AFC2E2F2C2F/data/Containers/Data/Application/6BC77056-4993-45BE-BDAC-DBA88E75F14B/Library/LocalDatabase/userCacheDB

SQLite version 3.32.3 2020-06-18 14:16:19
Enter ".help" for usage hints.
sqlite> select * from userCache;
1601539204.94958||America/Los_Angeles|message|stats/client_nav_event||{"ts":1601539204.949321,"client_app_version":"1.3.0","name":"app_launched","client_os_version":"13.7","reading":-1}
1601539205.43736||America/Los_Angeles|local-storage|connection_settings||{"ios":{"auth":{"method":"dummy-dev"}},"connectUrl":"http:\/\/localhost:8080"}

Again, this seems to be consistent with https://stackoverflow.com/a/24153116/4040267

You could also look at the logs from
https://github.com/e-mission/cordova-usercache/blob/master/src/ios/BEMBuiltinUserCache.m#L72 to see if they are meaningful.

Note that they will only be displayed on first install.

[shankari]

• the data is indeed cached on the phone and pushed up in a batch
  • this pushing happens at two times:
    • at the end of a trip assuming network connectivity at this point
    • every hour (roughly, roughly because there can be doze mode or other restrictions on periodic background access, so sometimes if the user does not have the app plugged in overnight, it could take 6-8 hours)
    • but as long as the user eventually has access to the internet, the data will make it to the server
• to manually trigger a push, Profile -> Developer -> End Trip and Force Sync
  • you will no longer see this in the user cache
  • you will see this in the webserver logs (/var/log/webserver.log)
• now the data is on the server, but has not yet been processed or matched
  • processing/matching happens as part of the pipeline (or will happen for the trip-additions after I write the code)
  • the user input keys that are matched are defined in conf/analysis/debug.conf.json.sample, so if you want to run this with trip_confirm objects you need to change the userinput.keylist field there
  • so you need to run the pipeline (./e-mission-py.bash bin/debug/intake_single.py similar to when you load the fake data)
  • any confirmed trips whose user input just got pushed up should now have the user input when you re-load, you should be able to verify this with a local debugger

Local database

• the local database is used to avoid pushing up data in real-time and using the users' data plan and keeping the radio on, which drains battery
• however, a too larger local database will eat up phone storage which is limited
• so when we can upload the data to the server (which is larger and has lots of storage and the storage is paid for by NREL instead of the app user), we delete the data from the phone
• after the data has been processed on the server, we largely use standard retrieval techniques to pull and work with the data
• the data is not "sent" to the phone, it is retrieved by the app using the network when the user launches the app

[shankari]

in terms of trip matching or figuring out which entries are currently valid,
let's say the user had an activity a1 from t1 to t5 and then a2 from t5 to t10
they now want to edit the first activity so it is from t1 to t7
how do we deal with this? would we delete a1? would we automatically shrink a2?

how will we support edits and deletes in general?

• all user inputs are immutable so if a user does delete an activity, we should mark it as DELETED and filter it out instead of actually removing the entry
• but I still don't know how to deal with edits because we have not had to do that before

Native approach easy for matching at least:

• don't allow edits; user has to delete activities and then re-enter them
  • maybe we can start with this to make the system work end to end and then optimize by auto-deleting when the user edits
• later: if the user tries to save an activity then we may delete all existing entries that overlap the new range

[sebastianbarry]

@shankari Here is what I understand concerning uploading the keys manual/trip_confirm_input and manual/timeuse_survey from the localdb to the server. Let me know if I'm off track with anything:

Firstly, I know that if we fill out the trip confirm survey, the response is saved on the localdb successfully (and same goes for timeuse survey):

Now, in order to upload the localdb to the server manually we must do the following:

• to manually trigger a push, Profile -> Developer -> End Trip and Force Sync
  • you will no longer see this in the user cache
  • you will see this in the webserver logs (/var/log/webserver.log)
    Doing this will run the pipeline, which contains a "matching" step which and that is what causes these keys to be pulled from the localdb to be uploaded to the server. When these are uploaded to the server, 3 things happen which will identify that it has successfully completed...

1. The localdb version of the key is deleted
2. The key is logged in the server and the databases logs (webserver.log and the docker logs respectively)
3. The trip will be associated with it's trip_confirm_input answer, so when we refresh the label screen we should see that reflected as trip with the filled out survey will no longer be in the to-label filter

---

The problem I am running into is that I can't press the "End Trip and Force Sync" button on any study/program with trip_confirm_input activated because it appears to be causing an issue in the control screen:

Program (Multilabel + timeuse survey) - End Trip and Force Sync working
Study ( Tripconfirm, no timeuse survey) - End Trip and Force Sync not working

Here is the error message I believe is associated with it:

	I debugged it and the ConfirmHelper.inputParamsPromise promise accepts, meaning the .then() runs. I think the issue is somewhere inside the .then() but I can't tell from where there is an undefined object

[shankari]

@sebastianbarry @JGreenlee, that is a complicated error to debug further/fix.
The Sydney team is the primary user of the ENKETO trip confirm functionality, and they don't use the dashboard.
So disable the dashboard (in www/templates/main.html) if the trip input type is ENKETO.
That should allow you to avoid that error to the control screen and force sync.

[sebastianbarry]

@shankari looks like commenting out the metrics screen didn't fix the control screen:

I tried this on both our places-survey branch, and also the master-for-platform branch (with all MULTILABEL changed to ENKETO)

[JGreenlee]

@sebastianbarry
Let's add a little try/catch to prevent the crash, at least for right now

[shankari]

so with the current design of the survey specification, we should also consider where we want to store the data keys.
Currently, we store the data.key along with the survey name.

        "TimeUseSurvey": {
          "formPath": "https://raw.githubusercontent.com/sebastianbarry/nrel-openpath-deploy-configs/surveys-info-and-surveys-data/survey-resources/data-json/time-use-survey-form-v9.json",
          "version": 9,
          "compatibleWith": 1,
          "dataKey": "manual/timeuse_survey"
        }
      },

• That's fine but then if we want to store a new type of survey (e.g. missing trip survey), we cannot do so without adding the new data type to the server

• Because there is currently a 1:1 mapping between the key and the data model python file on the server.

• Don't want to change this because then it will be harder for people to find the matches.

• Also, what data would we pull from the sever to match?

  • match on the server, we read inputs from the configured keys and try to match them
  • match on the phone, we read unmatched inputs based on the survey type and try to match them

so if we kept the data key associated with the survey type, then if the user was using a time use, we would need to read manual/timeuse_survey and if the user was using missing trip, we would need to read manual/missingtrip_survey

We should re-think this approach and potentially store the key in the buttons instead

For time-use, this will look like

      "buttons": {
        "trip-notes": {
          "surveyName": "TimeUseSurvey",
          "survey_data_key: "manual/trip_addition_input"
...
        },
       "place-notes": {
          "surveyName": "TimeUseSurvey",
          "survey_data_key: "manual/place_addition_input"
          ....
      }

and for the missing trip, it will look like

      "buttons": {
       "place-notes": {
          "surveyName": "MissingTripSurvey",
          "survey_data_key: "manual/place_addition_input"
          ....
      }

We theoretically don't even need to have the survey_data_key - that can be hardcoded in the HTML based on where the button is. So the directive will still take in the key, but when we use the directive, we can pass in manual/place_addition_input or manual/trip_addition_input.

If we hardcode the survey_data_key in the HTML, this could be even simpler

      "buttons": {
        "trip-notes": {
          "surveyName": "TimeUseSurvey",
...
        },
       "place-notes": {
          "surveyName": "TimeUseSurvey",
          ....
      }

and

      "buttons": {
       "place-notes": {
          "surveyName": "MissingTripSurvey",
          ....
      }

[shankari]

ASSUMPTION: there will only be one kind of trip addition and one kind of place addition per program and if they want multiple types of inputs, they will just use a more complex survey.

The advantage of this is that we can hardcode the survey key into the retrieval on both phone and server and not have to test through multiple different config options.

[shankari]

note also that trip additions are an array.

• users hit "Add trip activity" and they create a new entry
• user hits "Add trip activity" again, and they create another new entry
• then they go and delete the first entry (maybe a day later after we have processed everything)

how do we deal with the matching in this case?

my proposal is:

• each trip_addition_entry has a status of ACTIVE or DELETED
• when the user adds, we add an entry with ACTIVE
• when the user deletes, we add an entry with the same id and status DELETED

and in the matching, we remove DELETED entries from the array and add new ACTIVE entries to the array

we will maintain a local (in-memory) copy of the array in the trip object that we manipulate in addition to storing the data in the usercache.

• so if the user makes changes within the same app launch, we will manipulate the in-memory copy directly
• when the user refreshes (before processing), the in-memory array will go away, but we will reconstruct by mapping on the phone
• when the user refreshes (after processing), the matching would have been done on the server and we will just see the results in the retrieved trip.

matching on id:

• once the data has reached the server, there is a id that the mongodb database gives it which we can use
• but it is a good point that if the user has added a trip and deletes it before the data has been pushed, we will not have that id

Potential solutions:

• create an id on the phone when the user creates an addition (call it add_id to distinguish from _id in the database)
  OR
• match based on start and end ts

[shankari]

On the phone

• change the directive to take in the data key (JG)
  • pass it through to the survey code (SB)
• actually have an in-memory array (associated with the trip) to store the activities (JG work within one session)
• retrieve the unprocessed trip additions on reload to match on the phone (SB)
• implement the multi_non_deleted matching on the phone so the retrieved additions are inserted into the trip (SB)

On the server (KS)

• match draft additions (although I think this is much less critical because they don't show up on the label screen anyway)
• add more unit tests

[shankari]

Closing this since we have made the changes in #917 and #919 instead.