JSON import/export

[dkozar]

🙋 Feature Request

@nnoll suggested the feature in #490 (comment)

😯 Describe the feature

Project parameters should be able to be exported as JSON, and then imported back. This should bring back the original state.

I'd like the UX to propose solution for the UI.

💻 Examples

We already offer a JSON parameters download as a file. Which means we need to add the upload.

Here's an example of how the JSON import/export is usually implemented (link to the app here):

@nnoll please is this what we want, or we want to build a JSON file upload?

Related

• feat: Preset loader #490 (comment)

[nnoll]

@dkozar I was envisioning a true file upload instead of a string dump

[whiver]

For visibility, this is also related to what @nnoll suggested in my PR #381 too, but only for the case counts in my case and not simulation parameters.

[ivan-aksamentov]

File uploads and downloads are mostly already implemented.

The upload have been sitting here for a long time now (since when we planned custom case counts upload and comparison):

https://github.com/neherlab/covid19_scenarios/blob/40e646ef23453a2b7555529523deb9f18cc5a88d/src/components/Main/Compare/ComparisonModalWithButton.tsx

https://github.com/neherlab/covid19_scenarios/blob/40e646ef23453a2b7555529523deb9f18cc5a88d/src/components/Main/Compare/FileUploadZone.tsx

https://github.com/neherlab/covid19_scenarios/blob/40e646ef23453a2b7555529523deb9f18cc5a88d/src/helpers/readFile.ts

One just needs to implement a cohesive UX around it.

[shijurodhaz]

Hello, I'm new to open source contributions but was hoping to help out with this issue. I was thinking of implementing the UX as follows:

• Add an IMPORT button below the current EXPORT button
• Display a modal that allows the user to upload a JSON file
• If the file is uploaded and the user clicks DONE, the data gets populated or an error is displayed (if the file is formatted incorrectly)
• The user can also click CANCEL instead to go back to the main screen.

Does this make sense or am I not understanding the feature correctly? Thank you!

[ivan-aksamentov]

@shijurodhaz This is correct, yes, with a few additions:

• Instead of creating a new modal, we would like to merge this "scenario loader" widget (to replace current scenario dropdown) and to further extend it with the export functionality (issue: Replacing the scenario dropdown with "Load scenario" dialog #397, PR: feat: Preset loader #490)

• The dropzone and file reader are mostly implemented, as pointed out in my comment above

• As an example data for the upload you can use the following

  • for scenario population, epidemiological, simulation parameters and mitigation table, you can take one of the elements of the array (for one country) in: https://github.com/neherlab/covid19_scenarios/blob/master/src/assets/data/scenarios/scenarios.json
  • for age distribution - the corresponding country's element in this array:
    https://github.com/neherlab/covid19_scenarios/blob/master/src/assets/data/country_age_distribution.json
  • for severity table, you can take all of the: https://github.com/neherlab/covid19_scenarios/blob/master/src/assets/data/severityData.json

• The import feature should create a seamless experience with the corresponding export feature. Note that the export currently does not produce the sufficient data in order to reproduce the results (see the list above). That has to be adjusted.

• On input and output, data should be serialized/deserialized and validated using JSON-schema-generated types and validators. For example, see how scenarios are loaded from the default json files in getScenarioData(): https://github.com/neherlab/covid19_scenarios/blob/master/src/components/Main/state/scenarioData.ts
  Similarly for other pieces.

• Serialization and validation errors should be reported to the user. Not necessary to create a UI for that right now, but error handling should be very clearly defined and not just console.error() everywhere.

• Import should allow versioning. That is, in a newer version of the application, we should be able to salvage as much of the data as possible from user's file to create a run. For example, currently on master we have changed parameters r0 and mitigationStrength from being numbers to a pair of numbers (array of 2 elements). This means that if user would try to import a file exported in previous version, the import would break. Instead, exported file should contain a "version" field, so that we know which version of schema to use. After validation, a fallback mechanism should be applied that transforms old data to new data as much as possible. For example, In case of array/number change we can set both elements of the array to the same number.

I will review and merge #490 tomorrow. But the UI part is not going to be the hard one. It is mostly about getting all pieces of data correctly in all cases between export and import.

As a robustness chack: user should be able to import-export the same data in a loop and not loose any data and not produce any errors, no matter how many loop iterations are there.

[shijurodhaz]

@ivan-aksamentov Since issue #397 is related to this one and is still open, I could try tackling that one first and maybe return to this later.

[ivan-aksamentov]

@shijurodhaz The #397 was solved in #490 . Sorry for not making it clear.
We have only 2 days remaining before the release 1.2, so I think I will take over the import/export feature.

Feel free to explore our other issues if you have time
https://github.com/neherlab/covid19_scenarios/issues

[shijurodhaz]

Sure, I'll try to check the other issues out. Thank you.

[whiver]

@ivan-aksamentov I'm modifying FileUploadZone.tsx in #381 for the case count import. I'm adding more features, like the accepted file extensions, error handling and some styling. I think it's mergeable now, even if the UI is not perfect (I can improve it afterwards).

[ivan-aksamentov]

@whiver Thanks. We did not plan to merge it before we release 1.2. I guess we can resolve any conflicts afterwards.

[ivan-aksamentov]

Resolved in #635