Consider moving the data storage to eg. sqlite

[benzea]

Currently the data is stored by simply pickeling it. This works really great, but my guess is that this is a limitation in some use cases (you need to create a new project, etc. every time).

A first step to make SDAPS more flexible in this regard would be to move to an SQLite based storage per project. Once that works, one could consider using only one database for multiple projects.

Doing this could make it easier/possible to automatically sort scanned data into the correct project.

[benzea]

Maybe some comments (as there was interest by some party to work on it).

• There is a main object tree representing the questionnaire metadata. It needs to be serialized/deserialized to a database. (survey.questionnaire, objects in the questionnaire.py file)
• Data is mostly accessed indirectly using the index from the global survey object. One solution might be to load/store data whenever the index changes.
• Per image/questionnaire (called sheet internally) data storage objects are in sheet.py and data.py. These need to be loaded/stored when the index changes.
• The only code that seems to access the sheets lists directly seems to be the reorder module. That will need to be changed at some point (but it is fine if it is broken for a while)

[benzea]

Oh, and

• have a look at how data is accessed (trough DataObject). You may of course modify this as you want.

[benzea]

One more note, as I was just thinking about it. For a start we could only move the per-sheet data storage over into sqlite. The questionnaire/survey information could still stay in the current format.

That really means that we may only need a simple (sheet, id, key, value) storage system that supports values of different data types.

[benzea]

This has now happened. It just dumps json seralised objects into the database. One for the questionnaire and one for each "sheet".