User Story 1: Logger/streaming data

[DavidBlaa]

capabilities to handle gradually growing tabular data up to 10Mio records/tuples.

1. It should be possible to have multiple uploads resulting in 10 M record
2. It should be possible to upload a 10M records.
3. A potential performance improvement would be to reduce the validation and primary checks (David, needs to be done in any case)

• No PK at all (2 day; prototype needed to test and measure the effect)
• Append only file-based edits
• Online editing for cells, without checking the PK
• static PK at data structure level

4. The priority is to reduce end-user’s response time versus reducing the task time. (@javadch )

• A possible solution would be to manage long-running processes asynchronously.
• This needs queuing (multiple priority queues), retry, and notification (offline, online), workflow resuming mechanisms.
• Also, it may need a detailed activity logging.

[DavidBlaa]

only test the first 2 items

[DavidBlaa]

works in v2.12 rc4