New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User Story 1: Logger/streaming data #30

Closed
DavidBlaa opened this Issue May 22, 2018 · 2 comments

Comments

@DavidBlaa
Member

DavidBlaa commented May 22, 2018

capabilities to handle gradually growing tabular data up to 10Mio records/tuples.

  1. It should be possible to have multiple uploads resulting in 10 M record
  2. It should be possible to upload a 10M records.
  3. A potential performance improvement would be to reduce the validation and primary checks (David, needs to be done in any case)
  • No PK at all (2 day; prototype needed to test and measure the effect)
  • Append only file-based edits
  • Online editing for cells, without checking the PK
  • static PK at data structure level
  1. The priority is to reduce end-user’s response time versus reducing the task time. (@javadch )
  • A possible solution would be to manage long-running processes asynchronously.
  • This needs queuing (multiple priority queues), retry, and notification (offline, online), workflow resuming mechanisms.
  • Also, it may need a detailed activity logging.
@DavidBlaa

This comment has been minimized.

Show comment
Hide comment
@DavidBlaa

DavidBlaa Aug 7, 2018

Member

only test the first 2 items

Member

DavidBlaa commented Aug 7, 2018

only test the first 2 items

@DavidBlaa

This comment has been minimized.

Show comment
Hide comment
@DavidBlaa

DavidBlaa Aug 22, 2018

Member

works in v2.12 rc4

Member

DavidBlaa commented Aug 22, 2018

works in v2.12 rc4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment