Skip to content
This repository has been archived by the owner on Dec 16, 2019. It is now read-only.

Test ingestion on big DemoData #27

Closed
jhagberg opened this issue Mar 6, 2017 · 2 comments
Closed

Test ingestion on big DemoData #27

jhagberg opened this issue Mar 6, 2017 · 2 comments
Assignees
@silverdaz @nanjiangshu
Milestone
Sprint 13

Comments

@jhagberg
Copy link

jhagberg commented Mar 6, 2017 •
edited by nanjiangshu
Loading

400 Gb
@jhagberg jhagberg modified the milestones: Sprint 4, Sprint 5 Apr 7, 2017
@silverdaz
Copy link
Contributor

Tool scripts in the repo now, to help setting up the pipeline

@silverdaz silverdaz assigned nanjiangshu and unassigned andersla Apr 25, 2017
@jhagberg jhagberg modified the milestones: Sprint 7, Sprint 6 May 8, 2017
@nanjiangshu nanjiangshu changed the title Test ingestion on "real" DemoData Test ingestion on big DemoData May 18, 2017
@jhagberg jhagberg modified the milestones: Sprint 7, Sprint 8 May 23, 2017
@jhagberg jhagberg modified the milestones: Sprint 9, Sprint 8 Jun 12, 2017
@nicjar nicjar modified the milestones: Sprint 10, Sprint 9 Jun 21, 2017
@nicjar nicjar modified the milestones: Sprint 11, Sprint 10 Aug 9, 2017
@nicjar nicjar modified the milestones: Sprint 12, Sprint 11 Aug 24, 2017
@nanjiangshu nanjiangshu modified the milestones: Sprint 12, Sprint 13 Sep 21, 2017
@nanjiangshu
Copy link

nanjiangshu commented Sep 21, 2017 •
edited
Loading

Tested with two files of size 100GB of each using the implement of Docker version of LocalEGA. The pipeline was running on a desktop computer with 4-cores (8 threads) and 16GB RAM. Files are successfully processed and moved to the archive vault, and it took about 3 hours to process each file.

However, there is a small error which shows integer out of range when writing the record to the postgrel database. A new issue will be opened for debugging of this.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@silverdaz silverdaz

@nanjiangshu nanjiangshu

Labels
None yet
Projects
None yet
Milestone
Sprint 13
Development

No branches or pull requests
5 participants
@silverdaz @jhagberg @nicjar @nanjiangshu @andersla