IndexBuilderMain quits without finishing correctly or showing error message

[bumuellp]

Hello,
I tried to build an index for wikidata.2018_01_07.truthy.nt and it just quits at the same point without showing any message. The last output from IndexBuilderMain (Debug build):

IndexBuilderMain, version Apr 25 2018 19:57:49

Set locale LC_CTYPE to: en_US.utf8
Thu Apr 26 01:28:17.317 - DEBUG: Configuring STXXL...
Thu Apr 26 01:28:17.396 - DEBUG: done.
Thu Apr 26 01:28:17.397 - INFO: Making pass over NTriples /nfs/raid1/wikidata/wikidata.2018_01_07/wikidata.2018_01_07.truthy.nt for vocabulary.
Thu Apr 26 01:28:52.134 - INFO: Lines processed: 10,000,000
Thu Apr 26 01:29:27.725 - INFO: Lines processed: 20,000,000
Thu Apr 26 01:30:08.776 - INFO: Lines processed: 30,000,000
Thu Apr 26 01:30:41.368 - INFO: Lines processed: 40,000,000
Thu Apr 26 01:31:34.868 - INFO: Lines processed: 50,000,000
Thu Apr 26 01:32:07.549 - INFO: Lines processed: 60,000,000
Thu Apr 26 01:32:38.931 - INFO: Lines processed: 70,000,000
Thu Apr 26 01:33:10.401 - INFO: Lines processed: 80,000,000
Thu Apr 26 01:33:39.564 - INFO: Lines processed: 90,000,000
Thu Apr 26 01:34:09.474 - INFO: Lines processed: 100,000,000
Thu Apr 26 01:34:41.064 - INFO: Lines processed: 110,000,000
Thu Apr 26 01:35:13.795 - INFO: Lines processed: 120,000,000
Thu Apr 26 01:36:27.565 - INFO: Lines processed: 130,000,000
Thu Apr 26 01:36:57.448 - INFO: Lines processed: 140,000,000
Thu Apr 26 01:37:30.877 - INFO: Lines processed: 150,000,000
Thu Apr 26 01:38:02.920 - INFO: Lines processed: 160,000,000
Thu Apr 26 01:38:36.414 - INFO: Lines processed: 170,000,000
Thu Apr 26 01:39:09.024 - INFO: Lines processed: 180,000,000
Thu Apr 26 01:39:41.438 - INFO: Lines processed: 190,000,000
Thu Apr 26 01:40:13.883 - INFO: Lines processed: 200,000,000
Thu Apr 26 01:40:44.382 - INFO: Lines processed: 210,000,000
Thu Apr 26 01:41:16.908 - INFO: Lines processed: 220,000,000
Thu Apr 26 01:41:48.740 - INFO: Lines processed: 230,000,000
Thu Apr 26 01:42:21.168 - INFO: Lines processed: 240,000,000
Thu Apr 26 01:42:53.015 - INFO: Lines processed: 250,000,000
Thu Apr 26 01:43:23.460 - INFO: Lines processed: 260,000,000
Thu Apr 26 01:43:53.399 - INFO: Lines processed: 270,000,000
Thu Apr 26 01:48:29.093 - INFO: Lines processed: 280,000,000
Thu Apr 26 01:49:47.518 - INFO: Lines processed: 290,000,000
Thu Apr 26 01:50:33.165 - INFO: Lines processed: 300,000,000
Thu Apr 26 01:51:08.086 - INFO: Lines processed: 310,000,000
Thu Apr 26 01:51:43.910 - INFO: Lines processed: 320,000,000
Thu Apr 26 01:52:14.771 - INFO: Lines processed: 330,000,000
Thu Apr 26 01:52:45.578 - INFO: Lines processed: 340,000,000
Thu Apr 26 01:53:16.245 - INFO: Lines processed: 350,000,000
Thu Apr 26 01:53:45.715 - INFO: Lines processed: 360,000,000
Thu Apr 26 01:54:18.685 - INFO: Lines processed: 370,000,000
Thu Apr 26 01:54:49.505 - INFO: Lines processed: 380,000,000
Thu Apr 26 01:55:19.311 - INFO: Lines processed: 390,000,000
Thu Apr 26 01:55:49.265 - INFO: Lines processed: 400,000,000
Thu Apr 26 01:56:21.707 - INFO: Lines processed: 410,000,000
Thu Apr 26 01:56:52.822 - INFO: Lines processed: 420,000,000
Thu Apr 26 01:57:24.288 - INFO: Lines processed: 430,000,000
Thu Apr 26 01:57:57.092 - INFO: Lines processed: 440,000,000
Thu Apr 26 01:58:30.770 - INFO: Lines processed: 450,000,000
Thu Apr 26 01:59:05.115 - INFO: Lines processed: 460,000,000
Thu Apr 26 01:59:38.426 - INFO: Lines processed: 470,000,000
Thu Apr 26 02:00:11.201 - INFO: Lines processed: 480,000,000
Thu Apr 26 02:00:42.520 - INFO: Lines processed: 490,000,000
Thu Apr 26 02:01:12.596 - INFO: Lines processed: 500,000,000
Thu Apr 26 02:01:44.377 - INFO: Lines processed: 510,000,000
Thu Apr 26 02:02:20.818 - INFO: Lines processed: 520,000,000
Thu Apr 26 02:02:51.878 - INFO: Lines processed: 530,000,000
Thu Apr 26 02:03:23.206 - INFO: Lines processed: 540,000,000
Thu Apr 26 02:03:56.946 - INFO: Lines processed: 550,000,000
Thu Apr 26 02:04:30.181 - INFO: Lines processed: 560,000,000
Thu Apr 26 02:05:04.523 - INFO: Lines processed: 570,000,000
Thu Apr 26 02:05:38.386 - INFO: Lines processed: 580,000,000
Thu Apr 26 02:06:11.430 - INFO: Lines processed: 590,000,000

The comand was:
./IndexBuilderMain -n /nfs/raid1/wikidata/wikidata.2018_01_07/wikidata.2018_01_07.truthy.nt -i /local/data/bumuellp/wikidata_dump-copies/wikidata2018-a_index -a

System info:
*-cpu
product: AMD FX(tm)-8150 Eight-Core Processor
vendor: Advanced Micro Devices [AMD]
physical id: 1
bus info: cpu@0
size: 1400MHz
capacity: 3600MHz
width: 64 bits
capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp x86-64 constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core perfctr_nb cpb hw_pstate vmmcall arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold cpufreq
*-memory
description: System memory
physical id: 0
size: 31GiB

Could this be a memory issue or do you have any other suggestions?

Thank you and best Regards
bumuellp

[niklas88]

Anything in dmesg? Could be running out of memory. Also is this reproducible?

[bumuellp]

[9387028.648451] Out of memory: Kill process 26378 (IndexBuilderMai) score 976 or sacrifice child
[9387028.648496] Killed process 26378 (IndexBuilderMai) total-vm:54436144kB, anon-rss:32186196kB, file-rss:0kB

Yes it is reproducible and it seems to be a memory issue.
So I have to upgrade my memory? Or can i just change some swap configurations?

Thanks!