Make: 2019-03 enwiki fails during wiki.page_link with "A fatal error has been detected"

[gnosygnu]

The 2019-03 enwiki build failed with the error below

I think the problem is that the CREATE INDEX failed on the temp category pagelinks table b/c it has 1,215,821,988 rows.

I should have a fix sometime tomorrow.

---

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f583929b8ae, pid=3111, tid=0x00007f57dc4c9700
#
# JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
# Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.6.0
# Distribution: Custom build (Fri Nov 10 13:38:19 UTC 2017)
# Problematic frame:
# V  [libjvm.so+0x7fc8ae]  PerfLongVariant::sample()+0x1e
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# hs_err_pid3111.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
#
Aborted (core dumped)

[desb42]

Strange
I have just been building enwiki 2019-03-01 and have even just got past the html_dump phase (which is a first time for me)
I suspect that one difference is that for the categories, I now use file sizes set at 8Gb rather than the 2GB you use
specifically, i have
xowa.api.bldr.wiki.import.cat_link_db_max = 7800; // in MB
in xowa.gfs

This is to keep the build of commons and its categories under the 10 attach table limit

[gnosygnu]

Oops. Wrote down the wrong table above. Should be pagelinks db, not category db. Note that pagelinks will always have the most number of records (keeps track of all links from one page to another).

@desb42 Congrats on building enwiki (and thanks for the issues; will go through them later, but am spending part of weekend to release new XOWA version). I'm sort of surprised this didn't fail for you, but given that the error is in the sqlite lib (somewhere in C, not Java), then it's possible that this is an OS related issue. For comparison, the above error occurred on an openSUSE 13 system.

It took me a few attempts, but I finally got the code working. Part of the problem was that XOWA was indexing 1.2 billion varchar fields which seems to break some internal SQLite code (at least on Linux OS). The other part of the problem is that SQLite starts having issues when the database size goes beyond 50 GB (VACCUUMing starts failing, but this time in Java).

At any rate, build is continuing on my side. Thanks!

[desb42]

Having just read your comments in Pglnk_tempdb_mgr.java, I can confirm I build on a Windows 10 box

My enwik xowa.wiki.pagelinks.sqlite3 is 55.5GB with tables and row counts:
page_link 998,199,440
page_rank_temp 19,251,790
penalty_summary 80,468
temp_page_len 19,251,790
temp_page_score 19,251,790

Mine is just below what might be the threshold of a billion? causing the issue (as suggested)
I find it curious that my numbers seem to be different

[gnosygnu]

Thanks for the confirmation. I suspected you were on Windows, but didn't know which version

The 1.2 billion number is from the original enwiki-20190301-pagelinks.sql dump. I believe this collapses to 998 k b/c some pages will have multiple links to the same page . pseudo-example below.

My numbers are basically the same except for penalty_summary. See below.

Hope this helps.

page_id | target_id
1 | 2
1 | 3
1 | 2

---

db size = 55.6 GB

---

sqlite> .tab
page_link        penalty_summary  temp_page_score
page_rank_temp   temp_page_len    xowa_cfg       
sqlite> SELECT Count(*) FROM page_link;
998199440
sqlite> SELECT Count(*) FROM page_rank_temp;
19251790
sqlite> SELECT Count(*) FROM penalty_summary;
1199500
sqlite> SELECT Count(*) FROM temp_page_len;
19251790
sqlite> SELECT Count(*) FROM temp_page_score;
19251790
sqlite> SELECT Count(*) FROM xowa_cfg;
0

[gnosygnu]

Just one other note before closing the issue:

The 1.2 billion number is from the original enwiki-20190301-pagelinks.sql dump. I believe this collapses to 998 k b/c some pages will have multiple links to the same page . pseudo-example below.

Actually, the main reason this probably reduces is because of missing links (redlinks). The dump includes all links, even if they don't refer to a real page