Not able to load large Turtle files in V7

[jakubklimek]

Virtuoso 7
commit a9356b9c8a52d5ce7ac1864df922ce46c67acd2f (April 25)

When loading large files (2GB) using DB.DBA.TTLP_MT (file_to_string_output ('filename'), '', 'graphname');
virtuoso eventually either hangs (CPU usage 100% for a looong time but nothing happens) or it says Transaction deadlocked.

In virtuoso.ini:

;; Uncomment next two lines if there is 8 GB system memory free
       NumberOfBuffers          = 680000
       MaxDirtyBuffers          = 500000

the machine has 8GB of RAM.

In the log:

21:41:29 Write wait on column page 69382.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:39 Failed to get a buffer for a new page. Retrying.  If the failure repeats, an out of disk error will be signalled.  The cause of this is having too many buffers wired down for preread, flush or group by/hash join temp space.  To correct, increase the number of buffers in the configuration file.  If this repeats in spite of having hundreds of thousands  of buffers, please report to support.
21:41:50 Write wait on column page 4355.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:50 Write wait on column page 66409.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:50 Write wait on column page 69420.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:50 Failed to get a buffer for a new page. Retrying.  If the failure repeats, an out of disk error will be signalled.  The cause of this is having too many buffers wired down for preread, flush or group by/hash join temp space.  To correct, increase the number of buffers in the configuration file.  If this repeats in spite of having hundreds of thousands  of buffers, please report to support.
21:41:50 Failed to get a buffer for a new page. Retrying.  If the failure repeats, an out of disk error will be signalled.  The cause of this is having too many buffers wired down for preread, flush or group by/hash join temp space.  To correct, increase the number of buffers in the configuration file.  If this repeats in spite of having hundreds of thousands  of buffers, please report to support.
21:41:50 Failed to get a buffer for a new page. Retrying.  If the failure repeats, an out of disk error will be signalled.  The cause of this is having too many buffers wired down for preread, flush or group by/hash join temp space.  To correct, increase the number of buffers in the configuration file.  If this repeats in spite of having hundreds of thousands  of buffers, please report to support.
21:41:56 Write wait on column page 66408.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:56 Write wait on column page 69410.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:56 Write wait on column page 98049.  Waits should be on the index leaf page, except when col page is held for read by background write
21:41:56 Failed to get a buffer for a new page. Retrying.  If the failure repeats, an out of disk error will be signalled.  The cause of this is having too many buffers wired down for preread, flush or group by/hash join temp space.  To correct, increase the number of buffers in the configuration file.  If this repeats in spite of having hundreds of thousands  of buffers, please report to support.
21:41:56 Failed to get a buffer for a new page. Retrying.  If the failure repeats, an out of disk error will be signalled.  The cause of this is having too many buffers wired down for preread, flush or group by/hash join temp space.  To correct, increase the number of buffers in the configuration file.  If this repeats in spite of having hundreds of thousands  of buffers, please report to support.
21:42:08 Write wait on column page 45307.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 4377.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 4620.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 69486.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 98100.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 105993.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 66423.  Waits should be on the index leaf page, except when col page is held for read by background write
21:42:14 Write wait on column page 69425.  Waits should be on the index leaf page, except when col page is held for read by background write

[imitko]

please try using log_enable == 2 or 3 when loading

[jakubklimek]

OK, tried log_enable(3,1); before loading and still no luck. Either no entry in log and virtuoso just stops doing anything (0% CPU load) or the same error messages and low CPU load.

[openlink]

What dataset are you trying to load? Can you provide a link to it so we can do an in-house test on this?

[jakubklimek]

http://v7.xrg.cz/outIntegrated.n3.gz

[jakubklimek]

Tried again today with version ac7b123741c807ddfb6937a95cc970f4ba7075cf and got the same result:

15:43:55 Write wait on column page 2588.  Waits should be on the index leaf page, except when col page is held for read by background write
15:48:40 Write wait on column page 67819.  Waits should be on the index leaf page, except when col page is held for read by background write
15:48:47 Write wait on column page 107286.  Waits should be on the index leaf page, except when col page is held for read by background write
15:49:40 Write wait on column page 142116.  Waits should be on the index leaf page, except when col page is held for read by background write
15:52:13 Write wait on column page 40857.  Waits should be on the index leaf page, except when col page is held for read by background write
15:53:14 Write wait on column page 180762.  Waits should be on the index leaf page, except when col page is held for read by background write
15:53:20 Write wait on column page 180774.  Waits should be on the index leaf page, except when col page is held for read by background write
15:53:20 Write wait on column page 180777.  Waits should be on the index leaf page, except when col page is held for read by background write
15:54:02 Write wait on column page 2819.  Waits should be on the index leaf page, except when col page is held for read by background write
15:58:26 Write wait on column page 180917.  Waits should be on the index leaf page, except when col page is held for read by background write
15:58:32 Write wait on column page 220160.  Waits should be on the index leaf page, except when col page is held for read by background write
15:58:42 Write wait on column page 2820.  Waits should be on the index leaf page, except when col page is held for read by background write

[openlink]

Development is currently setting up a test with your data inhouse in order to debug this issue.

[HughWilliams]

Can you try setting DefaultIsolation = 2 in the [Parameters] section of the INI file, restart the server, and repeat the load test with an empty database?

DefaultIsolation = 2 sets the isolation level to read committed which minimises occurrence of locks during read/write operations.

As this has resolved slow load rate to the point of hanging for other users...

[jakubklimek]

OK, tried this setting. The load eventually finished, but took a long time and still there were these messages in the log:

18:16:18 **\* read-ahead of a free or out of range page dp L=28178, database not necessarily corrupted.
18:16:18 **\* read-ahead of a free or out of range page dp L=28297, database not necessarily corrupted.
18:16:18 **\* read-ahead of a free or out of range page dp L=28308, database not necessarily corrupted.
18:16:19 **\* read-ahead of a free or out of range page dp L=30771, database not necessarily corrupted.
18:16:28 **\* read-ahead of a free or out of range page dp L=35744, database not necessarily corrupted.
18:16:31 **\* read-ahead of a free or out of range page dp L=36727, database not necessarily corrupted.
18:16:33 **\* read-ahead of a free or out of range page dp L=49926, database not necessarily corrupted.
18:16:33 **\* read-ahead of a free or out of range page dp L=49943, database not necessarily corrupted.
18:16:33 **\* read-ahead of a free or out of range page dp L=49974, database not necessarily corrupted.
18:16:33 **\* read-ahead of a free or out of range page dp L=50086, database not necessarily corrupted.
18:16:33 **\* read-ahead of a free or out of range page dp L=50087, database not necessarily corrupted.
18:16:35 **\* read-ahead of a free or out of range page dp L=47274, database not necessarily corrupted.
18:16:36 **\* read-ahead of a free or out of range page dp L=36731, database not necessarily corrupted.
18:16:49 **\* read-ahead of a free or out of range page dp L=51392, database not necessarily corrupted.
18:17:37 Write wait on column page 65351.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 43529.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 128025.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 54277.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 95833.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 95861.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 117650.  Waits should be on the index leaf page, except when col page is held for read by background write
18:20:38 Write wait on column page 125994.  Waits should be on the index leaf page, except when col page is held for read by background write
18:24:13 **\* read-ahead of a free or out of range page dp L=33610, database not necessarily corrupted.
18:26:29 Write wait on column page 111906.  Waits should be on the index leaf page, except when col page is held for read by background write
18:26:29 Write wait on column page 111910.  Waits should be on the index leaf page, except when col page is held for read by background write
18:27:16 Write wait on column page 167074.  Waits should be on the index leaf page, except when col page is held for read by background write
18:27:16 Write wait on column page 167083.  Waits should be on the index leaf page, except when col page is held for read by background write
18:27:16 Write wait on column page 191505.  Waits should be on the index leaf page, except when col page is held for read by background write
18:28:37 **\* read-ahead of a free or out of range page dp L=193325, database not necessarily corrupted.
18:28:37 **\* read-ahead of a free or out of range page dp L=163296, database not necessarily corrupted.
18:28:46 **\* read-ahead of a free or out of range page dp L=194718, database not necessarily corrupted.
18:29:09 **\* read-ahead of a free or out of range page dp L=192178, database not necessarily corrupted.
18:29:45 **\* read-ahead of a free or out of range page dp L=179784, database not necessarily corrupted.
18:34:40 **\* read-ahead of a free or out of range page dp L=233732, database not necessarily corrupted.
18:35:03 **\* read-ahead of a free or out of range page dp L=193344, database not necessarily corrupted.
18:36:32 **\* read-ahead of a free or out of range page dp L=162211, database not necessarily corrupted.
18:36:32 **\* read-ahead of a free or out of range page dp L=162258, database not necessarily corrupted.
18:37:10 **\* read-ahead of a free or out of range page dp L=170316, database not necessarily corrupted.
18:37:56 **\* read-ahead of a free or out of range page dp L=194122, database not necessarily corrupted.
18:40:14 **\* read-ahead of a free or out of range page dp L=246401, database not necessarily corrupted.
18:41:00 **\* read-ahead of a free or out of range page dp L=162237, database not necessarily corrupted.
18:41:06 **\* read-ahead of a free or out of range page dp L=39719, database not necessarily corrupted.
18:41:06 **\* read-ahead of a free or out of range page dp L=39787, database not necessarily corrupted.
18:41:06 **\* read-ahead of a free or out of range page dp L=39797, database not necessarily corrupted.
18:41:27 **\* read-ahead of a free or out of range page dp L=179993, database not necessarily corrupted.
18:41:42 **\* read-ahead of a free or out of range page dp L=234038, database not necessarily corrupted.
18:41:52 **\* read-ahead of a free or out of range page dp L=158003, database not necessarily corrupted.
18:41:53 **\* read-ahead of a free or out of range page dp L=257281, database not necessarily corrupted.
18:41:53 **\* read-ahead of a free or out of range page dp L=257342, database not necessarily corrupted.
18:42:10 **\* read-ahead of a free or out of range page dp L=162519, database not necessarily corrupted.
18:42:14 **\* read-ahead of a free or out of range page dp L=218069, database not necessarily corrupted.
18:42:29 **\* read-ahead of a free or out of range page dp L=195162, database not necessarily corrupted.
18:42:37 **\* read-ahead of a free or out of range page dp L=194160, database not necessarily corrupted.
18:42:46 **\* read-ahead of a free or out of range page dp L=217918, database not necessarily corrupted.
18:43:39 **\* read-ahead of a free or out of range page dp L=192691, database not necessarily corrupted.
18:44:01 **\* read-ahead of a free or out of range page dp L=179773, database not necessarily corrupted.
18:44:41 **\* read-ahead of a free or out of range page dp L=193791, database not necessarily corrupted.
18:45:30 Write wait on column page 265892.  Waits should be on the index leaf page, except when col page is held for read by background write
18:45:37 **\* read-ahead of a free or out of range page dp L=242308, database not necessarily corrupted.
18:46:35 Write wait on column page 272897.  Waits should be on the index leaf page, except when col page is held for read by background write
18:46:39 **\* read-ahead of a free or out of range page dp L=233617, database not necessarily corrupted.
18:46:39 **\* read-ahead of a free or out of range page dp L=210687, database not necessarily corrupted.
18:46:56 **\* read-ahead of a free or out of range page dp L=253755, database not necessarily corrupted.
18:47:17 **\* read-ahead of a free or out of range page dp L=129668, database not necessarily corrupted.
18:49:50 **\* read-ahead of a free or out of range page dp L=223574, database not necessarily corrupted.
18:51:42 **\* read-ahead of a free or out of range page dp L=267150, database not necessarily corrupted.
18:52:42 **\* read-ahead of a free or out of range page dp L=258984, database not necessarily corrupted.
18:53:07 **\* read-ahead of a free or out of range page dp L=243075, database not necessarily corrupted.
18:53:14 **\* read-ahead of a free or out of range page dp L=240358, database not necessarily corrupted.
18:56:50 **\* read-ahead of a free or out of range page dp L=283995, database not necessarily corrupted.
18:58:21 Write wait on column page 212736.  Waits should be on the index leaf page, except when col page is held for read by background write
19:07:43 **\* read-ahead of a free or out of range page dp L=279042, database not necessarily corrupted.
19:07:43 **\* read-ahead of a free or out of range page dp L=259088, database not necessarily corrupted.

[imitko]

these messages can be neglected unless other issue at same time.
does virtuoso hang or crash or it is still loading, is there a cpu, disk i/o ?

[jakubklimek]

This time Virtuoso did not crash, there were peaks of both CPU activity and IO activity alternating with nothing happening in approx. 20 seconds intervals - 20 s of activity, 5-10s of nothing.

[imitko]

this issue has been fixed in develop/7 and will be part of the upcoming release

[jakubklimek]

Tried latest develop/7 commit 3fed005 and it still hangs. The log contains:

15:02:32 **\* read-ahead of a free or out of range page dp L=181768, database not necessarily corrupted.
15:02:32 **\* read-ahead of a free or out of range page dp L=181783, database not necessarily corrupted.
15:02:32 **\* read-ahead of a free or out of range page dp L=181804, database not necessarily corrupted.
15:02:32 **\* read-ahead of a free or out of range page dp L=119416, database not necessarily corrupted.
15:02:32 **\* read-ahead of a free or out of range page dp L=119515, database not necessarily corrupted.
15:02:32 **\* read-ahead of a free or out of range page dp L=119569, database not necessarily corrupted.
15:02:32 **\* read-ahead of a free or out of range page dp L=119626, database not necessarily corrupted.
15:04:31 Write wait on column page 4381.  Waits should be on the index leaf page, except when col page is held for read by background write
15:04:31 Write wait on column page 68858.  Waits should be on the index leaf page, except when col page is held for read by background write
15:04:31 Write wait on column page 168776.  Waits should be on the index leaf page, except when col page is held for read by background write
15:04:31 Write wait on column page 182551.  Waits should be on the index leaf page, except when col page is held for read by background write
15:04:31 Write wait on column page 227880.  Waits should be on the index leaf page, except when col page is held for read by background write
15:05:05 **\* read-ahead of a free or out of range page dp L=141973, database not necessarily corrupted.
15:05:42 **\* read-ahead of a free or out of range page dp L=137251, database not necessarily corrupted.
15:07:31 **\* read-ahead of a free or out of range page dp L=141528, database not necessarily corrupted.
15:09:56 **\* read-ahead of a free or out of range page dp L=124232, database not necessarily corrupted.
15:10:45 **\* read-ahead of a free or out of range page dp L=143355, database not necessarily corrupted.
15:11:09 **\* read-ahead of a free or out of range page dp L=148671, database not necessarily corrupted.
15:12:18 **\* read-ahead of a free or out of range page dp L=137050, database not necessarily corrupted.

[jakubklimek]

Still happening in 7.0.1. d9af7aa

[jakubklimek]

Has not happened for a long time, can be closed?

[andrejsab]

Having the same issue trying to load DBpedia. I'm using newest develop/7
and then fails

...
A bad parent link has been detected by looking at the would be parent and finding no leaf pointer.  The pages follow, parent first.
21:39:59 Consult your documentation on how to recover from this situation.  Doe a crush dump and restore or try the autocorrect options in the virtuoso.ini file
21:39:59 virtuoso-t() [0x8c07ea]
21:39:59 virtuoso-t() [0x8c0848]
21:39:59 virtuoso-t() [0x48d74e]
21:39:59 virtuoso-t() [0x48dbac]
...
21:39:59 virtuoso-t(sf_sql_execute_w+0x69) [0x5d4b99]
21:39:59 virtuoso-t() [0x8c4e3f]
21:39:59 virtuoso-t() [0x8cae4c]
21:39:59 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fdb739e8182]
21:39:59 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdb73202efd]
21:39:59 GPF: gate.c:840 fatal consistency check failure

[HughWilliams]

Have you attempted the crash dump and restore as suggested?

Which Dbpedia dataset(s) was being loaded when the error occurred?

[andrejsab]

I solved my problem.

There were spaces before NumberOfBuffers in my ini file, so virtuoso used default 20000 buffer size. I don't have ssd drive, just a SAS and considering I was using bulk load (so there were 4 loaders running in parallel) virtuoso couldn't handle using db from disk and loading triples from the same disk in parallel.

now that buffer size is 10000000, it works ok.

[jakubklimek]

Not happening anymore