New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitor: Locks are held for a long time and Missed Delete errors #411

Closed
hanito opened this Issue May 20, 2015 · 4 comments

Comments

Projects
None yet
4 participants
@hanito
Copy link

hanito commented May 20, 2015

I am loading :baseKB by running multiple loaders. I have 6 cores and 8 GB ram.

I am getting the following errors:

  • missed delete of name id cache ns/m.010w32b 0 (0x31aaca0 )
  • Monitor: System is under high load. Adding cluster nodes or using more replicated copies may needed
  • Monitor: Locks are held for a long time

My question is will that affect the loading process?

Bulk file:

isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
wait 
isql 111 dba dba exec="checkpoint;
isql 111 dba dba exec="commit WORK";
isql 111 dba dba exec="checkpoint;
isql 111 dba dba exec="exit;

I wonder if this error is cause because i am not setting log_enable to 3

@hanito hanito changed the title missed delete of name id Monitor: Locks are held for a long time and Missed Delete errors May 20, 2015

@HughWilliams

This comment has been minimized.

Copy link
Collaborator

HughWilliams commented May 20, 2015

For Virtuoso 7 you should not be running one rdf_loader_run() process per core, but rather it is recommended you run ( number-of-cores ÷ 2.5 ) rdf_loader_run() processes for best parallel loading of data on a multi-core machine. The RDF Bulk loader docs incorrectly stated one per core thus I have updated this.

I presume you have tuned your Virtuoso instance for the available memory on your machine or the size of the dataset being loaded ?

You should not have to set log_enable 3. The bulk loader sets it to 2 which should suffice unless you are performing graph replication, which is not supported in open source.

@HughWilliams

This comment has been minimized.

Copy link
Collaborator

HughWilliams commented May 27, 2015

What do you mean ?

Did you accidentally post this comment to git#44 instead of this issue:

Nope it hasn't I GUESS!
I am running the latest version of virtuoso 7.2.0. and I am still have the same problem.

18:04:13 Write wait on column page 225593. Waits should be on the index leaf page, except when col page is held for read by background write
18:04:13 Write wait on column page 225773. Waits should be on the index leaf page, except when col page is held for read by background write
@hanito

This comment has been minimized.

Copy link
Author

hanito commented May 27, 2015

@HughWilliams I did The following So far it's working perfectly. I am not getting CPU errors nor memory errors.

  1. Error: cpu% is low while there are large numbers of runnable threads
    Which leads to a deadlock

    Solution:

    In the virtuoso.ini file:

    ThreadCleanupInterval   = 0
    ThreadThreshold         = 0
    
  2. Memory errors

    Solution:

    MaxCheckpointRemap      = 150000 (because I have 8gb)
    CheckpointInterval      = 1
    

    This way i'll be doing a checkpoint every 1 minute. So I'll be freeing the memory and writing it to the disk.

    A checkpoint writes the current in-memory modified pages (known as dirty pages) and transaction log information from memory to disk and, also, records information about the transaction log.


Script:

#disable indexing
isql 1111 dba dba exec="DB.DBA.VT_BATCH_UPDATE ('DB.DBA.RDF_OBJ', 'ON', NULL);"
#clear list
isql 1111 dba dba exec="delete from DB.DBA.load_list;"
#load data
isql 1111 dba dba exec="ld_dir('/data/data_split', '*.nt', 'http://freebase.com');"
isql 1111 dba dba exec="set isolation='uncommitted';"
isql 1111 dba dba exec="rdf_loader_run();" 
#checkpoint
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="commit WORK;"
isql 1111 dba dba exec="checkpoint;"
#re-enable indexing
isql 1111 dba dba exec="DB.DBA.RDF_OBJ_FT_RULE_ADD (null, null, 'All');"
isql 1111 dba dba exec="DB.DBA.VT_INC_INDEX_DB_DBA_RDF_OBJ ();"

@TallTed TallTed closed this Aug 13, 2018

@qshi95

This comment has been minimized.

Copy link

qshi95 commented Dec 23, 2018

Hi, follow your steps. My virtuoso.db file have be broken, I can’t remove it. the ‘ls’ ‘mv’ ‘cp’ operation all can’t be finished. I don’t know why. Do you have any advices? Thank you! @hanito

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment