Loading Data Virtuoso

Install Virtuoso

Option 1: Standard distributions (note that it might be an old version)

# Debian-based systems
$sudo apt-get install virtuoso-opensource

Option 2: LOD2 stack packages (usually latest version) (See this page for more options)

# download the repository package 
$ wget http://stack.lod2.eu/lod2stable-repository.deb
# install the repository package 
$ sudo dpkg -i lod2stable-repository.deb
# update the repository database 
$ sudo apt-get update
# install lod2-virtuoso-opensource
$ sudo apt-get install lod2-virtuoso-opensource

Option 3: Build from source

# Install necessary packages
$ sudo apt-get install autoconf automake libtool flex bison gperf gawk m4 make openssl
# download the code
$ git clone https://github.com/openlink/virtuoso-opensource.git
$ cd virtuoso-opensource
# Build 
$ ./autogen.sh
$ ./configure --with-readline # use --help for more installation options
$ make
$ make install
# If you don't override the path in ./configure VOS is installed in /usr/local/virtuoso-opensource/

Initial Setup

By default, the Virtuoso server listens for HTTP at TCP port 8890, and for SQL data access (via iSQL, ODBC, JDBC, OLE DB, ADO.NET, etc.) at TCP port 1111. These ports may be changed by editing the virtuoso.ini file.

The virtuoso.ini file is located in /etc/virtuoso/ on a dpkg / rpm installation or in `{install-dir}/var/lib/virtuoso/db/' when build from source (unless configured otherwise).

Similarly, the database resides in /var/lib/virtuoso/db/virtuoso.db or {install-dir}/var/lib/virtuoso/db/virtuoso.db. If the server is terminated normally the virtuoso.db contains all the data and can be copied as a normal file to be opened by another server.

dpkg / rpm installation provide init.d scripts (that may not be enabled by default) and can be invoked by sudo service virtuoso-opensource {start|stop|force-stop|restart}. Compiled installations can reuse the existing script or start with {install-dir}/bin/virtuoso-t -f <virtuoso.ini file>

Loading data

Command Line Interface (more powerful / faster)

The easier way to load multiple files in Virtuoso is to use the Bulk Loader.

Put all the data you want to load in a folder and place a global.graph file containing the the graph you want to place them in (http://dbpedia.org in our case).

Open ISQL (in some Virtuoso distribution isql is called isql-vt)

# path to isql <isql port> <username> <password>
$ isql 1111 dba dba

Load the data

-- Set the files to be loaded

-- ld_dir() to load only from the specified directory, excluding any subdirectories -- 
-- arguments '<source-filename-or-directory>', '<file name pattern>', 'default graph iri in case *.graph files are missing'
SQL> ld_dir ('/path/to/files', '*.ttl.gz', 'http://dbpedia.org');
-- ld_dir_all() to load from the specified directory, including any and all subdirectories -- 
SQL> ld_dir_all ('/path/to/files', '*.ttl.gz', 'http://dbpedia.org');

-- run the loading
SQL> rdf_loader_run();

Check the original article for more options / details

Web Interface (more user friendly / slower)

• Open the Conductor interface: http://localhost:8890/conductor (port might differ in your installation)
• Navigate to Linked Data -> Quad Store Upload
• Upload a file or point to a remote URL
• Remember to put the proper Named Graph IRI (usually http://dbpedia.org)

SPARQL Endpoint access

Access to the RDF data is available at http://localhost:8890/sparql. You can set the default graph to http://dbpedia.org in the virtuoso.ini file.

Tips & Tricks

When hosting multiple databases, it's a common practice to store the databases in a separate folder such as /opt/virtuoso/7.1.0/databases/dbpedia39_1111_8890/.

Inside that folder there is a virtuoso.ini file where all paths are relative e.g.

DatabaseFile                    = virtuoso.db
ErrorLogFile                    = virtuoso.log
LockFile                        = virtuoso.lck
TransactionFile                 = virtuoso.trx
xa_persistent_file              = virtuoso.pxa
...

and starting script such as

#!/bin/bash
wd=`pwd`
iniFile="$wd/virtuoso.ini"
SLEEP=30 # seconds

while :
do
        {path-to-VOS}/bin/virtuoso-t -f -c "$iniFile"
        echo "VOS stopped: waiting for $SLEEP seconds until restart..."
        sleep $SLEEP
done

Scripts

You can find useful scripts here. (The scripts should be adapted with current configuration)

a full download & load example would be the following

$ sh dbpedia_download.sh
# Run clear if you want to delete existing data before loading
$ sh virtuoso-run-script.sh clear_graph.sql
# Disabling auto-indexing makes loading faster
$ sh virtuoso-run-script.sh auto_indexing_disable.sql
# You need to adapt the folders in the script
$ sh virtuoso-run-script.sh load_data.sql
$ sh virtuoso-run-script.sh auto_indexing_enable.sql
# If you use the FCT plugin, the following re-indexes the data
$ sh virtuoso-run-script.sh fct_plugin_reindex.sql