ObdalibQuestBenchmarksFishmark

Table of Contents 1. Introduction 2. Benchmark dataset 3. Benchmark Machine 4. Benchmark framework 5. Benchmark results Virtuoso Opensource 6.1.6 RDF Views Quest 1.7alpha

1. Introduction

Fishmark is a benchmark for comparing the performance of storage systems that expose SPARQL endpoints. Such systems include native RDF stores, Named Graph stores, systems that map relational databases into RDF, and SPARQL wrappers around other kinds of data sources. It is a derivative of Berlin SPARQL Benchmark (BSBM) and was developed in Machester University. The Fishmark benchmark is built around a real world data extracted from Fishbase website.

This document presents the results of running the Fishmark benchmark against:

 * Virtuoso Opensource 6.1.6 RDF Views
 * Quest 1.7alpha

SQL version of the benchmark was run against MySQL 5.5.

2. Benchmark dataset

We used datasets that were provided in the Fishmark web page.

3. Benchmark Machine

The benchmarks were run on a machine with the following specification:

Hardware:

 Intel Core i5 CPU 650 @ 3.20GHz × 4
 16 GB RAM
 320 GB (7200rpm) HDD
 1 TB (7200rpm) HDD

Software:

 Operating System: Ubuntu 12.04 64bit, Linux kernel 3.2.0-30-generic
 Java version "1.7.0_07"
 Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01)
 

General changes to software:

MySQL 5.5 changes to my.cnf:

 key_buffer		= 819M
 innodb_buffer_pool_size = 11192M
 max_allowed_packet	= 16M
 thread_stack		= 1M
 thread_cache_size       = 64
 thread_concurrency     = 20//
 
 sort_buffer_size = 16M
 read_buffer_size = 8M
 read_rnd_buffer_size = 16M
 bulk_insert_buffer_size = 128M
 thread_cache = 64
 

All datasets and database data files were put on the 1T disk.

4. Benchmark framework

Benchmark application has several modes for query generation and execution. In our experiment we generated 22 queries, checking that each of them had result sets (no query has 0 results). Then we ran the exact same queries for each system under test. You can find the queries in this Zipped file

5. Benchmark results

This section presents detailed results of running the Fishmark benchmark against:

#td style="background: #ffd"
*Virtuoso Opensource 6.1.6 RDF Views
*Quest 1.7alpha
*MySQL 5.5

Virtuoso Opensource 6.1.6 RDF Views

Mapping Files:

This mapping file was produced and used by our team for the benchmark:

SQL File

Quest 1.7alpha

Mapping Files:

These mapping files were produced and used by our team for the benchmark:

OWL File OBDA File

Benchmark Query results: QpS (Queries per Second)

	Virtuoso	Quest
Query 1	36	347
Query 2	-	202
Query 3	181	286
Query 4	73	245
Query 5	71	198
Query 6	272	292
Query 7	274	268
Query 8	164	175
Query 9	102*	153
Query 10	66	260
Query 11	0.2	26
Query 12	43	285
Query 13	4	27
Query 14	4	25
Query 15	13	182
Query 16	0.1	6
Query 17	2	51
Query 18	17	136
Query 19	19	176
Query 20	9	30
Query 21	3	313
Query 22	13	323

• Query 9 gives incorrect(empty) result for Virtuoso

Benchmark results: QMpH (Query Mixes per Hour)

Virtuoso	Quest
190	9,000