Skip to content

Setting up own SPARQL endpoint

Jump to bottom
ghrgriner edited this page Oct 7, 2026 · 13 revisions

I set up my own RDF database and SPARQL endpoint for working with Wikidata lexicographical data and for DBnary (the latter to a more limited extent). Some findings are documented here. I set this up for personal access on my laptop. Setting up a server for production use by many users will presumably require more configuration.

Computing environment

I used Apache Jena Fuseki server on a laptop running Debian 13 (Trixie). Full computer specs are here.

Setup

The steps should be as follows:

  1. Install latest OpenJDK from Debian repositories

  2. Download and install both Apache Jena Fuseki (the server) and Apache Jena. I installed the tar files into /opt/apache-jena-6.2.0/ and /opt/apache-jena-fuseki-6.2.0.

  3. Using whatever user you will run Fuseki as (not root!), set FUSEKI_BASE to a directory (e.g., ~/fuseki/run) and start the Fuseki server and then shut it down. Starting the server will initialize a FUSEKI_BASE directory. Then, move ~/fuseki/run to /opt/apache-jena-fuseki-6.2.0, which is the default location for FUSEKI_BASE in a stand-alone deployment. Any databases you create can be added to the default location in run/databases. However, note that databases can be installed in any directory and referred to using the --loc parameter by path (when starting Fuseki or tdb2.tdbquery) but automatic detection of the databases won't occur. Once the folder has been moved, unset FUSEKI_BASE (so that the default is used) or set FUSEKI_BASE to the new location.

  4. Make a folder to put the database in. I named the folder dbn for DBnary and wd for Wikidata.

    > mkdir /opt/apache-jena-fuseki-6.2.0/run/databases/dbn

  5. Run the program to load the Turtle file(s)

    # Inside directory /opt/apache-jena-fuseki-6.2.0

    > ../apache-jena-6.2.0/bin/tdb2.tdbloader --loc run/databases/dbn /path/to/my_data.ttl

  6. Create the database stats (must be named stats.opt), then move the stats file to the directory specified by --loc.

    > ../apache-jena-6.2.0/bin/tdb2.tdbstats --loc run/databases/dbn > ~/stats.opt

  7. Run the server. The last parameter can be a name of the user's choosing. It doesn't need to match anything in the Turtle files for example. I chose 'dbnary' for consistency with the public DBnary endpoint. Note that the server is read-only by default. The --update flag can be used to allow updates.

    > ./fuseki-server --loc=run/databases/dbn /dbnary

  8. Open a web browser at http://localhost:3030

Note that steps (7) - (8) above are only necessary to run the Fuseki server. It's also possible to query the database using other methods after steps (1)-(6). For example, you can pass a file with a query to with the command tdb2.tdbquery. Note that only one query at a time should be run with this command, even if the query is read-only. It's also possible to run commands against a Fuseki server using the command-line command rsparql. In this latter case, multiple queries can be run simultaneously.

General comments

  • I ran some queries through the web interface that lasted a long time. There did not appear to be any way to abort the query from the web server interface, so I used CTRL-C on the server process.

  • I found values were often blank in the 'Table' tab showing results when TSV output was selected in the web interface. However, clicking on the 'Response' tab displays the results and downloading with this tab open gives a TSV file with the values populated. It looks like it is boolean and numeric values that have this problem. I'm not sure if this was a problem with our browser or the server software.

  • I loaded the database using TDB2, but it looks like everything I did would work similarly using TDB. See here for a comparison.

  • Apache Jena was easy to get working, but in hindsight, I might have been better off choosing a different RDF store. QLever is in line to be the next engine for the Wikidata Query Service and Virtuoso Open-Source is the engine used by the public DBnary endpoint. Different database engines have different extensions to SPARQL1.1. For example, QLever supports named queries with a WITH statement (as does Blazegraph) and Wikidata example queries use this widely, while Apache Jena has no such feature. If I were running QLever locally, I could use its advanced features in local queries and share these without modification with the Wikidata community. Explain plan output and other optimization features will also vary by SPARQL engine, and it may make sense to prioritize proficiency in these for QLever.

  • Apache Jena has pretty thorough online documentation, but the documentation of its explain plan output looks incomplete. For example this page has an empty 'SSE Grammar' section with only placeholder text.

Wikidata comments

  • The paper by Fahl et al [1] was useful in presenting the server options. This paper also made it clear that setting up a local RDF data store for the whole Wikidata was not feasible without additional hardware which I have no intention of acquiring. I did not see instructions online for setting up a server only with the lexeme Turtle file, but I was able to load the Turtle files using the above steps without incident.

  • The database uses 35G on disk after loading and takes about 20 minutes to load and index (combined).

  • The Wikidata lexicographical data download is missing some properties that will likely be needed for analysis. For example, the labels for languages are not present in the download file. (TBD: did I check or just assume?) These missing properties can be extracted from the full database as Turtle files and then loaded into the local database. See here for an example script using curl.

  • After conducting various analyses using tdb2.tdbquery, I tried to start a Fuseki server for the same database. However, the server wouldn't start with the error: "Unable to check TDB lock owner, the lock file contents appear to be for a TDB2 database. Please try loading this location as a TDB2 database". I manually removed the lock files and started Fuseki. I don't recall if this was with the --tdb2 flag which shouldn't be necessary, but no records were found in the database. Some research suggested my database directory had files both for a TDB and TDB2 database. I assume what happened is I had accidentally loaded the Turtle file described in the previous bullet as a TDB file. In any case, I deleted the database and re-created it.

DBnary comments

  • The Development page on the DBnary website has advanced user instructions that include installing a server and mirroring DBnary data. However, the links were broken. (Bug report has been filed.) This is one reason I didn't bother setting up default prefixes in my server or loading the Turtle files into named graphs. I assume these details would be available in the instructions for setting up a mirror once the links are fixed.

  • The initial load went pretty fast so I decided to load for all the languages the dbnary_ontolex.ttl, dbnary_statistics.ttl, and dbnary_lime.ttl files. I had assumed I had enough disk space but received warnings about running out of space on loading and stopped the load. I think the load was on schedule to finish pretty fast, probably in 0.5-1 hour. I had 35G free to begin with and the ttl files were 13G unzipped. I then decided to only load the Italian files. These were 138M unzipped. The database folder after I loaded the files used 467M.

  • I did not configure any default prefixes in the database. Any queries I ran, I added a PREFIX statement.

  • I did not load the Turtle files into a named graph. The public DBnary endpoint has a number of named graphs, two for each Wiktionary, and then some others. The name of the graph can be specified when running tdbloader.

  • The public DBnary endpoint is more forgiving when the SPARQL does not conform to the standard. Some queries that were copied from the public endpoint did not run until I aligned them to the standard. Two points of note were: (1) aliases of count statements must be in parentheses: select ?g (COUNT(*) as ?count) .... On the public endpoint there is an example query that starts Select ?g ?p count(?p) as ?count and this runs on the public endpoint. (2) when output is counted, variables not counted should be in the 'group by' clauses. The public endpoint has a query: SELECT count(?t) , ?l WHERE {...} with no GROUP BY.

  • I ran some simple queries, and then found the issue mentioned elsewhere where not all the lexical entries were in the lime:lexicon and I switched to the public endpoint to investigate this. From there, I was able to get all the queries I wanted running on the public endpoint and stopped testing with my own endpoint. However, my own endpoint might still be useful for running Wikidata queries (on the lexeme graph only).

References

[1] Fahl W, Holzheit T, Westerinen A, et al. Getting and hosting your own copy of Wikidata. Wikidata '22: Wikidata workshop at ISWC 2022.

Clone this wiki locally