Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time
January 23, 2023 10:36
August 19, 2020 08:06
December 18, 2020 12:58
February 2, 2023 11:01
August 31, 2018 13:03
January 23, 2023 12:30
February 2, 2023 11:29


SolrWayback 4.4.0 software bundle has been released

SolrWayback bundle release 4.4.0 can be downloaded here:

About SolrWayback

SolrWayback is a web application for browsing historical harvested ARC/WARC files similar to the Internet Archive Wayback Machine. The SolrWayback uses on a Solr server with ARC/WARC files indexed using the warc-indexer.

SolrWayback comes with additional features:

  • Free text search in all resources (HTML pages, PDFs, metadata for different media types, URLs, etc.)
  • Interactive link graph (ingoing/outgoing) for domains.
  • Export of search results to a WARC file. Streaming download, no limit of size of resultset.
  • CSV text export of searc result with custom field selection.
  • Wordcloud generation for domain.
  • N-gram search visuzalisation.
  • Visualization of search result by domain.
  • Visualization of various domain statistics over time such as size, number of in and out going links.
  • Large scale export of linkgraph in Gephi format. (See
  • Image search similar to google images.
  • Image geo search by location on map using EXIF metadata information in images.
  • Search by uploading a file. (e.g., image, PDF) See if the resource has been harvested and find HTML pages using the image.
  • View all fields indexed for a resource and show warc-header for records.
  • Configure alternative playback engine to any playback engine using the playback-API such as OpenWayback or pywb.

Live demo

The National Széchényi Library of Hungary has kindly set up the following demo site for SolrWayback

See also

The Warc indexer is the indexing engine for documents in SolrWayback. It is maintained by the British Library.

Archon/Actika is a book keeping application for warc-files and can start multiple concurrent warc-indexer jobs for large scale netarchives.

SolrWayback Screenshots

Lising of search results with facets.

Image search, show only images as results.

Solrwayback showing the playback of an archived webpage with playback toolbox overlay.


Interactive domain link graph

Github like visualization of crawltimes

Search in images by gps location in images having exif location information about the location.

Generate a wordcloud for a domain

n-gram visualization of results by year, relative to the number of results that year.


The API for linking to and browsing archived webpages is the same as for Internet Archive:

Improving playback with the SolrWayback Root servlet

For moderne browsers that supports Serviceworkers, the root servlet will be obsolete. But for better playback in legacy browsers, the root servlet can improve the playbackback. See if the browser supports serviceworkers.

Installing the root-servlet will improve playback of sites that are leaking URLs. The root-servlet will catch relative leaks (same domain) even without using proxy mode. The leaks will then be redirected back into SolrWayback to the correct URL and crawltime. The root-servlet is included in the bundle install. In Tomcat it must be named ROOT.war. Link to SolrWayback root proxy: Absolute URL live-leaks (starting with http://domain...) will not be caught and can leak to the open web. Open network (F12) to see if any resources are leaking, or turn-off the internet connection to be sure there is no live leaks during playback.


  • Works on macOS/Linux/Windows.
  • JDK 8/9/10/11
  • A nice collection of ARC/WARC files or harvest your own with Heritrix, Webrecorder, Brozzler, Wget, etc.
  • Tomcat 8+ or another J2EE server for deploying the WAR-file
  • A Solr 7.X server with the index build from the Arc/Warc files using the Warc-Indexer version 3.2.0-SNAPSHOT +
  • (Optional) chrome/(chromium) installed for page previews to work. (headless chrome)

Build and usage

  • Build the application with: mvn package
  • Deploy the target/solrwayback-*.war file in a web-container
  • Copy src/test/resources/properties/ and /src/test/resources/properties/ to user/home/ folder for the J2EE server
  • Modify the property files. (default all urls http://localhost:8080)
  • Open search interface: http://localhost:8080/solrwayback

Build and test with Docker

Currently disabled as docker image has not been maintained.


Thomas Egense ( Feel free to send emails with comments or questions.

SolrWayback software bundle 4 install guide

With this download you will be able to index, search and playback web pages from your WARC files. The bundle contains Solr, the warc-indexer tool and SolrWayback installed on a Tomcat webserver. Just unzip the bundle and copy two files to your home directory and explore your WARC files.


Download :

Unzip and follow the instructions below.

Install instructions


Copy the two files properties/ and properties/ to your HOME folder (or the home-folder for Tomcat user)

Optional: For screenshot previews to work you may have to edit and change the value of the last two properties : chrome.command and screenshot.temp.imagedir. Chrome(Chromium) must has to be installed for screenshot preview images.

If there are errors when running a script, try change the permissions for the file ( etc). Linux: chmod +x


SolrWayback requires both Solr and Tomcat to be running.


  • Start tomcat: apache-tomcat-8.5.60/bin/
  • Stop tomcat: apache-tomcat-8.5.60/bin/
  • (For windows navigate to apache-tomcat-8.5.60/bin/ and type startup.bat or shutdown.bat)
  • To see Tomcat is running open: http://localhost:8080/solrwayback/


  • Start solr: solr-7.7.3/bin/solr start
  • Stop solr: solr-7.7.3/bin/solr stop -all
  • (For windows navigate to solr-7.7.3/bin/ and type solr.cmd start or solr.cmd stop -all)
  • To see Solr is running open: http://localhost:8983/solr/#/netarchivebuilder


SolrWayback uses a Solr index of WARC files to support freetext search and more complex queries.
If you do not have existing WARC files, see steps below on harvesting with wget.

The script in the indexing-folder allows for multi processing and keeps track of already indexed files, so the collection can be extended by adding more WARCs and running the script again.

Call indexing/ -h for usage and how to adjust the number of processes to use for indexing. Example usage that will index all WARC-files in the warcs1 folder.

THREADS=6 ./ warcs1/*

This will start indexing files from the warcs1 folder using 20 threads. Assigning a higher number of threads than CPU cores available will result in slower indexing. Each indexing job require 1GB ram, so this can also be a limiting factor.

You can also populate the collection and collectionid field in Solr with custom values:

THREADS=6 INDEXER_CUSTOM="--collection_id  collection1 --collection corona2021" ./ warcs1/*

You can then enable facetting on these fields in

The script keeps track of processed files by checking if a log from a previous analysis is available. The logs are stored in the status-folder (this can be changed using the STATUS_ROOT variable). To re-index a WARC file, delete the corresponding log file.

The script is not available for Windows. For that platform only a more primitive script is provided that also works for Linux/MacOs.

  1. Copy ARC/WARC files into folder: indexing/warcs1
  2. Start indexing: call indexing/ (or batch_warcs1_folder.bat for windows)

Indexing can take up to 20 minutes for 1GB warc-files. After indexing, the warc-files must stay in the same folder since SolrWayback is using them during playback etc.

Having whitespace characters in WARC file names can result in pagepreviews and playback not working on some systems. There can be up to 5 minutes delay before the indexed files are visible from search. Visit this url after index job have finished to commit them instantly: http://localhost:8983/solr/netarchivebuilder/update?commit=true
There is a similar script to show how to easily add new WARC files to the collection without indexing the old ones again.

For more information about the warc-indexer see:

Deploying under another directory context.

Renaming the solrwayback.war to collection1#solrwayback.war in the tomcat/webapps/ folder will have tomcat mapping the application from 'http://localhost:8080/solrwayback/' to 'http://localhost:8080/collection1/solrwayback/'. The requires defining the property in webapp.prefix=/collection1/solrwayback/ The 'wayback.baseurl' in also needs to be fixed to match. It will not work to try renaming a Tomcat deployment descriptor to collection1#solrwayback.xml. (hopefull fixed in a later release) The two property files must then be placed in the a /collection1 folder in the home-directory. This way multiple SolrWaybacks can be running in the same Tomcat server. None of the two property files must exist in the home-directory with this setup.

Scaling and using SolrWayback in production environment.

The stand alone Solr-server and indexing workflow using can scale up to 20000 WARC files of size 1GB. Using 20 threads indexing a collection of this size can take up to 3 weeks. This will result in an index about 1TB having 500M documents and this will require to changing the Solr memory allocation to at least 12GB. Storing the index on a SSD drive is required to reach acceptable performance for searches. For collections larger than this limit Solr Cloud is required instead of the stand alone Solr that comes with the SolrWayback Bundle. A more advanced distributed indexing flow can handled by the archon/arctika index workflow. See:

Deleting an Index

If you want to index a new collection into solr and remove the old index.

  1. Stop solr
  2. Delete the folder solr-7.7.3/server/solr/configsets/netarchivebuilder/netarchivebuilder_data/index (or rename to index1 etc, if you want to switch back later)
  3. Start solr
  4. Start the indexing script

Faster indexing

A powerful laptop can handle up to 8 simultaneous indexing processes with Solr running on the same laptop. Using an SSD for the Solr-index will speed up indexing and also improve search/playback performance drastically.


Click the question mark in the search-field to get help with the search syntax for more complex queries and using field queries.

The toolbar icon opens a menu with the available tools.


How to do your own web harvest websites (macOS/Linux only):

  • Using the wget command is an easy way to harvest web sites and create WARC files. The WARC files can then be indexed into SolrWayback.
  • Create a new folder, since there will be several files written in this folder. Navigate to that folder in a prompt.
  • Create a text file call url_list.txt with one URL per line in that folder.
  • Type the following in a prompt:
    wget --span-hosts --warc-cdx --page-requisites --warc-file=warcfilename --warc-max-size=1G -i url_list.txt

The script will harvest all pages in the url_list.txt file with all resources required for that page (images, CSS, etc.) and be written to a WARC file(s) called warcfilename.warc. The optional --span-hosts parameter will also harvest resources outside the domain of the page and can be removed

  • To perform a multi-level harvest:
    wget --span-hosts --level=1 --recursive --warc-cdx --page-requisites --warc-file=warcfilename --warc-max-size=1G -i url_list.txt
    where level=1 means "starting URLs and the first level of URLs linked from the starting URLs". This will substantially increase the size of the WARC file(s).


log4shell security alert - Patch your SolrWayback Bundle if you are using a release before 4.2.3.

SolrWayback itself does not use log4j 2+ and is not directly affected by CVE-2021-44228.

The SolrWayback bundle uses Solr 7.7.3, which is affected by log4shell. Please follow the Solr log4shell mitigation guide if the bundled Solr is used before versio 4.2.3. The quickest fix, taken from the guide, is

  • (Linux/MacOS) Edit your file to include: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"
  • (Windows) Edit your file to include: set SOLR_OPTS=%SOLR_OPTS% -Dlog4j2.formatMsgNoLookups=true

If another version of Solr is used, note that Solr >= 7.4 and < 8.11 are vulnerable. See the mitigation guide above for details.


A search interface and wayback machine for the UKWA Solr based warc-indexer framework.







No packages published