Skip to content

SolrWayback bundle 5.4.3

Latest
Latest

Choose a tag to compare

View all tags
@thomasegense thomasegense released this 07 May 11:24
5.4.3
bdfbce8

The SolrWayback distribution is an out of the box solution for exploring archived webpages in ARC/WARC format.
Runs under Windows/Linux/MacOs.

SolrWayback bundle version 5+ now require java 11 or java 17 and no longer runs under java8. Tomcat and Solr has both been upgraded
from version 7 to version 9. SolrWayback webapp will be backwards compatible with a solr7 index. If you have a large index build under solr7 just keep the solr7 and do not use the new solr9 folder.

Download: https://github.com/netarchivesuite/solrwayback/releases/download/5.4.3/solrwayback_package_5.4.3.zip

How to install:
Unzip the bundle and read 'install guide' section in the README.md file in the root of the zip-file.
Solr must now be started with a -c (for cloud) argument:
solr-9/bin/solr start -c -m 4g

How to upgrade from a previous version:
Replace the solrwayback folder with the new folder, but keep the solr7 folder if you already have build an index and do not want to reindex.
Compare properties in solrwayback.properties and solrwaybackweb.properties with yours and add new missing properties.

Changelog:

5.4.3

Added

  • New solr fields has been added. The SolrWayback GUI has not implemented the feature yet for similarity search. The new fields can still be used for search.
  • Four new fields added to Solr: nsfw_probability, is_nsfw, is_virus and virus_description. They will only be populated if warc-safe enrichment is enabled.
  • New fields for image hashes added to Solr. P-hash and PDQ-hash (with dihedral variations) are calculated. For details see version 3.5 changes for warc-indexer: https://github.com/netarchivesuite/warc-indexer/blob/master/CHANGES.md
    This feature will make it possible to detect visual identical image that is not exact binary hash match. This include scaling,rotations, image filters etc.
    The feature is default enabled and require new solr configuration(schema.xml) is upload Solr cloud. Can be disabled in the warc-indexer config3.xml
    See the README.md file in the solr_config folder to see how to upload a new configuration to Solr cloud.

Changed

  • Navigation tracker auto poll increased from 2 seconds to 60 seconds (auto poll will be removed in a later version)
  • Switched from deprecated NoOpResponseParser to the new InputStreamResponseParser. Caching of SolrResponses is now handled by storing the response in the NetarchiveSolrClient.

Fixed

  • About text no longer reappears when manually clearing the ngram search query
  • Ngram chart is now properly cleared when the search box clear function is invoked
  • Fixed bug preventing searches from being performed when clicking on data points on ngram chart
  • Resolved bouncing/layout shift of the search box during ngram searches
  • Fixed TypeError: undefined is not an object crash when performing ngram searches
  • Fixed TypeError: undefined is not an object crash when exporting ngram chart data
  • Fixed ngram chart sizing and ensured full options are passed to the chart engine

Dependencies

  • Upgraded Solr dependencies:
    • solr@9.10.1
  • Upgraded frontend dependencies:
    • vite@^6.4.2
    • vitest@^3.2.6

Tests

  • Improve test coverage for url normalisation:
    • NormalisationStandard.canonicaliseURL
    • PathResolver.waybackAPIResolverHelper
    • UrlUtils url_norm query building and default-port handling

5.4.2

  • Fixed regression error in harvest calendar from VUE2 to VUE3 upgrade.
  • Meta refresh tag (almost works as 302 MOVED), is now also url replaced since serviceworker do not catch this. Closing ##490

5.4.1

5.4.0

  • Frontend upgraded from VUE2 to VUE3. Components still needs to rewritten to VUE3 Composition API. Thanks to Ben O'Brien (@obrienben) for the upgrade.

  • Remove port in the url_norm field. This wil fix playback for very old ARC files from IA (1995-2002). They have port:80 in the ARC HTTP-header.
    This require the index has been build with warc-indexer version 3.4.0+. If normalisation mode is set to LEGACY in solrwayback.properties the port will not be removed.
    See ukwa/webarchive-discovery#284 for more information

  • Added functionality to the DOMAIN STATS tool in the toolbox. Added a visualisation for average context length over time and a toggle button for showing the graphs combined or on individual canvases. Thanks to @VictorHarbo for this feature

  • Added start script to start both Tomcat and Solr in root folder of SolrWayback Bundle (Linux+Windows)

  • Added Navigation History tracker - a feature for documenting the browsing path of a user through the archive. Thanks to @VictorHarbo and @jorntx at the WEBCHILD-project for implementing this feature.

  • Fixed some unittest that depended on System Locale or property files in home folder. Closing #475

  • Created default text template files for "About the Collection" and "About us" instead of the default Royal Danish Library ones.

  • Bumped Jackson dependencies from 2.14.0 to 2.20.0

  • Bumped commons-fileupload from v1.3.3 -> 1.6.0

  • Improve test coverage for multiple classes:

    • JsonUtils
    • StreamingRawZipExport
    • All Memento related classes
    • WordCloudImageGenerator
    • Facade, Specifically the methods related to:
      • Smurf visualisation
      • Wordcloud generation
    • ImageUtils
    • NetarchiveSolrClient
    • SolrStreamingExportClient

See full changelog: https://github.com/netarchivesuite/solrwayback/blob/master/CHANGES.md

Contributors

jorntx, obrienben, and VictorHarbo
Assets 3
Loading