Release_1.921.9828-dev

Attached file yacy_v1.921_20181121_9828.tar.gz can be used as a YaCy system update (available as "dev 1.921/9828" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution. A MS Windows installer and a MacOS package are also provided.

Contributors by alphabetical order :

• @ivanhercaz, @luccioman, @Orbiter, @otteresk, @theel0ja

Changes included since release 1.921.9788-dev (2018-09-24) :

• Bug fixes :
  • Fixed date column rendering in Rejected URLs overview
  • yacy#241 : Fixed JDK11 compatibility issue caused by a concurrency issue in the custom classloader used for template classes
  • Fixed "TypeError: table.tBodies[0] is undefined" host browser JS error
  • yacy#246 : Fixed memory leak by proper termination of Host browser and link structure Solr query threads
  • yacy#249 : Fixed gzip decompression regression on index transfer APIs
  • yacy#248 : Fixed concurrency issue on cache used for circles rendering in images
  • Fixed detection of absolute data folder path on MS Windows
  • yacy#251 : Fixed index statistics chart refreshing by enabling soft autocommit in default Solr config
• Enhancements :
  • Fixed a small typography issue in Bookmarks.html page
  • Improved normalization and edition of blacklist path patterns having non ascii chars (issue yacy#237)
  • Snapshots :
    • Made snapshots size customizable in Solr snapshots response writer
    • Improved HTML and CSS rendering of the Solr snapshots response writer
    • Render relevant HTTP status codes on snapshot api errors
    • Added support for snapshots generation to Docker images and when running on MS Windows
  • Added new crawler attribute for finer control over Media Type detection
  • Added missing CSS width units to some HostBrowser.html styling
  • Limit the length of initially visible text in link structure graph nodes
  • Evil bots may crawl harder
  • Trace a message on incomplete proper server finish when killing process
• Build dependencies upgrades :
  • Apache Ant : from 1.10.1 to 1.10.5 in Docker alpine image flavor
• Dependencies upgrades :
  • d3js : from 3.4.4 to 5.7.0
• Documentation :
  • Added hint about JVM option useful to debug JVM crashes
  • Improved formatting of some Readme sections
  • Added details to the keystore configuration properties documentation
  • Added a link to YaCy Gitter chat room in Readme

Release_1.921.9788-dev

Attached file yacy_v1.921_20180922_9788.tar.gz can be used as a YaCy system update (available as "dev 1.921/9788" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution.

Contributors by alphabetical order :

• @edom, @luccioman, @pnhofmann, @reger24, @tangdou1

Changes included since release 1.921.9701 (2018-06-23) :

• Bug fixes :
  • Fixed unsafe conccurent access to generic SimpleDateFormat instances
  • Fixed Table_API_p.html current table page loss on row editing
  • Fixed JavaScript sorting of tables with cells containing an input field
  • Fixed removal of URLs from the delegatedURL remote crawl stack
  • Fixed exceeding max size of failreason_s Solr field on large link list
  • Fixed a NPE case on the Solr OpenSearch response writer
  • yacy#209 : fixed a NullPointerException case
  • yacy#187 : fixed crawl queue folder naming for IPv6 hosts on MS Windows filesystems
  • Fixed NullPointerException case on malformed crawl queue folder name
  • Fixed rendering of crawl queues page for URLs with raw IPV6 addresses
  • yacy#216 : Removed unnecessary '+' character URL decoding from search query and fixed encoding of '+' character on search pages links
  • yacy#217 : Added possibility to hide or show image results with rendering errors
  • Fixed logs folder checking and creation
  • Fixed rendering of the YMarks.html page
  • Fixed raw IPV6 addresses snapshots read/write on FAT32 and NTFS fs
  • yacy#227 : Use a constant default target file name for seed SCP upload method
  • yacy#228 : Hide password values from visible HTML in the Advanced Config page
• Enhancements :
  • Updated the JRE URL from 8u131 to 8u171 for the MS Windows installer
  • Added server side columns sorting on the Process Scheduler table
  • Added and updated hint messages about remote crawler status
  • Properly lock solrInstances for reboot and restoration of embedded Solr
  • Removed some unnecessary synchronization locks
  • Consistently render empty facets and facets having only entries at zero
  • Solr HTML writer :
  • Override qf and df Solr params with defaults only when they are not set
  • Optionally render the response header
  • Improved rendering of relative urls : api link and stylesheet
  • Improved support for external Solr index
  • added support for the OpenSearch and yjson response writers
  • Prevent entering empty OpenSearch URLs in ConfigHeuristics_p.html
  • Added a link to MediaWiki dumps summary in import page for convenience
  • Added a parser for XZ compressed archives.
  • Updated pdf cache clear steps consistently with current pdfbox version
  • Improved Docker image building
  • Replaced RWI ranking JQuery sliders with standard HTML range inputs
  • Search portal widget :
  • Limit search API calls rate
  • Configured local peer as default portal search widget backend
  • Fixed portal search widget results favicon url
  • Fetch result pages one by one when scrolling in portal search widget
  • Made possible to provide an absolute data root path for start script
  • Added a first version of Snapcraft configuration file for Snap packaging
  • Improved support for Java logs file pattern options
  • Updated html input field type for seed upload with file method
  • Enforced access controls to System settings pages
  • Use accessible labels in the Server Access Settings page
• Translation :
  • Large update to Chinese translations
• Data sources upgrades
  • Updated bootstrap hosts
  • Upgraded the OpenGeoDB dump URL
• Dependencies upgrades :
  • icu4j : from 61.1 to 62.1
  • commons-compress : from 1.16.1 to 1.17
  • pdfbox : from 2.0.9 to 2.0.11
  • Solr : from 6.6.3 to 6.6.5
  • Jetty : from 9.4.11.v20180605 to 9.4.12.v20180830
  • BouncyCastle : from jdk15:1.46 to jdk15on:1.60
  • Updated the JRE URL to 8u181 for the MS Windows installer
• Documentation :
  • Use standard Java annotation syntax instead of custom Javadoc tag
  • Added contributor guidelines
  • Removed deprecated warning comments about imports and Debian installer

Release_1.921.9701-dev

Attached file yacy_v1.921_20180622_9701.tar.gz can be used as a YaCy system update (available as "dev 1.921/9701" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution.
A Debian package yacy_1.921.9701_all.deb can also be found at https://bintray.com/luccioman/yacy_search_server/yacy/1.921.9701.

Contributors by alphabetical order :

• @luccioman, @Orbiter, @reger24, @SebastianoPistore

Changes included since release 1.921.9598 (2018-03-10) :

• Bug fixes :
  • Fixed error 414 (URI Too Long) when manually selecting to many RSS items
  • yacy#169 : Removed hard-coded patch for Solr 5.0 on ranking boost function
  • Fixed NPE case in Solr select servlet on external Solr only setup
  • Fixed text snippet computation : don't reject ones starting with a space char
  • Properly handle embedded Solr partial results
  • Fixed NullPointerException when omitHeader=true on external Solr server
  • Fixed unresolve-pattern case on old html title element on Connections_p.html page
  • Properly render json string attributes in the crawl profile html editor
  • Do not block whole server startup on persisted crawl profile load error
  • Made CrawlProfile loading tolerant to malformed json string attribute
  • Fixed potential infinite loop case (does not occur in current code base)
  • Fixed a NullPointerException case in the suggest api
• Enhancements :
  • Parse more audio metadata :
    • URLs, tag fields
  • Handle 'fl' (Field List) parameter when requesting internal Solr with YaCy html and exml writers
  • RSS Loader :
    • Added support for enclosures (media links)
    • Reduced log verbosity on feed items with no link
  • Crawler : - Crawler :
    • Added a new crawler document filter type on documents Media Type (MIME)
    • Added a new crawler document filter type using Solr syntax
  • Keep the selected view mode when typing a new URL in the ViewFile page
  • Table API :
    • Display the total number of matching items when filtering
    • Enable page navigation then using a search query
    • Fixed start record on the last results page
    • Adjusted api table query matching strategies
  • User help :
    • Updated link URL to IANA Media Types with https
    • Updated link to Solr Function Queries documentation page
    • Added hint to clarify results rendered dates and 'Sort by date' switch
    • Adjusted localhost/qualified account admin access informational texts
    • Added hint message inciting to check accounts settings on fresh install
    • Updated YaCy home page embedded links from http to https scheme
  • External Solr index usage :
    • Optionally allow external Solr server with self-signed certificate
    • Adjusted Solr select servlet output when using an external Solr only
    • Ensure no remote Solr is attached before "Shut Down and Re-Start Solr"
    • Ensure an embedded Solr is available for Solr dump/restore operations
    • Improved rendering of remote Solr admin URLs
    • Added controls on mode switch when attached to remote Solr instance(s)
    • Added Solr HTML writer support for responses from remote instances
    • Fixed Solr response header duplication when requesting external Solr
  • Text snippets :
    • Added analysis optional setting to compute statistics on text snippets
    • Reduced text snippet extraction processing time and memory footprint
  • Added a start script option to run as a foreground process without JMX
  • Enforced access controls on basic administration settings pages.
  • Added support for preferred https in peers communication on more operations
  • Added a UI section to configure encryption of peers communications
  • Removed remaining uses of deprecated Seed.getIP()
  • Make sure for image resource url enabled index image pixel size fields are filled if at least one of the image size fields is enabled in index
  • Corrected icons_sizes_sxt to SolrType.string in Solr Schema
  • Upgraded ConfigNetwork_p html doctype, added language attribute and cleaned up html errors
  • Prevented duplication of Solr query highlight fields parameters
  • Suggestions api :
    • Adjusted timeout management
    • Limited the rate of calls to the API when typing in search field
  • Small perf improvement : initialize threads names early when possible
  • Communication with remote peers Solr index :
    • Added missing default Solr http client connection timeout initialization
    • Trace level log message for monitoring remote solr response times
    • Removed use of deprecated ConcurrentUpdateSolrClient constructor
    • Use a common pooled http connection manager for remote solr instances
    • Apply remote solr configured timeout on getting connection from pool
    • Consistently allow gzip compression of remote Solr responses
    • Add a max acceptable limit to the size of Solr responses on p2p search
    • Use the heavy ConcurrentUpdateSolrClient only when necessary
  • Added a search page customization setting to display or not favicons
  • Added gzip compression of responses returned to user-agents accepting it
  • Made outgoing pools max total connections user configurable
  • Reuse expired connections evictors threads provided by apache and solr
  • Allow edtion of scheduled next execution dates for finer control (in Table_API_p.html page)
  • Removed no more used html transformer classes
• Initial seeds : Removed seed URLs pointing to server low.audioattack.de since it will not be updated anymore
• Translation :
  • Russian : Removed unnecessary html end line tag with invalid syntax for ConfigNetwork_p.html
  • Added Italian translation.
• Dependencies upgrades :
  • commons-codec : from 1.10 to 1.11
  • Solr : from 6.6.2 to 6.6.3
  • Jetty : from 9.4.8.v20171121 to 9.4.11.v20180605
  • pdfbox : from 2.0.8 to 2.0.9
  • icu4j : from 60.2 to 61.1
  • jsoup : from 1.11.2 to 1.11.3
  • jwat : from 1.1.0 to 1.1.1
  • slf4j : from 1.7.24 to 1.7.25
• Documentation :
  • Added some JavaDoc to the RSSMessage class.
  • Updated external Solr setup basic instructions
  • Removed Docker Cloud deploy button as service was shut down on May 21st

Developer Release_1.921.9596-dev

Attached file yacy_v1.921_20180228_9596.tar.gz can be used as a YaCy system update (available as "dev 1.921/9596" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution.

Contributors by alphabetical order :

• @JeremyRand, @luccioman, @reger24, @tangdou1

Changes included since release 1.921.9513 :

• Bug fixes :
  • Fixed internal tables exact value match iterator
  • Fixed stored URL in web cache when redirection(s) occurs.
  • Replaced improper ByteBuffer.equals() implementation by Arrays.equals()
  • Fixed NullPointerException case on Table init with relative file path.
  • Enable results suggestions (Did you Mean) even when RWI is not enabled.
  • Remove old query terms from search results suggestions links.
  • Fixed loss of other modifiers on keywords/tags search navigation links.
  • Do not clear all search modifiers when unselecting one modifier.
  • Fixed loss of "meanCount" search param when using facets or page buttons.
  • Fixed loss of search modifiers on bookmark, recommand or delete result.
  • Fixed issue yacy#158 : completed div CSS class ignore in crawl
  • Fixes issue yacy#160 : handle properly syntax exceptions with a user friendly message
  • Fixes loss of information on multiple blacklist entries editions
  • Fixes loss of entries when moving entries from one list to another
  • Fixed issue yacy#156 : introduced a new option to clean up (or not) search cache on crawl start
  • Added missing vocabulary navigator increment on results from RWI
  • Fixed NPE case when on audio resource parsed with null tag
• Enhancements :
  • Revised ReIndex and ReCrawl operations :
    • Enforced access controls
    • Made possible to customize selection query and preview results before launching a recrawl
    • Added a stop condition to the Recrawl busy thread
    • More details for recrawl job (running and terminated) reports
    • Record recrawl calls to make them schedulable
    • More comprehensive log on rejected recrawls caused by date constraint
    • Added a specific default crawl profile for the recrawl job.
  • Removed unnecessary reflection usage for workflow tasks.
  • Removed unused duplicated HTML id on header hidden field
  • Revised the RDFaParser main launcher for minimal proper operation.
  • Added an utility to generate/update XLIFF master file from lng files.
  • Added HTML microdata typed items parsing capability.
  • Adjusted and moved Language Navigator to be member of the navigatior plugin
  • Reuse the same Pattern instance when matching multiple key/values
  • Adjusted last blacklist entry example for a more accurate description
  • Improved the autotagging vocabulary editor :
    • Added option to choose field delimiter and line start in import from CSV
    • Handle escaped line breaks and separators in vocabulary import from CSV
    • Enforced access controls
    • Provide user interface messages on vocabulary creation read/write errors
  • Use https rather than http in links and queries to openstreetmap.org
  • Fixed CrawlStartExpert.html and Vocabulary_p.html HTML validation errors
  • Make StreamResponse usable in Java try-with-resources statements
  • Allow creation of vocabularies from remote CSV file URLs.
  • Added HTML5 embedded audio for results playing on supporting browsers
  • Updated the list of audio file formats supported by the audioTagParser
• Translation :
  • Updated Chinese translation for various pages
  • Updated master and French translation for the IndexReIndexMonitor_p page
• Dependencies upgrades :
  • metadata-extractor : from 2.10.1 to 2.11.0
  • httpclient : from 4.5.4 to 4.5.5
  • junit : from 4.11 to 4.12
  • commons-compress : from 1.15 to 1.16.1
  • jaudiotagger : from 2.0.3 to 2.2.5
• Tests :
  • Removed time condition on HostBalancer initialization in JUnit test.
  • Added a manual performance test for the HostBalancer.
  • Shutdown daemon threads at the end of dbtest
• Documentation :
  • Fixed some typos in the Readme.

Developer Release_1.921.9513-dev

Attached file yacy_v1.921_20180102_9513.tar.gz can be used as a YaCy system update (available as "dev 1.921/9513" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution.

Contributors by alphabetical order :

• @him2him2, @luccioman, @Orbiter, @reger24, @Scre13

Changes included since release 1.921/9460 :

• Bug fixes :
  • Fixed url mask filter generated when protocol modifier is not null
  • Fixed URL parsing with fragment and empty path
  • Fixed issue yacy#145 : use HTTP Post operation for resetting memory monitoring state.
  • Fixed issue yacy#147 : removed Java 1.8 no more necessary version checking
  • Fixed loading default settings in Performance Settings of Queues and Processes
• Enhancements :
  • Do more locale neutrale case conversions when possible, for improved compatibility with systems using Turkish language locale
  • Made possible to use https for remote search on peers with SSL enabled.
  • Removed redundant setting of timeout for remoteinstance
  • Apply tld query modifier on Solr host_s mandatory field.
  • Improved accuracy of URLs search filters : protocol, tld, host, file ext
  • Made "site:" query modifier case insensitive.
  • Improved support for internationalized domain names on "site:" modifier
  • Made "tld:" modifier case insensitive and IDN complient.
  • Removed deprecated jetty continuation class from urlproxyservlet
  • Removed use of deprecated Jetty IPAccessHandler for client filtering.
  • Added a crawl filter based on
    tag class names
  • Show/hide or show public surftip button depending on current config status
  • Started implementing optional https preference for protocol operations
  • Enable full size images preview for users with extended search rights
  • Added optional search parameter/setting to control content domain filter
  • Made TokenizedStringNavigator (used for keyword search facet) active
• Dependencies upgrades :
  • POI : from 3.16 to 3.17
  • jsoup : from 1.10.3 to 1.11.2
  • httpcore : from 4.4.6 to 4.4.8
  • jetty : from 9.4.7.v20170914 to 9.4.8.v20171121
  • commons-compress : from 1.14 to 1.15
  • icu4j : from 60.1 to 60.2
  • httpclient : from 4.5.3 to 4.5.4
  • commons-io : from 2.5 to 2.6
• Automated tests :
  • Added a basic JUnit test for the Visio parser (vsdParser)
  • Fixed SegmentTest test case time dependant occasional failures
• Documentation :
  • Upgraded URLs from HTTP ro HTTPS in README.md
  • Updated links to Java Regular Expressions documentation to version 8

Developer Release_1.921.9460-dev

Attached file yacy_v1.921_20171117_9460.tar.gz can be used as a YaCy system update (available as "dev 1.921/9460" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution.

Changes included since release 1.921/9232 :

• Bug fixes :
  • Fixed loss of index page form values on 'more options' link click.
  • Fixed blacklist returned location URL on empty parameters
• Enhancements :
  • Added an optional login link/status to the search public top nav bar.
  • Enable HTTP Digest authentication for non admin users.
  • Use accessible labels for search home page radio buttons.
  • Updated search page options descriptions (shortcuts, language and spatial restrictions)
  • Added language HTML attribute to the search home page.
  • Customized Threads with generic name for easier monitoring.
  • Removed old hard-coded holiday dates from DateDection class.
  • Updated log path in informative message of stop script.
  • Updated top-level known domains list.
• Translation :
  • Renamed Chinese & Greek lng files using ISO639-1 codes.
• Dependencies upgrades :
  • solr : from 6.6.1 to 6.6.2
  • pdfbox : from 2.0.7 to 2.0.8
  • icu4j : from 59_1 to 60.1
  • com.twelvemonkeys.imageio : from 3.3.1 to 3.3.2
• Automated tests :
  • Added a html parser charset detection unit test
  • Improved some JUnit tests isolation and resources release

Developer Release_1.921.9232-dev

Changes included since august 30 2017 :

• New Features :
  • Added a JavaScript search results resorting mode for authenticated users to improve accuracy of results sorting in realtime
  • Add a configurable limit to tags initially displayed in search results
  • Made the dates navigator max elements number user configurable.
  • Fixed a NullPointerException case on images encoding errors.
  • Fixed search result Snapshots link.
  • Consistently encode snapshot image with format requested on the API.
  • Fixed NullPointerException cases on snapshot images parsing.
• Bug fixes :
  • Mantis 766 : ensure private search features are not lost on Digest auth timeout (http://mantis.tokeek.de/view.php?id=766)
  • Prevent integer overflow cases in Chart graphs
  • Fixed array out of bounds in YJsonResponseWriter and OpensearchResponsriter
  • Fixed NullPointerException case on 'Browser' lang selection
  • Fixed RWI distance calculation on multi words search queries.
  • Mantis 772 : fixed JPEG snapshot resizing when running on OpenJDK.
• Enhancements :
  • Improved search progress bar detailed statistics accuracy
  • Improved search navigators counters accuracy
  • Updated internal ISO 639-1 language codes with latest standards.
  • Improved search for audio, video and application-specific content types
  • Improved paginations buttons refreshing strategy.
  • Improved accessibility of histograms widgets.
  • Restrict Search Result Layout modification to HTTP POST only.
  • Fixed ConfigSearchPage_p HTML validation errors.
  • Added missing accessible labels to ConfigSearchPage_p.html
  • Added links to the optional keyword tags of search result
  • Adjusted filetype: query modifier parameter to lower case
  • Added textual hints to language radio buttons labels
  • Added partial bzip2 stream parsing support
  • Updated deprecated SolrInputDocument.addField() with boost value
  • Properly close resources (even on error) on OS and ThreadDump classes.
• Translation :
  • Updated master.lng for IndexExport_p.html, ConfigSearchPage_p.html files
  • Added French translation for ConfigSearchPage_p.html
• Documentation :
  • Updated Java version information on Readme
• Dependencies upgrades :
  • solr : from 6.6.0 to 6.6.1
  • raphael.js : from 2.1.3 to 2.2.7
  • Jetty : from 9.4.6.v20170531 to 9.4.7.v20170914
• Automated tests :
  • Added bzipParser Junit test
  • Adjusted ResponseHeaderTest to succeed on slow or highly loaded CPU
  • Fixed and enabled Travis automated build
  • Fixed pdf2image conversion with imagemagick on PDFs having transparency

Developer Release_1.92.9199-dev

Attached file yacy_v1.92_20170504_9199.tar.gz can be used as a YaCy system update (available as "dev 1.92/9199 (unsigned)" in YaCy page : "Dowload System Update" - /ConfigUpdate_p.html) or as a cross-platform binary distribution.

Includes :

• Bug fixes :
  • Fixed endless loop case in wikicode processing
  • Fixed unresolved_pattern case on missing post parameter api/message.html
  • Fixed surrogates import monitoring page (mantis 740 : http://mantis.tokeek.de/view.php?id=740)
  • Fixed Index Export feature for compatibility with old indexed documents (mantis 682 :
    http://mantis.tokeek.de/view.php?id=682 and issue yacy#116)
  • Copied fix from yacy_grid_parser for wrong array type
  • Fixed blacklist Regex containing '+' characters rendering
• Enhancements :
  • Activated hosts navigator as a plugin
  • MediaWiki dump import :
    • used a text input for dump file selection
    • improved error reports on various wiki dump import prerequisites failure cases
    • added support for direct import on remote URLs
    • extended WikiCode template inclusion syntax support
    • improved monitoring
    • added the possibility to schedule import
  • Improved support for non ASCII chars in local file system URLs
  • Improved http client close time on stream processing errors
  • Added url input field as source for WarcImporter
  • Took out mailto collected in internal parsed documents
  • Fixed "Unchecked conversion" compilation warnings
  • Crawl results page : apply table lines number limit
  • Improved solr index dump format detection on import
  • Added patch to rewrite altered yacy grid schema into yacy schema
  • Handle '?' and '+' chars as valid wild cards when adding to blacklist.
  • Improved new blacklist entries URL scheme detection
• New features :
  • new facet using keywords
  • added yacy grid flatjson surrogate parser
  • added a servlet to list users and edit them in quick and easy overview
• Translation :
  • added missing text from ConfigRobotsTxt_p to master.lng
• Dependencies upgrades :
  • solr : from 5.5.3 to 5.5.4
  • jwat : from 1.0.4 to 1.0.5
• Automated tests :
  • Fixed unit test MultiProtocolURL(file) assertion for Windows path with drive letter

Developer Release_1.92.9159

Attached file yacy_v1.92_20170404_9159.tar.gz can be used as a YaCy system update or as a cross-platform binary distribution.

Includes :

• Bug fixes :
  • Removed deprecated "localMissCount" prop from yacysearchlatestinfo.json.
  • Fixed NullPointerException case occurring when local solr index is disabled in search.
  • Fixed NullPointerException in HTMLResponseWriter on missing document title.
  • Fixed a NullPointerException case possible on Index Export.
  • Corrected fromDate init value on missing param in api/timeline_p servlet
  • Corrected dublincore title property text to lowercase in htmlresponsewriter
  • Fixed htmlParser <script> text extraction on code containing expression
  • Fixed call parameter for ConnectionInfo in MonitorHandler
  • Fixed delta time calculation in PerformanceSearch_p for the 1. entry
  • Fixed ReindexSolrBusyThread new and unexpected repeat of same query with low number of found documents
  • Fixed proxyservlet response url to respect http scheme if a relative Location header is returned.
  • Fixed unresolved pattern case on /yacysearchlatestinfo.json api
  • Fixed unresolved pattern case in search results progress bar (fix for mantis 715)
  • Fixed default/httpd.mime Z file extension to lower case
  • Fixed NullPointerException case and API URL link on Solr HTML output for webgraph core.
• Default configuration :
  • Removed suche.sueddeutsche.de and faroo new from opensearch default config.
  • Fixed datacite.org heuristics base url.
  • Updated Archive-It heuristics URL.
• Enhancements :
  • Modified some threads naming for easier monitoring.
  • Improved termination of timed out remote solr requests to peers.
  • Replaced absolute redirection locations by relative ones when possible for easier reverse proxy integration.
  • Make ConfigParser a protected page, for consistent behavior of locked
  • Used java.net.URLDecoder to improve Multiprotocol non ASCII characters support
  • Added robots.txt support for heuristics federated search.
  • Reduced self generated content for text_t (visible text index field)
  • Added hint of query syntax in AccessTracker log
  • Refactored and enforced Solr mandatory fields for proper operation
  • Added extract of queries.log in form of top search word cloud (last 7 days) to AccessTracker_p.html
  • Fixed some typos in French translations
  • Deprecated obsolete custom X-...-Scheme header constant.
  • Used existing FORMAT_ANSIC Dateformatter in HeaderFramework.
  • Use pre-defined "Connection" header key, replace depreceated
  • Further avoid to set connect info properties as header value
  • Extended DCEntry.getLanguage convert to ISO639-1 codes for more languages
  • Eliminated some compiler unchecked and deprecation warnings
  • Improved pdf text extraction resource handling.
  • Implemented RequestHeader getRequestURI, getRequestURL for legacy request
  • Removed double occurance of geo:lat in rss tokens
  • Made HTTP dDgest authentication default
  • Updated translation for ConfigNetwork_p.html
  • Added switchboardconstants for server ports config keys
  • Display the local search word statistic in alphabetic order
  • Added hint text to default ServerAcess Port Settings page
  • Updated master lng file with added text in Settings_ServerAccess
  • Updated shell scripts to be compatible with HTTP Digest authentication
  • Updated dump/restore shell scripts to use the new API IndexExport_p.html
  • Extended Apache HTTP Digest Auth. for use of YaCy encoded password
  • Enforced access controls on some administrative actions.
  • Added label text for search word statistic (AccessTracker_p.html) to master.lng
  • Removed seedlist bootstrap target (not working for some longer time)
  • URL Viewer : only display the link to metadata when metadata exists
  • Set Config Portal as a private administration page.
  • Escaped HTML eventually active content from recorded API call comments.
  • Added CookieTest_p.html text to master.lng
• New features
  • New Debug/Analysis advanced settings subsection.
  • Support for HTML OpenSearch results.
  • Privacy enhancement : added settings to control referrer policy.
  • Introduced the option to configure a shutdown port.
  • Import from Warc archives
  • Export to elasticsearch
• Dependencies upgrades :
  • ant : from 1.10.0 to 1.10.1 in the Docker alpine flavor image.
  • jetty : from 9.2.19.v20160908 to 9.2.21.v20170120
  • httpclient : from 4.5.2 to 4.5.3
  • pdfbox : from 2.0.3 to 2.0.5
  • metadata-extractor : from 2.9.1 to 2.10.1
  • jsch : from 0.1.53 to 0.1.54
  • jsoup : from 1.10.1 to 1.10.2
  • icu4j : from 57_1 to 58_2
  • slf4j : from 1.7.21 to 1.7.24
  • xmpcore : from 5.1.2 to 5.1.3
• Automated tests :
  • Adjusted DefaultServlet test case to recent changes
  • Removed test case for Standard_MemoryControl which will always fail
• Documentation :
  • Added hint how to build with maven (for the first time) to readme

Developer Release_1.92.9066

Attached file yacy_v1.92_20170201_9066.tar.gz can be used as a YaCy system update or as a cross-platform binary distribution.

Includes :

• Bug fixes :
  • webstructure.xml API used with a domain name 'about' parameter (mantis 720)
  • WatchWebStructure_p.html render to include https URLs (mantis 721)
  • Display of crawler pending URLs counts in HostBrowser.html page (mantis 722)
  • unresolved pattern admin parameter in HostBrowser.html "load & index" links
  • Ensured that RWI Index.Segment IODispatcher is not blocking on shudown (mantis 723)
  • local image search pagination regression (related to issue yacy#90)
  • unresolved pattern on directory entries in HostBrowser.xml api (mantis 725)
• Enhancements :
  • Moved "Augmented Browsing" settings to "URL/Web Proxy Access Settings" in "Advanced settings" subsection
  • Used DigestURL.hosthash() where possible instead of re-implementing
  • Made sure webstructure.xml API produces valid XML.
  • Removed obsolete lastmodified calculation in WebgraphConfig
  • Added an optional parameter to webstructure.xml api.
  • Deleted outdated and unmaintained Netbeans project
  • Added ukr and pol to DCEntry.getLanguage ISO639-2 3-char language code
  • Display Index Browser links requiring auth only when authenticated.
  • Fixed Index Browser page HTML validation errors and switched to HTML5
  • Updated French translation of HostBrowser.html
  • Added dc.date.modified and dc.date.created to date parser
  • Improved Index Browser accessibility with semantically richer html tags.
  • Added LSB Tag section to the Debian package descriptor
  • Adjusted the Field-Reindex Thread to verify and update the document id
  • Added user-friendly controls over disk usage configuration settings (see issue yacy#103)
  • Added consistency check for related index fields upon load and save of index schema.
  • Refactored the DHT-Trigger section in Performance_p.html page.
• Dependencies upgrades :
  • commons-compress : from 1.12 to 1.13
  • Solr/Lucene : from 5.5.2 to 5.5.3
• Automated tests :
  • Fixed 2 failing JUnit tests
  • Added automated unit tests and perfs test for WebStructureGraph class
• Documentation :
  • Cleaned-up some Javadoc warnings
  • Detailed HostBrowser, WebStructureGraph, and webstructure classes Javadoc
  • Updated README.md - removed deprecated URL