v4.8.0-beta00012
Pre-releaseThis release contains important bug fixes and performance enhancements.
Known Issues
-
After installation, when upgrading from versions of Lucene.Net 4.8.0-beta00009 or higher may require a restart of all instances of Visual Studio in order to reload the code analysis analyzer.
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
-
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Change Log
Breaking Changes
Lucene.Net.Facet
: RenamedLRUHashMap
>LruDictionary
. Changed all members to be virtual to allow users to provide their own LRU cache.Lucene.Net.Facet.FacetsConfig
: RemovedProcessSSDVFacetFields
from public API (as was done in Lucene), avoid lock (this)Lucene.Net.Facet.TaxonomyReader
: ChangedDoClose()
toDispose(bool)
and implemented proper dispose pattern. Avoid lock (this).Lucene.Net.Facet.WriterCache
: RenamedNameInt32CacheLRU
>NameIntCacheLru
,NameHashInt32CacheLRU
>NameHashInt32CacheLru
. Refactored to utilize a generic type internally using composition to avoid boxing/unboxing without exposing the generic closing type publicly. Added publicINameInt32CacheLru
as a common interface betweenNameIntCacheLru
andNameHashInt32CacheLru
.Lucene.Net.Facet.Taxonomy.TaxonomyReader
: RestructuredChildrenIterator
intoChildrenEnumerator
Lucene.Net.Facet.Taxonomy.CategoryPath
: ChangedFullPathLength
from a method to a propertyLucene.Net.Facet.DrillSideways
: ChangedScoreSubDocsAtOnce
from a method to a propertyLucene.Net.Facet
: RefactoredOrdAndValue
into a generic struct that can be used in bothTopOrdAndSingleQueue
andTopOrdAndInt32Queue
. Added Insert method toUtil.PriorityQueue<T>
to allow adding value types without reading the previous value for reuse.Lucene.Net.Analysis.Common.Miscellaneous.CapitalizationFilter
: Changed default behavior to use invariant culture instead of the current thread's culture to match Lucene, which seems more natural when using filters inside of analyzers. This also fits more in line with how other filters are selected.- #279 -
Lucene.Net.Analysis.Compound.Hyphenation.TernaryTree
: RenamedIterator
>Enumerator
,Keys()
>GetEnumerator()
- #279 -
Lucene.Net.Benchmarks.ByTask.Feeds.DirContentSource
: RenamedIterator
>Enumerator
Bugs
- #269 - Removed cast from
NGramTokenAnymousInnerClassHelper::IsTokenChar(int)
that was causing surrogate pairs to fail in theTestUTF8FullRange()
tests ofNGramTokenizerTest
andEdgeNGramTokenizerTest
- Fixed potential issue with
ArgumentException
s being thrown fromchar.ConvertToUtf32(string, int)
by reverting back toCodePointAt()
method inTestCharTokenizers.TestCrossPlaneNomalization()
. Lucene.Net.QueryParser.Surround.Query.ComposedQuery::MakeLuceneSubQueriesField()
: Added missing using block on enumerator- #296 - Fixed surrogate pair and culture-sensitivity issues with many analyzers.
Lucene.Net.Analysis.Common
: Fixed classes that were originally using invariant culture to do so again. J2N'sCharacter
class default is to use the current culture, which had changed from the priorCharacter
class from Lucene.Net.Support that used invariant culture. FixesTestICUFoldingFilter::TestRandomStrings()
.- Lucene.Net.ICU: Fixed ThaiWordBreaker to account for surrogate pairs. Also added locking to help with thread safety. Note that the class is still not completely thread-safe, but this patch fixes the behavior.
Lucene.Net.Spatial.Util.ShapeFieldCache
: Removed unnecessary array allocationLucene.Net.TestFramework
: FixedLineFileDocs
to read byte by byte the same way that Lucene does, except using aBufferedStream
to improve performance.Lucene.Net.TestFramework
: FixedNightlyAttribute
,WeeklyAttribute
,AwaitsFixAttribute
, andSlowAttribute
so they work at the class levelLucene.Net.Analysis.Icu.Segmentation.ICUTokenizer
: Corrected call toICU4N.UChar.IsWhiteSpace()
rather thanSystem.Char.IsWhiteSpace()
, which may return different results.Lucene.Net.TestFramework.Search.SearchEquivalenceTestBase
: Fixed exception when usingOpenBitSet.FastGet()
instead ofOpenBitSet.Get()
, since the size of the bit set is unknown.Lucene.Net.Index.DocumentsWriterFlushControl
: Fixed issue due to misbehaving locking on Monitor.TryEnter(), the code was restructured to disallow any thread that doesn't have a lock intoInternalTryCheckoutForFlush()
so the threads do note compete for a lock.- #274 -
Lucene.Net.Facet
: Fixed null reference exception inDrillSidewaysScorer
from patch in Lucene 4.10.4 https://issues.apache.org/jira/browse/LUCENE-6001 Lucene.Net.Facet.Taxonomy.WriterCache.Cl2oTaxonomyWriterCache
: Fixed locking onDispose()
method and made it safe to call dispose multiple times- Reviewed and added asserts that existed in Lucene and were missing in Lucene.NET. Effectively, this meant we were missing several test conditions that have now been put into place.
Lucene.Net.ICU
: Added locking toICUTokenizer
to only allow a single thread to manipulate theBreakIterator
at a time. This is a temporary fix to get the tests to pass until a solution is found for makingBreakIterator
threadsafe.- #332 -
Lucene.Net.Replicator
: Fixed an issue inIndexInputStream
that meant the read method could return a number larger than what was passed in for read count or what the buffer could hold, it should instead return the total number of bytes that was read into the buffer, which logically can't be bigger than the buffer it self. Lucene.Net.Tests.Index.TestIndexWithThreads::TestRollbackAndCommitWithThreads()
: Must catch and ignoreAssertionException
, as was done in LuceneLucene.Net.Search.TopScoreDocCollector
: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. FixesTestSearchAfter::TestQueries()
,TestTopDocsMerge::TestSort_1()
,TestTopDocsMerge::TestSort_2()
.Lucene.Net.Sandbox.Queries.SlowFuzzyTermsEnum
: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. FixesTestTokenLengthOpt()
.Lucene.Net.Search.FuzzyTermsEnum
: Disabled optimizations forAccept()
method on .NET Framework because of float comparison failures on x86 in Release mode. FixesTestTokenLengthOpt()
.- Fixed several references to
J2N.BitConversion
that were calling the overload that normalizes NaN when they should have been calling the raw bit conversion instead (as was done in Lucene). - #323 -
Lucene.Net.Configuration
: Removed theIConfigurationRoot
interface from theConfigurationRoot
class when targeting a version ofMicrosoft.Extensions.Configuration
less than 2.0. This will allow the end user to upgradeMicrosoft.Extensions.Configuration
seamlessly to versions 2.0 or higher. - #286 -
Lucene.Net.CodeAnalysis
: Separated CSharp and VisualBasic into different assemblies to prevent cross-language dependency issues when using analyzers
Improvements
- #261 PERFORMANCE - Fixed
FSTTester
to delete while iterating forward instead of using.ElementAt()
to iterate in reverse, which takes about 3x longer - #261 PERFORMANCE -
Lucene.Net.Facet.Taxonomy.WriterCache.NameInt32CacheLRU
: Changed fromDictionary
toConcurrentDictionary
so we can delete items from the cache while forward iterating through it. - #261 PERFORMANCE -
Lucene.Net.Index.FieldInfos
: ChangedBuilder.FieldInfo()
method toTryGetFieldInfo()
to optimize check for value - Upgraded NuGet dependency J2N to 2.0.0-beta-0009
- Upgraded NuGet dependency ICU4N to 60.1.0-alpha.352
- Upgraded NuGet dependency Morfologik.Stemming to 2.1.6-beta-0007
- #261 PERFORMANCE - Use J2N's
ICollection<T>.ToArray()
extension method that usesICollection<T>.CopyTo()
, which takes precedence over the LINQIEnumerable<T>.ToArray()
extension method. Benchmarks show about a 1/3 increase in performance. - #261 PERFORMANCE -
Lucene.Net.Support.IO.FileSupport::CreateTempFile()
: Optimized the check for invalid characters Directory.Build.props
: Disabled warnings for features that require .NET Standard 2.1- #261 PERFORMANCE - Eliminated several calls to
FirstOrDefault()
,LastOrDefault()
,Skip()
,First()
, andLast()
Lucene.Net.Support.ListExtensions
: Factored out BinarySearch in favor of implementation from J2NLucene.Net.Suggest.FreeTextSuggester
: Converted fromSubList().Clear()
toRemoveRange()
- #261 PERFORMANCE - Changed handling of
LineFileDocs
to unzip the file to a temp directory once per test run instead of using aMemoryStream
to pick a random line from the file on each test. This significantly improves performance of many of the tests. Lucene.Net.Analysis.Icu.Segmentation.ScriptIterator
: Removed static constructor and initialized static state inline- Converted all explicit
Analyzer
classes to using theAnalyzer.NewAnonymous()
method to declareAnalyzer
s inline. - #261 PERFORMANCE -
Lucene.Net.Tests.Util.TestCollectionUtil
: Optimized by using array instead of list for sorting tests - #261 PERFORMANCE -
Lucene.Net.Util
: Switched implementation ofDisposableThreadLocal
with that from RavenDB, with permission from its maintainers (https://issues.apache.org/jira/browse/LUCENENET-640?focusedCommentId=17033146&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17033146). The new implementation improves GC during several operations. Lucene.Net.TestFramework.Util.LuceneTestCase
: RemovedTaskMergeScheduler
completely from random testing- #261 PERFORMANCE - Moved scratch
BytesRef
instances outside of the loops that they were nested in so they can be reused in each iteration (as was done in Lucene) - #261 PERFORMANCE -
Lucene.Net.Facet.Taxonomy.CachedOrdinalsReader
: Refactored locking to make reads more efficient - #261 PERFORMANCE -
Lucene.Net.Facet.Taxonomy.Directory.DirectoryTaxonomyReader
: Refactored to useReaderWriterLockSlim
to make reads more efficient - #261 PERFORMANCE -
Lucene.Net.Facet.Taxonomy.Directory.DirectoryTaxonomyWriter
: Refactored locking for better efficiency - #265 -
Lucene.Net.Facet.Taxonomy.WriterCache.Cl2oTaxonomyWriterCache
: Added proper dispose pattern - #265 -
Lucene.Net.Facet.Taxonomy.WriterCache.LruTaxonomyWriterCache
: Added proper dispose pattern Lucene.Net.Facet.Taxonomy.Directory.TaxonomyIndexArrays
: Changed to useLazyInitializer
and avoid lock (this)- #261 PERFORMANCE -
Lucene.Net.Tests.Facet
: Convert int to string in the invariant culture Lucene.Net.Analysis.ICU
: Updated Segmentation files to Lucene 8.6.1 to account for the latest features of ICU- #261 PERFORMANCE -
Lucene.Net.Util.AttributeSource
: Eliminated unnecessary try catch and made more efficient by usingTryGetValue
instead ofContainsKey
followed by a lookup Lucene.Net.Util
: StreamlinedDefaultAttributeFactory
to make the get/update process of creating an attributeWeakReference
atomic- #208 - Switch to simpler LIFO thread to
ThreadState
allocator during indexing. Technically, this is something from releases/lucene-solr/4.8.1, but profiling indicates it makes a huge difference in multithreaded scenarios - SWEEP - Removed unnecessary .NET Framework references from all test projects
- Converted remaining compilation constants from target platforms to features to make it simpler to change targets. Eliminated references to
NETSTANDARD
. - Inverted logic so
FEATURE_STACKTRACE
is enabled rather than disabled when theSystem.Diagnositcs.StackTrace
class is available. Lucene.Net.Constants
: Refactored to useSystem.Runtime.InteropServices.RuntimeInformation
on .NET FrameworkLucene.Net.Expressions
: Eliminated .NET settings file and reusedJavascriptCompiler.properties
file in .NET Framework so we don't have to branch for different target platforms. Simplified reading the settings by using J2NPropertyExtensions
.- #261 PERFORMANCE -
Lucene.Net.Support.AssemblyUtils
: restructured to useIEnumerable<T>
for deferred execution - #279 -
Lucene.Net.Index.Terms
/TermsEnum
,Lucene.Net.Suggest
: Refactored iterators into enumerators. Deprecated the iterators. - #279 -
Lucene.Net.Util.FilterIterator<T>
: Converted toFilterEnumerator<T>
using a predicate passed into the constructor rather than having to subclass. DeprecatedFilterIterator<T>
. Swapped only usage inFieldFilterAtomicReader
with a LINQ query/yield return, since performance is better. - #279 -
Lucene.Net.Util.MergedIterator<T>
: Converted toMergedEnumerator<T>
and deprecatedMergedIterator<T>
- #279 -
Lucene.Net.Codecs.Memory.DirectDocValuesConsumer
: RenamedIteratorAnonymousInnerClassHelper
>Enumerator
,IterableAnonymousInnerClassHelper
>EnumerableAnonymousInnerClassHelper
New Features
- Added
DeadlockAttribute
to identify tests that are known to have threading contention issues and may deadlock during test runs - Added ability to turn asserts on in a
Release
build by using the system property "assert": "true". This is necessary to ensure all of the test conditions are being hit in all builds and to enable more thoroughCheckIndex
from the command line. Lucene.Net.TestFramework
: Added ability to turn off asserts when running tests by ignoring a few tests that require the asserts to be enabled in order to pass. This makes it possible to ensure that Lucene.NET works properly with asserts disabled. This feature didn't exist in Lucene.Lucene.Net.Search.FieldCacheDocIdSet
: Added public constructor with predicate parameter for filtering without having to create a subclass