Fix #2030 : all new from createComponents()

[idodeclare]

Hello,

Harmful re-use from createComponents() seems to be the actual problem. Let's see if this fixes that issue.

—CF

[vladak]

Tried the test from #2030 (comment) , seems to do the trick.

[vladak]

In general this looks good. Could you rebase after #3041 goes in to see if the tests are clean ?

[vladak]

What about HistoryAnalyzer's use of JFlexTokenizer in its createComponents() ?

[idodeclare]

What about HistoryAnalyzer's use of JFlexTokenizer in its createComponents() ?

No harmful reuse there

[idodeclare]

Oh I need to make a slight improvement

[tarzanek]

interesting ... @vladak will you retest?

[vladak]

@tarzanek sure, TaaS :-)

I think a test case should be added.

[vladak]

For the record, what was the improvement ?

[vladak]

For the record, what was the improvement ?

I see, using method reference:

<<<<<<< HEAD                                                                    
        this.symbolTokenizerFactory = createPlainSymbolTokenizerFactory();      
=======                                                                         
        this.symbolTokenizerFactory = this::createPlainSymbolTokenizer;         
>>>>>>> c9160011b169e04969ce098c2253c452aef6b200

however I fail to see how this can work.

[tarzanek]

so the fix seems to be the use of functional supplier to provide symbol factory:

Supplier<JFlexTokenizer> symbolTokenizerFactory;

how exactly does it work in detail, I'd love to know too
(functional means thread safe locking + immutable objects where possible, but not sure what exactly the Supplier uses to stay thread safe)
@idodeclare ?

[vladak]

Anyway, the updated fix works so merging.

[idodeclare]

so the fix seems to be the use of functional supplier to provide symbol factory:

Supplier<JFlexTokenizer> symbolTokenizerFactory;

To clarify, it's a functional supplier of new tokenizers.

how exactly does it work in detail, I'd love to know too
(functional means thread safe locking + immutable objects where possible, but not sure what exactly the Supplier uses to stay thread safe)
@idodeclare ?

@tarzanek , it's thread safe because there is no sharing of data until the output of the supplier is passed along to Lucene after the function returns. No locking required. The object is not immutable but we rely on Lucene also not sharing the object among threads.