Crash when additional training data is used

[pmachapman]

This crash occurs on Serval QA when performing a build with additional data files specified in Scripture Forge:

fail: Hangfire.AutomaticRetryAttribute[0]
      Failed to process the job '69c33911e6566da664e21119': an exception occurred.
      System.NullReferenceException: Object reference not set to an instance of an object.
         at SIL.Machine.Corpora.NParallelTextCorpus.<>c.<GetRows>b__18_3(TextRow e)
         at System.Linq.Enumerable.ListSelectIterator`2.ToArray()
         at SIL.Machine.Corpora.NParallelTextCorpus.GetRows(IList`1 enumerators)+MoveNext()
         at SIL.Machine.Corpora.NParallelTextCorpus.GetRows(IEnumerable`1 textIds)+MoveNext()
         at SIL.Machine.Corpora.MergedTextCorpus.GetRows(IEnumerable`1 textIds)+MoveNext()
         at System.Linq.Enumerable.WhereIterator[TSource](IEnumerable`1 source, Func`3 predicate)+MoveNext()
         at SIL.Machine.Corpora.TextCorpusEnumerator.MoveNext()
         at SIL.Machine.Corpora.NParallelTextCorpus.GetRows(IList`1 enumerators)+MoveNext()
         at SIL.Machine.Corpora.NParallelTextCorpus.GetRows(IEnumerable`1 textIds)+MoveNext()
         at SIL.Machine.Corpora.ParallelTextCorpus.GetRows(IEnumerable`1 textIds)+MoveNext()
         at System.Linq.Enumerable.<ToArray>g__EnumerableToArray|324_0[TSource](IEnumerable`1 source)
         at SIL.ServiceToolkit.Services.ParallelCorpusPreprocessingService.PreprocessAsync(IReadOnlyList`1 corpora, Func`3 train, Func`4 inference, Boolean useKeyTerms, HashSet`1 ignoreUsfmMarkers) in /app/src/ServiceToolkit/src/SIL.ServiceToolkit/Services/ParallelCorpusPreprocessingService.cs:line 175
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 58
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 102
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 102
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 102
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 102
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 102
         at Serval.Machine.Shared.Services.TranslationPreprocessBuildJob.WriteDataFilesAsync(String buildId, IReadOnlyList`1 corpora, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/TranslationPreprocessBuildJob.cs:line 102
         at Serval.Machine.Shared.Services.PreprocessBuildJob`1.DoWorkAsync(String engineId, String buildId, IReadOnlyList`1 data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/PreprocessBuildJob.cs:line 49
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 56
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 125
         at Serval.Machine.Shared.Services.HangfireBuildJob`2.RunAsync(String engineId, String buildId, TData data, String buildOptions, CancellationToken cancellationToken) in /app/src/Machine/src/Serval.Machine.Shared/Services/HangfireBuildJob.cs:line 129
         at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*)
         at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

[Enkidu93]

I believe Mike also hit this issue in his tests. Already taking a look.

[Enkidu93]

Nevermind! That was a separate problem entirely. Since the code where this bug is has just been refactored, maybe you could re-test this after this week's QA release?

[pmachapman]

Since the code where this bug is has just been refactored, maybe you could re-test this after this week's QA release?

I have recreated it locally on master. I can look at this if you like?

[Enkidu93]

I have recreated it locally on master. I can look at this if you like?

OK, great. If you have time, that'd be great! I'm not sure exactly how the additional data files are set up in SF. (We should consider extending ServalApiTests:Nmt_Paratext() to be set up similarly). Mike's failing builds had the same message but it turned out to be just a version-mismatch problem and completely unrelated.