Remove parsing perf bottleneck in WordEmbeddingsTransform

[adamsitnik]

This PR improves the performance of reading large text files and affects two of our most time-consuming benchmarks.

Info:

BenchmarkDotNet=v0.11.2, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
Frequency=3507503 Hz, Resolution=285.1031 ns, Timer=TSC
.NET Core SDK=3.0.100-alpha1-009697
  [Host]     : .NET Core 2.1.5 (CoreCLR 4.6.26919.02, CoreFX 4.6.26919.02), 64bit RyuJIT
  Job-OXDQNP : .NET Core 2.1.5 (CoreCLR 4.6.26919.02, CoreFX 4.6.26919.02), 64bit RyuJIT

Before:

Method	Mean
WikiDetox_WordEmbeddings_OVAAveragedPerceptron	286.7 s
WikiDetox_WordEmbeddings_SDCAMC	184.1 s

After:

Method	Mean
WikiDetox_WordEmbeddings_OVAAveragedPerceptron	169.02 s
WikiDetox_WordEmbeddings_SDCAMC	65.32 s

Which is two minutes less to read the huge file for both benchmarks which results in a x3 boost for WikiDetox_WordEmbeddings_SDCAMC and 40% improvement for WikiDetox_WordEmbeddings_OVAAveragedPerceptron

Reading the file was a bottleneck:

I have applied all possible optimizations and parallelized this operation.

I am going to post a detailed description on Monday.

[adamsitnik]

/cc @danmosemsft

[tannergooding]

This seems like just the kind of thing the Utf8Parser in System.Memory was meant to handle....

[adamsitnik]

@shauheen done #1608

[danmoseley]

Seems all feedback was addressed? Just needs rebase?

[adamsitnik]

Just needs rebase?

And a two approvals ;)

[danmoseley]

@shauheen do you have further feedback?

[shauheen]

Branch still needs to be updated with master.

changes have been made

[eerhardt]

using System;

need a copyright header

---

Refers to: src/Microsoft.ML.Core/Utilities/LineParser.cs:1 in 0190b0c. [](commit_id = 0190b0c, deletion_comment = False)

[eerhardt]

using Microsoft.ML.Runtime.Internal.Utilities;

Add copyright header.

---

Refers to: test/Microsoft.ML.Tests/Transformers/LineParserTests.cs:1 in 0190b0c. [](commit_id = 0190b0c, deletion_comment = False)

[adamsitnik]

@shauheen @eerhardt I have addressed all issues, PTAL one more time.

@tannergooding thanks for pointing the Utf8Parser. I did not use it in this PR because I don't know if every input file is Utf8. In the next PR I will provide "fast path" for UTF8 files.

[eerhardt]

using System;

Still needs a copyright header.

---

In reply to: 439979964 [](ancestors = 439979964)

---

Refers to: src/Microsoft.ML.Core/Utilities/LineParser.cs:1 in 0190b0c. [](commit_id = 0190b0c, deletion_comment = False)