-
Notifications
You must be signed in to change notification settings - Fork 0
sklearn
Verdict: use ML.NET (or SharpLearning for a sklearn-like API), except text
vectorization, which is the gap filled natively by Lodestar.Text (exact
CountVectorizer/TfidfVectorizer semantics).
| sklearn need | Recommended .NET |
|---|---|
| Pipelines, training, deployment |
ML.NET (Microsoft.ML) |
| sklearn-like API (trees, ensembles) | SharpLearning |
CountVectorizer / TfidfVectorizer to the character
|
Lodestar.Text |
classification_report, roc_auc_score, the averaging modes |
Lodestar.Metrics |
dotnet add package Microsoft.MLusing Microsoft.ML;
var ml = new MLContext(seed: 0);
IDataView data = ml.Data.LoadFromTextFile<Row>("data.csv", hasHeader: true, separatorChar: ',');
var pipeline = ml.Transforms.Concatenate("Features", "f1", "f2")
.Append(ml.Regression.Trainers.Sdca(labelColumnName: "Label"));
var model = pipeline.Fit(data);-
TfidfVectorizeris non-standard. The sklearn formula (smooth_idf, per-row L2 normalization) must be reproduced to the character — ML.NET'sFeaturizeTextdoes not reproduce it. That is exactly the reason forLodestar.Text. See../equivalence.md. -
min_df/max_df, n-gram bounds: on the Lodestar side, not ML.NET.
This is the pitfall that used to read "check the definitions before comparing to sklearn", which names the trap without getting anyone out of it.
precision_score(y_true, y_pred, average=…) returns a different number, not
a different presentation, for each mode. On an imbalanced problem the modes do
not disagree slightly — they disagree by a factor of two, and every one of them
is arithmetically correct.
A worked example, taken from this repository's own oracle corpus
(binary_imbalanced: 190 samples of class 0, 10 of class 1, a classifier with
30 % label noise). Its confusion matrix is [[133, 57], [4, 6]], so the model
finds 6 of the 10 positives and calls 57 negatives positive:
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| 0 | 0.971 | 0.700 | 0.813 | 190 |
| 1 | 0.095 | 0.600 | 0.164 | 10 |
average= |
Precision | Recall | F1 | What it means |
|---|---|---|---|---|
"micro" |
0.695 | 0.695 | 0.695 | Pool every sample, then score once. On a full label set this is accuracy. |
"macro" |
0.533 | 0.650 | 0.489 | Mean of the per-class scores. The 10-sample class weighs exactly as much as the 190-sample one. |
"weighted" |
0.927 | 0.695 | 0.781 | Mean of the per-class scores weighted by support. The majority class dominates. |
"binary" |
0.095 | 0.600 | 0.164 | Not an average: class posLabel alone, ignoring the other. sklearn's default. |
Macro F1 says 0.489, weighted F1 says 0.781, for one model on one dataset. Report either without naming the mode and the reader learns nothing. The two are answering different questions: macro asks how the model does on a class picked at random, weighted asks how it does on a sample picked at random.
In C#, the mode is an enum rather than a string, so a typo is a compile error
instead of a ValueError at the end of a run. One
ConfusionMatrix.Compute
pass feeds both
F1.Score and
ClassificationReport.Compute:
using Lodestar.Metrics;
ConfusionMatrix cm = ConfusionMatrix.Compute(yTrue, yPred); // one O(samples) pass
double macro = F1.Score(cm, Averaging.Macro); // 0.489
double weighted = F1.Score(cm, Averaging.Weighted); // 0.781
double[] perClass = F1.PerClass(cm); // [0.813, 0.164]
Console.WriteLine(ClassificationReport.Compute(cm).ToText()); // what sklearn printsTwo differences from the Python spelling are deliberate. average=None becomes
F1.PerClass, a method,
because it returns one value per class rather than a scalar — an enum member
cannot change its method's return type. And
Averaging.Binary throws on a target with more than two classes instead of
guessing which class was meant. Both are recorded in
../decisions/0016.
Absent classes. A class with no predictions gives 0/0. sklearn returns 0 and
emits an UndefinedMetricWarning; a warning is easy to miss in a log and has no
natural .NET equivalent. Lodestar.Metrics makes the choice explicit —
ZeroDivision.Zero (sklearn's value), One, NaN, or Throw, which raises
UndefinedMetricException rather than letting a silent 0 flow into a report.
Every function, with its sklearn call and its deliberate divergences, is in
../equivalence.md.
dotnet add package Lodestar.MetricsGuide to be expanded as real needs arise.
- 0001-target-framework
- 0002-unicode-comparison-unit
- 0003-provenance-and-licensing
- 0004-levenshtein-myers-backlog
- 0005-hamming-jellyfish-divergence
- 0006-ratcliff-autojunk
- 0007-metaphone-scope
- 0008-italian-enza-nltk-divergence
- 0009-sample-consumes-a-local-feed
- 0010-stop-word-list-provenance
- 0011-persistence-format
- 0012-per-package-versioning
- 0013-sentencepiece-parity-scope
- 0014-precompiled-normalizer
- 0015-sonar-rules-in-the-build
- 0016-metrics-package-placement
- 0017-bpe-parity-scope
- 0018-multiclass-roc-auc-parallelism-is-opt-in
- 0019-the-net-analysers-run-in-the-build-too
- 0020-normalize-is-a-projection-not-a-parameter
- 0021-multioutput-is-a-method-not-an-enum
- 0022-added-token-matching-flags
- 0023-byte-level-decode-substitutes
- 0024-weighted-median-averages-within-scikit-learns-epsilon
- 0025-quickselect-replaces-a-full-sort-for-the-median
- 0026-r2-and-explainedvariance-split-their-undefined-cases-differently
- 0027-r2-and-explainedvariance-vectorize-only-a-single-output
- 0028-log1p-is-kahans-identity-not-math-log-1-plus-x
- 0029-balanced-accuracy-adjusted-is-left-to-ieee-754-at-the-edge
- 0030-cohen-kappa-keeps-scikit-learns-expected-matrix-orientation
- 0031-nosamplecorrect-mirrors-numpys-float64-upcast
- 0032-fbeta-substitutes-tp-predicted-and-support-algebraically
- 0033-compensated-sum-is-neumaiers-variant
- 0034-dropout-is-refused-for-want-of-a-user
- 0035-a-null-pre-split-is-removed-with-invert-not-isolated
- 0036-a-member-may-ship-without-an-oracle-if-it-says-so
- 0037-the-guards-run-before-the-commit
- 0038-the-gate-confronts-an-exception-tag-with-the-page-that-documents-it
- 0039-mutual-information-returns-zero-on-an-empty-input
- 0040-a-curve-is-a-sealed-class-per-curve
- 0041-one-sample-file-per-public-class
- 0042-phonetic-encoders-refuse-a-null-word
- 0043-the-equality-table-is-sized-to-the-pattern
- 0044-compression-belongs-to-the-caller
- 0045-a-console-call-carries-its-reason-on-the-line
- 0046-check-adr-immutable-runs-in-ci-only
- 0047-one-gate-per-kernel-not-one-per-alphabet
- 0048-the-gate-depends-on-the-kernel-and-the-alphabet
- 0049-two-gates-per-kernel-tested-where-the-width-is-known
- 0050-the-sentencepiece-bpe-lineage-stays-a-bpe-model
- benchmark_latest
- decisions
- equivalence
- matplotlib
- migration
- nightly_run
- numpy
- pandas
- performance
- pytorch
- seaborn
- sklearn
- statsmodels