simflip

sentence-transformers' BinaryClassificationEvaluator evaluates a model with four similarity functions and, for each, carries a greater_is_better flag that it feeds into its threshold search as high_score_more_similar. The Manhattan and Euclidean score functions return a negative distance, so higher means more similar, but both are flagged greater_is_better=False. The flag therefore tells the evaluator that a lower score is more similar, the opposite of what the score functions compute, and the reported accuracy, F1, average precision, and MCC come out for the inverted classifier.

The contradiction

In sentence_transformers/util/similarity.py, both distance-based score functions return a similarity:

• pairwise_manhattan_sim (docstring "manhattan similarity, i.e. negative distance"): return -torch.sum(torch.abs(a - b), dim=-1)
• pairwise_euclidean_sim (docstring "euclidean similarity, i.e. negative distance"): return -torch.sqrt(torch.sum((a - b) ** 2, dim=-1))

In evaluation/binary_classification.py, the four metrics are registered with a direction flag:

metric	score function	returns	greater_is_better	consistent
cosine	pairwise_cos_sim	similarity	True	yes
dot	pairwise_dot_score	similarity	True	yes
manhattan	pairwise_manhattan_sim	negative distance (similarity)	False	no
euclidean	pairwise_euclidean_sim	negative distance (similarity)	False	no

All four score functions return a similarity where higher is more similar, so all four flags should be True. Cosine and dot are correct; Manhattan and Euclidean are not. The likely origin is a refactor: the evaluator once used a positive distance with the flag False (low distance is more similar, correct), and the score was later negated into a similarity without flipping the flag.

What it does

greater_is_better is passed as high_score_more_similar into find_best_acc_and_threshold and find_best_f1_and_threshold, which sort scores by reverse=high_score_more_similar before sweeping a threshold, into average_precision_score(labels, scores * (1 if greater_is_better else -1)), and into predicted_labels = (scores >= thr) if greater_is_better else (scores <= thr). With the flag False on a negative-distance similarity, every one of these runs backwards. On separable pairs:

Manhattan similarity (higher = more similar):
  greater_is_better=False (current code): reported accuracy = 49.5%
  greater_is_better=True  (correct)     : accuracy         = 100.0%

The evaluator reports a near-chance classifier for Manhattan and Euclidean. The trigger is narrow (only when the evaluated similarity functions include euclidean or manhattan; the cosine and dot defaults are correct), which is why a standing inversion goes unnoticed.

Layout

• excerpt.py: the score-function returns and the greater_is_better flags, quoted with line numbers from util/similarity.py and evaluation/binary_classification.py (Apache-2.0).
• contradiction.py: for each metric, the flag its similarity score implies versus the flag it is registered with; only Manhattan and Euclidean disagree.
• consequence.py: a faithful copy of find_best_acc_and_threshold, showing the reported accuracy invert for Manhattan under the current flag.
• test_simflip.py: the score functions are similarities, Manhattan and Euclidean contradict while cosine and dot do not, and the reported accuracy is near-chance under the current flag.

Reproduce

python contradiction.py
python consequence.py
python test_simflip.py

The direction flags and score returns are quoted from the master branch; the fix is to set greater_is_better=True for the Manhattan and Euclidean entries, matching cosine and dot.