Skip to content

[Code scan] NEB zero failure rates are emitted as missing metrics #435

Description

@njzjz

This issue was found by a Codex global repository scan of tracked non-test files at commit 8c93925cb10b401b2b83c738bd9263fd74474468.

Relevant code

result_df["type"] = result_df["traj"].apply(lambda x: x.split("_")[0])
result_df["error"] = result_df.apply(
lambda x: np.abs(x["pred_Ea"] - x["Ea"]), axis=1
)
type_percentages = (
result_df[result_df["error"] > ERROR_THRESHOLD].groupby("type").size()
/ result_df.groupby("type").size()
* 100
)
type_percentages = type_percentages.round(2)
# update key names
type_percentages.rename(
index={
"desorption": "ø_Desorption",
"dissociation": "ø_Dissociation",
"transfer": "ø_Transfer",
},
inplace=True,
)
results = type_percentages.to_dict()
result_df.dropna(inplace=True)
results["success_rate"] = len(result_df) / NUM_RECORDS * 100
results["MAE_Ea"] = mean_absolute_error(result_df["Ea"], result_df["pred_Ea"])
results["MAE_dE"] = mean_absolute_error(result_df["dE"], result_df["pred_dE"])

Impact

The NEB failure percentage is computed from only the rows where error > ERROR_THRESHOLD. If a reaction type has zero failures, that type is absent from the numerator groupby, so the division can produce a missing value instead of an explicit 0.0 failure rate.

A perfect category should be represented as 0.0, not as a missing metric. Otherwise downstream JSON and leaderboard plots can treat a valid result as absent.

Suggested fix

Build the denominator counts first, reindex the failure counts against those types with fill_value=0, then divide. Add a regression case where one NEB type has no failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions