Problems about the "Sanity checks" assertion code in tracking evaluation

[huangyangyi]

Hi, I met a problem when using the tracking evaluation, with this assertion triggered:

nuscenes-devkit/python-sdk/nuscenes/eval/tracking/algo.py

Lines 158 to 161 in 05d05b3

	# Sanity checks.
	unachieved_thresholds = np.sum(np.isnan(thresholds))
	duplicate_thresholds = len(thresholds) - len(np.unique(thresholds))
	assert unachieved_thresholds + duplicate_thresholds + len(thresh_metrics) == self.num_thresholds

The assertion was triggered by the case of multiple NaNs occurring in the threshold list, e.g. thresholds=[nan, nan, nan, nan, nan, nan, nan, nan, 0.14626915256182352, 0.17571373816047395, 0.1982034333050251, 0.22260631062090397, 0.24383812966558294, 0.2603936386294663, 0.2848243469541723, 0.3308456484003034, 0.3668973971960257, 0.3824963077902794, 0.39220086903106877, 0.41123709362000227, 0.4171593593699591, 0.42581389248371126, 0.4534411536795752, 0.4758617915213108, 0.5016027579412741, 0.5305963274505403, 0.5490754961967468, 0.5555436402559281, 0.5697322227060795, 0.5863827850956184, 0.5895171731710434, 0.5930080786347389, 0.5991135852776776, 0.6053177920034692, 0.6182821939388911, 0.6370792790101125,0.6611657304068407, 0.6718776701018214, 0.6763528749346733, 0.7061878126114607] and num_thresholds=40.
Not sure about the purpose of this sanity check. Is multiple NaNs occurence an invalid situation here?
P.s. Temporally I fixed this by changing L160 to:

duplicate_thresholds = len(thresholds) - len(np.unique(thresholds)) - (0 if unachieved_thresholds <= 1 else unachieved_thresholds - 1)

[whyekit-motional]

@huangyangyi the occurrence of multiple NaNs is valid and accounted for via:

nuscenes-devkit/python-sdk/nuscenes/eval/tracking/algo.py

Line 159 in 05d05b3

unachieved_thresholds = np.sum(np.isnan(thresholds))

Perhaps you might want to check the self.tracks_pred for the scene_id that is triggering this assertion