Inconsistencies in inferencing results

[JaydencoolCC]

The same frame point cloud is inputed into the model, but the result is different each time. For example. for the same frame point cloud, two cars are detected by the model for the first time, but three cars are detected by the model for the second time.The results are very erratic. Can you explain that, please?

[JaydencoolCC]

Can anyone answer my question? Whether the network is any random layer?

[Tongjiaxun]

Do you know the reason ? I meet this question too.

[JaydencoolCC]

Do you know the reason ? I meet this question too.

I do not know the reason? I want to contact the author, but I don't know how to contact the author.Hope the author can give a reply.

[Tongjiaxun]

Maybe the reason is the sampling of the points.

[JaydencoolCC]

I've read the code of sampling method,there is no randomness. For fps, each time sampling starts from the first point(index 0) , so the points sampled by the same frame point cloud are fixed each time.

[JaydencoolCC]

Maybe the reason is the sampling of the points.

Looking forward to your new discovery.

[Tongjiaxun]

The first point of FPS is random. I have tried the model provided by Openpcdet like pointrcnn and the result is incosistent either. But I run pointpillar which donnot use FPS the result is consistent . You can try is again.

[JaydencoolCC]

The first point of FPS is random. I have tried the model provided by Openpcdet like pointrcnn and the result is incosistent either. But I run pointpillar which donnot use FPS the result is consistent . You can try is again.

I don't agree with you.I don't know whether you tried to understand the code.In my experience, each time sampling starts from the first point(index 0).The first point of FPS is no randomness. Here are my experimental results. Hope to your new discovery.
Best regards!

[JaydencoolCC]

[JaydencoolCC]

The first point of FPS is random. I have tried the model provided by Openpcdet like pointrcnn and the result is incosistent either. But I run pointpillar which donnot use FPS the result is consistent . You can try is again.

Each time you run the code, you will get the same sample points.I konw what you mean, but sampling starts from the first point(index 0) in Openpcdet. You can try to read the code written by cuda. Thanks!

[JaydencoolCC]

@yifanzhang713
I really want to konw why.
Hope you can answer this question, thanks!

[yifanzhang713]

I think the reason is about the common property of point-based detection techniques. We are committed to improve it in the next version.

Due to the inherently stochastic in our framework (different random sampling initializations in pre-processing, different initiation point in FPS, maximum number in query group, etc. ) Hence, even if we provide the trained weights, it is possible that the results that you obtain differ slightly from the ones presented in the paper.

Hope these can help you!

[JaydencoolCC]

@yifanzhang713 Thank you for your prompt reply.
I see what you mean. But my question is that when the point cloud of the same frame was tested twice(each time only input the one same sample), the detection results are different each time. For example. when input sample A, the model detects two boxes. But then the same operation detects three boxes.Here are some of my opinions and experience.
Firstly, The sampling results do not change in pre-processing(mask_points_and_boxes_outside_range) because the same sample has fixed number of points, Secondly, sampling always starts from the first point(index 0) in FPS written by openPCDet. The first point of FPS is no randomness, so we will get the same sample points. Thirdly, I think maximum number in query group don't cause the inconsistency of two inferences for the same sample.
Hope you can show more details about the question.
Best regards!

[yifanzhang713]

Yeah, I agree to what you mean. But what cannot be ignored is that the sample_points and shuffle_points operation in the data prepossessing, which may affect all the results of processes mentioned above.

[JaydencoolCC]

@yifanzhang713
As shown in the code, shuffle_points operation do not be implemented in the testing. And sample_points operation(mask_points_and_boxes_outside_range) also do not change result of sampling for the point cloud of the same frame. There is no randomness for the same sample in the code writted by you.
I hope you can give more constructive reasons and new discovery.
Loking forward to your reply!
Best regards!

[Birdylx]

@JaydencoolCC Hi, here is my discovery about the randomness. It seems like the shuffle operation in sample_points method over here:

IA-SSD/pcdet/datasets/processor/data_processor.py

Line 215 in 67db515

np.random.shuffle(choice)

if you set numpy with fixed seed, you might get a consistent results

[JaydencoolCC]

@JaydencoolCC Hi, here is my discovery about the randomness. It seems like the shuffle operation in sample_points method over here:

IA-SSD/pcdet/datasets/processor/data_processor.py

Line 215 in 67db515

np.random.shuffle(choice)

if you set numpy with fixed seed, you might get a consistent results

I think so. Thank you so much!