added evaluation script for PPHumanSeg model #130
Conversation
fengyuentau
added
the
evaluation
|
@labeeb-7z Thanks for your contribution! Could you add the evaluation results of Results of accuracy evaluation with tools/eval.
*: 'quant' stands for 'quantized'. |
|
@WanliZhong here's the output I got for quantized model : |
There is too much loss of accuracy in the quantized model and we need to consider re-quantizing the model. Thank you for the evaluation script! 👍 |
|
@fengyuentau For more reference on the evaluation metric you can have a look at the metrics module of PaddleSeg which is used by the evaluation script of PaddleSeg I linked in PR description. |
There was a problem hiding this comment.
I ran the official evaluation script and the mIoU of the model should be 0.8602. I found there are many differences between your implementation and the official evaluation script. Below are some differences I found.
I am not saying you have to follow the official implementation completely but just make sure it is correct. If you cannot ensure correctness, it will save a lot of time.
|
I've moved the *_all variables outside the loop (apologies for that). As for ignoring index and adding 1, I've tried to followed the PaddleSeg implementation. And for the one_hot vectors, again I have implemented the one_hot function of PaddleSeg and followed their evaluation script. Let me know of any other changes. The updated miou is : |
| pbar.set_description( | ||
| "Evaluating {} with {} val set".format(model.name, self.name)) |
There was a problem hiding this comment.
These two lines should be placed outside the for loop since the description does not need to be updated every single iteration.
There was a problem hiding this comment.
This was referred from eval script of icdar.
I've changed it for this script, should I also change it for icdar?
There was a problem hiding this comment.
I've changed it for this script, should I also change it for icdar?
Not in this pull request.
|
@fengyuentau I've made the suggested changes, Thanks! |
* added evaluation script for PPHumanSeg * added quantized model, renamed dataset * minor spacing changes * moved _all variables outside loop and updated accuracy * removed printing for class accuracy and IoU * added 2 transforms * evaluation done on same size tensor as input size with mIoU 0.9085 * final changes * added mIoU and reference
This PR is meant to add an evaluation script for the PP Human segmentation model present in the zoo as mentioned in this issue #119 .
I referred to the val script used by PaddleSeg which can be found here and here and implemented the evaluation script for PPHumanSeg in the method other evaluation scripts on zoo are written.
This is the ouput I'm getting after running the evaluation script :
Let me know if this PR can be improved further. Thanks!