Evaluating test data

[aabobakr]

While training a deep network, the framework displays the training and validation loss which is fine. After the training ends, I can test a single image or many images at once.

I am wondering, how can I evaluate the overall loss over the test dataset? , Currently, I have to export the model and test it outside digits to compute the average loss on the test data.

The only option available in digits is to upload images and see the classification result one by one on the browser. Am I missing something?

[gheinrich]

On the classification dataset creation page there is a field where you can specify a percentage of images to hold out for the test dataset. If you specify anything greater than 0 a file test.txt will be created in the dataset job folder. This file includes a list of image files and their associated ground truth. You can use this file in the Classify Many menu of the model page. Admittedly, it would be nice if DIGITS could display the network accuracy when the ground truth is provided in the image list, as is the case for test.txt. Is this what you're asking?

As an interim solution you may use the REST API to programmatically perform inference on your test.txt file. If your model job ID is 20151129-1613289ffb then you may do:

curl localhost:5000/models/images/classification/classify_many.json -XPOST -F job_id=20151129-1613289ffb -F image_list=@/path/to/test.txt

This will output a large JSON array that you can use to compute the accuracy yourself if you parse your test.txt file to retrieve the ground truth. Again, I think DIGITS should do this for you and it sounds like a valid enhancement request!

[lukeyeager]

Closing as a duplicate of #17. Please continue the discussion there if you have anything to add.

Please continue the discussion here if you have any questions about Greg's [fantastic] answer.

[aabobakr]

@gheinrich Yes that is exactly what I am asking about. But, your REST API suggestion is fantastic. I will give it a try.

[andrubrown]

@gheinrich I'm trying to implement your REST API solution but I'm running into the following error:

$ curl localhost:34448/models/images/classification/classif_many.json -XPOST -F job_id=20160110-174623-a9b8 -F image_list=@/media/DIGITS/test_01.txt
{
  "error": {
    "message": "'ImageClassificationDatasetJob' object has no attribute 'train_task'",
    "trace": [
      "Traceback (most recent call last):",
      "  File \"/usr/lib/python2.7/dist-packages/flask/app.py\", line 1475, in full_dispatch_request",
      "    rv = self.dispatch_request()",
      "  File \"/usr/lib/python2.7/dist-packages/flask/app.py\", line 1461, in dispatch_request",
      "    return self.view_functions[rule.endpoint](**req.view_args)",
      "  File \"/usr/share/digits/digits/model/images/classification/views.py\", line 330, in image_classification_model_classify_many",
      "    dataset = job.train_task().dataset",
      "AttributeError: 'ImageClassificationDatasetJob' object has no attribute 'train_task'",
      ""
    ],
    "type": "AttributeError"
  }

I've taken a look at these line in the app but I still can't find where it breaks. Any thoughts?

[gheinrich]

Hi @andrubrown the error you get suggests that you might have accidentally specified the job ID of your dataset. You need to specify the job ID of your model. If you click on your model on the DIGITS homepage then you will be taken to a URL that might resemble http://localhost:5000/models/20160114-002017-5802. In this example the job ID is 20160114-002017-5802 so your command might look like:
curl localhost:5000/models/images/classification/classify_many.json -XPOST -F job_id=20160114-002017-5802 -F image_list=@list.txt

[andrubrown]

@gheinrich you are totally right my mistake. Works like a charm. Thanks!

[szm-R]

Hi, @gheinrich is there a way to do this method (REST API) for DetectNet? I checked and it only seemed to have classification and regression.

[gheinrich]

The REST API applies to DetectNet in the same way as it does for regression networks. You should just have to replace the Job ID and path to files. Let us know if you're having difficulties with this.

[szm-R]

@gheinrich, Hello again and thank you, I tested and it worked on DetectNet giving a bunch of bounding box coordinates, now regarding the inference time, how can I get the pure inference time used to process the image? I mean without the overhead of calling to localhost and ...? when I use inference many in the GUI form I get 13 second for 46 images (282 ms each) but with this I get Time Spent: 0:00:04 for one image (which I think means 4 seconds!)

[gheinrich]

You could try to test several image lists, each with a different number of images, and do the linear regression to figure out the slope of the inference time as a function of the number of images. Check this online tool for the linear regression.

[szm-R]

hello again, Isn't there a way to run detectnet outside digits, without using curl? suppose we want to use it for a real time or near real time application then curl would be really impractical (it takes ages to infer a single image) for example in caffe there are cpp files which can be called via terminal to perform classification or feature extraction. I've seen this which uses a python script to perform classification, and have checked the code but I'm not sure how can I change it to run detectnet

[szm-R]

Hi, I was able to change classification example,py (from here) to perform detection and it's quite good with much better speed. however the speed pattern is strange! I have changed the code to load the model once and then loop over images from input directory here the first image takes about 200 ms to process, the second one takes 6!!! seconds and the rest about 50 ms (no matter which images are first or second), I really can't understand the reason!?!

another thing, as it's mentioned in classification example, the provided code only performs mean pixel subtraction, can anyone tell me how I may change this?! because in my actual training I have used Image mean (at inference time there's no option to choose mean type and I don't know what it does but there I get slightly better results than this python script for which I'm not providing any mean files because I don't understand what it means by npy format, the mean file format created when making lmdb data set is .binaryproto so what is this npy?!) and maybe I should ask this first, what is the difference between mean pixel and Image?! I've seen #169 but I don't understand it very much

[gheinrich]

what is the difference between mean pixel and Image?!

Hello, I would suggest you search in the DIGITS users list for answers to this question.

[szm-R]

Thank you, I will, but can you kindly answer my other questions as well?

[gheinrich]

Hello, if you are using DetectNet we recommend that you don't enable mean subtraction during pre-processing - see walk-through. Mean subtraction in DetectNet is done within the network in the train_transform, val_transform and deploy_transform layers for the train, val and deploy phases respectively.

Regarding the question on speed: are all your images the same size? You can check the shape of net.blobs['data'] before you call net.forward() to verify this.

[ShervinAr]

@szm2015 Hello, could you please kindly provide the classification example.py that you changed to perform detection?

[lukeyeager]

@ShervinAr: please see #1404.

[szm-R]

This is the code I used (which is the same code provided in DIGITS-master/examples/classification with some little tweaks to get detectnet output)

…

On Thu, Jan 19, 2017 at 10:52 PM, Luke Yeager ***@***.***> wrote: @ShervinAr <https://github.com/ShervinAr>: please see #1404 <#1404>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#472 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APaJX4afVjm9fuRgT09PkwgPiM_FvkFyks5rT7fdgaJpZM4G0qqp> .