WIP: Python API Updates #2

Merged
merged 19 commits into from Dec 27, 2014

Projects

None yet

2 participants

@patricksnape
Contributor

As @davisking and I briefly discussed over email, I've been working on some updates to the Python API. This is currently a work in progress (WIP) but I wanted anyone to be able to take a look at what I've got so far.

Current Additions

  • Exposed test_simple_object_detector using existing images and bounding boxes. This is a very similar addition to being able to train with existing images and boxes.
  • Exposed the dlib.point type denoting an (x, y) point.
  • Exposed the full_object_detection. This contains a dlib.rectangle and a vector of dlib.point.
    • Exposed a constructor to enable training
  • Exposed the shape_predictor class
    • Is callable: shape_predictor(numpy_image, dlib.rectangle) -> dlib.full_object_detection
  • Exposed a new shape_predictor_training_options in the same vein as the simple_object_detector_training_options
  • Exposed train_shape_predictor
    • From both XML and already loaded images and dlib.full_object_detections
    • train_shape_predictor(dataset_filename, predictor_output_filename, options)
    • train_shape_predictor(images, full_object_detections, predictor_output_filename, options)
  • Exposed test_shape_predictor
    • From XML and already loaded images and dlib.full_object_detections. The scales parameter is the overload that allows normalisation of the error. This returns the mean average error as a double.
    • test_shape_predictor(dataset_filename, predictor_filename)
    • test_shape_predictor(images, full_object_detections, predictor_filename)
    • test_shape_predictor(images, full_object_detections, scales, predictor_filename)
  • Very mild refactoring of some things so I could try and use them across the predictor and the detectors.

Changes I would like to see

Here I just wanted to list a few things that I would like to change that would count as breaking/more radical changes to the current API

  • Change the current training API so that it takes keyword arguments instead of the struct. This would feel a lot more Pythonic and is totally possible as long as the default arguments are not objects.
  • Change the current training API so that saving is optional, and by default it actually just returns you the detector. This would be a lot better for cases whereby you might not want to waste time saving down and reloading back from disk. This is cleaner to implement if the output_path is a keyword argument and defaults to an empty string.

Changes @davisking may want to see

Here I'm going to be so bold as to guess things that @davisking may want to see, but are not things that I personally need.

  • Rendering of the fitted shapes using a window type viewer, similar to how detections are currently viewed.
  • A Python example showing training and fitting of a shape predictor.
  • Perhaps further updating the Python examples so that they are more Pythonic? At the moment my PyCharm flags up a lot of PEP8 problems as well.
patricksnape added some commits Dec 9, 2014
@patricksnape patricksnape First attempt at adding to the boost API
Exposes the test method, but takes in images and bounding boxes.
Seems fairly simply to extend the API
70db61c
@patricksnape patricksnape Wrap the dlib point for Python 85f0c0f
@patricksnape patricksnape Add a list of points
Also, change point x and y to be properties
c0d0adb
@patricksnape patricksnape Refactor rgb_pixel out of object detection
Also, move the vectorize template into its own header to
stop having to declare it again in vector.
68ae858
@patricksnape patricksnape Update the gitignore to ignore vim and idea 76edd49
@patricksnape patricksnape Add a new conversion header
This deals with converting python objects to dlib objects
315a2b1
@patricksnape patricksnape Add wrappers for the shape predictors
This includes the full_object_detection, a new struct in the same
vein as the simple_object_detector_training_options and of
course, the shape predictor classes themselves.

All of training, fitting and testing are wrapped.
e3aee32
@patricksnape patricksnape Python3 friendly printing in examples cc3bb49
@patricksnape patricksnape Add example of testing detector using existing data 60318cb
@patricksnape patricksnape Make shape predictor pickleable
Fix typo as well
32ad0ff
@davisking
Owner

I looked it over and it looks good so far. This will be a very nice and welcome addition to dlib. :)

Definitely include some example programs that show how to use these tools, which necessarily includes something for rendering their outputs. But that should be easy using the existing dlib functionality. And anything that makes the examples more pythonic is of course welcome so long as it doesn't make them more complicated to understand.

patricksnape added some commits Dec 11, 2014
@patricksnape patricksnape Sort out PEP8 issues in the examples af82bc4
@patricksnape patricksnape Add rendering of faces (lines)
Can either be a list of full_object_detections or a single
full_object_detection. I couldn't get the vector type to work
for full_object_detection due to a template error.
697aecb
@patricksnape patricksnape Refactor the GUI code out
I also cleaned up a bunch of code. I'm not sure why the
simple_object_detector was keeping track of the upsample amount,
since it can't even be passed as an argument to the constructor.
Therefore, I removed the simple_object_detector_py and the second
declaration of the hog object detector. I also changed the
view code to optionally take keyword args of color and added
a single view of a rectangle.

Finally, I added viewing of the shape parts.
e801bd6
@patricksnape patricksnape Properly handle turning the GUI off 5b485a6
@patricksnape patricksnape Add a save method to detectors and predictors
Also, removed the saving of the upsample which I missed from
before (since I'm not using the struct now). I understand why
the upsample was being saved, but I don't necessarily agree it
is particularly useful as you should really be upsampling on
a case by case basis at test time.
8db3f4e
@patricksnape patricksnape Update the interface to be more Pythonic
This is the biggest change so far. Now, there are two different
classes of interface. One where you pass ONLY file paths,
and one where you pass ONLY Python objects.

The file paths are maintained to keep a matching interface with
the C++ examples of dlib. So shape predicition and object
detection can be trained using the dlib XML file paths and then
serialize the detectors to disk.

Shape prediction and object detection can also be trained using
numpy arrays and in-memory objects. In this case, the predictor
and detector objects are returned from the training functions.
To facilitate serializing these objects, they now have a 'save'
method.

Tetsing follows a similar pattern, in that it can take either XML
files are or in-memory objects. I also added back the concept of
upsampling during testing to make amends for removing the
simple_object_detector_py struct.
dd19ce8
@patricksnape patricksnape Change logic for upsampling printing 30869fb
@patricksnape patricksnape Add facial landmark prediction examples for Python fbe597b
@patricksnape
Contributor

OK. I think this is pretty close to being ready. There are only two things you may have a problem with:

  1. I removed the custom simple_object_detector_py struct. Mostly because I felt this was adding a lot of unnecessary complexity. To allow for similar functionality, I allowed an optional kwarg to the testing that says how many times to upsample. In theory, knowing the upsampling over the training data isn't particularly useful, except in the case where you test the training data again. Therefore, I allowed you to pass a scale if you need one, but by default it is 0. This simplified the code a lot inside object_detection.cpp
  2. I changed the interface into two streams
    • Path based, which matches the dlib C++ API. For example, training an object detector takes the path to the XML, the output path for the SVM and the options struct. Testing takes the path to the XML and the path to the SVM.
    • Python object based. Here, all parameters are in-memory objects, and in-memory objects are returned from these methods. For example, training a detector takes the numpy array images, the dlib rectangles and the options struct -- it returns a simple_object_detector. Testing takes the images, the ground truth bounding boxes and an simple_object_detector object. To make training in this way a bit friendlier to the old style, I also added a save method to both simple_object_detector and shape_predictor.

I added visualisation and examples for the new predictor API.

@davisking
Owner

I like having the object based API as well. So definitely keep that.
However, I want to keep the upsampling because of the following use case.
Imagine someone annotates a bunch of objects that are 40x40 pixels in size,
what should happen when they train (assuming the default 80*80 sized
window)? If we don't have the automatic upsampling then they will get an
error about their boxes being too small and maybe it tells them to upsample
their images manually and try again. Moreover, they will also have to
remember to upsample their images during testing or they won't be able to
find those small objects when they go and use it.

It's also quite difficult for many people to understand these issues. I
know they seem trivial but I get unending amounts of questions on things
like this. Most people who email me are not as sharp as you are :). So
definitely keep the upsampling. It will make things easier for a lot of
users.

Cheers,
Davis

On Thu, Dec 11, 2014 at 10:07 AM, Patrick Snape notifications@github.com
wrote:

OK. I think this is pretty close to being ready. There are only two things
you may have a problem with:

  1. I removed the custom simple_object_detector_py struct. Mostly
    because I felt this was adding a lot of unnecessary complexity. To allow
    for similar functionality, I allowed an optional kwarg to the testing that
    says how many times to upsample. In theory, knowing the upsampling over the
    training data isn't particularly useful, except in the case where you test
    the training data again. Therefore, I allowed you to pass a scale if you
    need one, but by default it is 0. This simplified the code a lot inside
    object_detection.cpp
  2. I changed the interface into two streams
    • Path based, which matches the dlib C++ API. For example, training
      an object detector takes the path to the XML, the output path for the SVM
      and the options struct. Testing takes the path to the XML and the path to
      the SVM.
    • Python object based. Here, all parameters are in-memory objects,
      and in-memory objects are returned from these methods. For
      example, training a detector takes the numpy array images, the dlib
      rectangles and the options struct -- it returns a
      simple_object_detector. Testing takes the images, the ground truth
      bounding boxes and an simple_object_detector object. To make
      training in this way a bit friendlier to the old style, I also added a
      save method to both simple_object_detector and shape_predictor.


Reply to this email directly or view it on GitHub
#2 (comment).

@patricksnape patricksnape Re-add the cached object detector
A little bit hacky, but should be fine. Supports both fhog
detectors and the "cached" simple_object_detector. Also, maintains
the upscale parameter for testing
37af35b
@patricksnape
Contributor

@davisking OK! I think I've got that working. I don't suppose you have any sort of test suite I can validate against to make sure I haven't broken anything? Obviously I created my own notebook and tried different combinations, but it is easy to miss things!

@davisking
Owner

There is the test suite in dlib/test, but it only tests the C++ API. I don't have any tests that call in through the Python API. I've been kinda negligent in that regard. (sorry for the slow response, I responded via gmail but it didn't show up here for some reason)

@patricksnape
Contributor

@davisking Take a look at this: http://nbviewer.ipython.org/gist/patricksnape/ff559e7b9fe4ffed1066

This is my lazy attempt at demonstrating to you that everything works. It requires Menpo (because I am very lazy) but I think menpo is pretty simple to install. I suggest you use conda, but if you don't want to, you can pip install it (just make sure you have IPython installed into the same environment as menpo).

@davisking davisking merged commit 37af35b into davisking:master Dec 27, 2014
@davisking
Owner

This is great. I just spent a while going over it. I've merged the changes into the repository and also tweaked a few things. All the things I changed were minor, stuff like clarifications in the comments and I also changed the serialization functions you added to not append a version number to the ends of each file since it's not needed. The shape_predictor already has a version number inside it (it's the first thing serialized). I only added that version number at the end of the python object detectors so I could version the upsampling amount and any other stuff I later decided to append into the simple object detector.

I also removed the comment about not using the XML file at the end of the train_shape_predictor.py file since there isn't any example code for that. I'm fine with it the way it is, but if you want to further elaborate the example to include that then feel free to do so :)

Anyway, thanks for the patch. This is super :)

@patricksnape
Contributor

Awesome! Looked through your changes, that all seems fair. Definitely some
of my mistakes stemmed from my lack of experience with the dlib API. Let me
know if there's anything else I can help with. I'm excited to see the next
release!

On Sat, 27 Dec 2014 20:35 Davis E. King notifications@github.com wrote:

This is great. I just spent a while going over it. I've merged the changes
into the repository and also tweaked a few things. All the things I changed
were minor, stuff like clarifications in the comments and I also changed
the serialization functions you added to not append a version number to the
ends of each file since it's not needed. The shape_predictor already has a
version number inside it (it's the first thing serialized). I only added
that version number at the end of the python object detectors so I could
version the upsampling amount and any other stuff I later decided to append
into the simple object detector.

I also removed the comment about not using the XML file at the end of the
train_shape_predictor.py file since there isn't any example code for that.
I'm fine with it the way it is, but if you want to further elaborate the
example to include that then feel free to do so :)

Anyway, thanks for the patch. This is super :)


Reply to this email directly or view it on GitHub
#2 (comment).

@davisking
Owner

No worries. Thanks for the great patch :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment