tc.image_classifier.create should have a way to use extracted features

[NeerajKomuravalli]

The model building with turi create takes most of its time building features and if someone wants to retrain the model with the same data or add just some amount of data it goes through the process of extracting features again and this is a waste of time. Turi Create should have a method of extracting the features and saving the same in sFrame data format and using the same to create the model with either the extracted features or the image data itself.
I could not really find a method which addresses this anywhere in the code or documentation and I would really think this a great feature to have. Please let me know if this is already there and I am not able to find it and if it's not there I would not mind working on it and send a merge request.

[TobyRoseman]

This is a great idea. It's something we have talked about doing before, see #1916.

The main reason we haven't done it already is that we're not sure where to put this new function. The same deep features are used by both the image classifier and the image similarity toolkit.

So do we add the get_deep_features method under tc.image_classifier.? Or add it under tc.image_similarity.? We could add it in both places but I think that could be confusing.

My current thinking is that the best places to add it might be under tc.image_analysis..

If you have any opinions about where to add it, please let me know.

[NeerajKomuravalli]

I think using the same piece of code at two places will be both confusing and redundant so yes that might be a bad idea but why not create a new module where we can just extract features and use the same in both places. In addition we can make the tc.image_classifier.create accept data in both formats, sframe with image_path or sframe with extracted features and will have an option of saving features after model creation.

What are your thoughts?

[TobyRoseman]

I agree we'll need to modify the tc.image_classifier.create (and tc.image_similarity.create) to accept deep features rather than just images.

I think adding a new module (say we call it deep_features) would be a little strange since there would only be one thing inside that of that module. If we add:
tc.deep_features.get_deep_features(image_sarray, model_name).
There isn't going to be anything else under tc.deep_features.

Another option would be to add a new module and make each model have its own function, i.e. remove the model_name parameter. Then we would have three method:
tc.deep_features.Resnet(image_sarray)
tc.deep_features.Squeezenet(image_sarray)
tc.deep_features.VisionFeaturePrint_Scene(image_sarray)
This might be a cleaner solution.

Let me know which of these two options you had in mind. Or if I misunderstood and you had something else in mind.

[NeerajKomuravalli]

I like the later one better and yes It will be relatively cleaner and we can definitely do it.
I don't mind working on it and sending a PR but is there anything in specific I need to keep in mind before working on it?
Or let me know if you have someone else in your mind who will be working on it.

[NeerajKomuravalli]

any comments?

[TobyRoseman]

@NeerajKomuravalli - Sorry for the delay. I wanted to discuss this matter with my manager before responding. We think having one function under tc.image_analysis would be best.

I think it should look like tc.image_analysis.get_deep_features(images, model_name)
Where images is an SArray of images and model_name is a string.

Below is a list of things that will should to be done. There is likely more needed but this should be most of it.

Get Deep Features

• Create tc.image_analysis.get_deep_features and write the docstring
• Parameter checking for tc.image_analysis.get_deep_features. Make sure images is a non-empty SArray of images. Make sure model_name is a string with one of the three allowed values (note “VisionFeaturePrint_Scene” is only allowed on certain macOS version -- see existing code). If any of the parameter aren't right the user should get an informative error message.
• Implement tc.image_analysis.get_deep_features. You should be able to just use the existing code from either the image classifier or image similarity.

Image Classifier

• Modify code to check if the input are deep features. We already do something similar for the sound classifier. Creating a helper function here would probably be good so you can also use it for image similarity too.
• If deep features are not passed in, the code should just use tc.image_analysis.get_deep_features to get the deep features. This is so we don't have duplicated code.
• Update the docstring.
• Write image classifier unit tests for this new functionality.

Image Similarity

• Modify code to check if the input are deep features.
• If deep features are not passed in, the code should use tc.image_analysis.get_deep_features`.
• Update the docstring.
• Write unit tests for this new functionality.

Let me know if you have any questions. Feel free to put up a work in progress pull request at any point, if you would like feedback.

[NeerajKomuravalli]

@TobyRoseman Thanks for the response, that was a very detailed doc. I agree with all your point and will try to send a PR at the earliest and will let you know if I have any doubts.
Thanks!

[NeerajKomuravalli]

@TobyRoseman I was working on all the things we discussed and I realized, to extract features we need to call the _image_feature_extractor._create_feature_extractor.extract_features and it accepts parameters dataset and feature which is basically SFrame and feature column name respectively. So if we want to call this from tc.image_analysis.get_deep_features we need to send SFrame dataset in get_deep_features or we need to change the extract_feature function in _image_feature_extractor.
I personally feel like we should change the input parameters of tc.image_analysis.get_deep_features from SArray to SFrame. This will make sure we don't have to change a lot of the code and yet the same can be achieved by just passing dataset, feature, model_name and batch_size. And even when people want to save features they can directly save the the output as it has a direct association of images and the features (This is a side benefit and it's advantages are subjective and might not be as important).
Let me know what are you thoughts?

[TobyRoseman]

I think dataset should be a SArray not an SFrame. There's no reason for it to be an SFrame since we only need the image column in order to generate the deep features. It's just cleaner/simpler to have it be an SArray.

I don't think it should be much work to for it be an SArray rather than an SFrame. The current code should not need to change too much.
If dataset is now a SArray, I think you can just change:
image_sf = tc.SFrame({"image": dataset[feature]})
to:
image_sf = tc.SFrame({"image": dataset})

You're right we will need a batch_size parameter in tc.image_analysis.get_deep_features. We should also have an optional verbose flag. So let's add verbose=True and batch_size=64 to the definition of tc.image_analysis.get_deep_features.

[NeerajKomuravalli]

yes, you are right. I can just create a SFrame and get this done and there is no need for accepting SFrame as input.

[NeerajKomuravalli]

Issue turicreate:

I am sorry for the delay I have caused in working on this, I could not really get a lot of time to work on it.
I have made the required changes and I have run into some issues and I would appreciate your input input in it.
Below are the changes I made with respect to our conversation a few weeks before:

Image Analysis

1. Added get_deep_features function as discussed
2. Added support function to find if extracted feature column is present in the SFrame and if present it returns the column name of that feature otherwise raises error saying it’s not present. This function is used in image_classifier and image_similarity to find out if the feature column is of type image or extracted feature and if the feature column is not passed it will try to find the feature column with either type image or extracted_features. (This is very similar to _find_only_image_column in _internal_utils but I was not sure if I should add it there or in image_analysys but I ended up adding it in image_analysis itself)

Image classifier

1. Changed the create function to accept feature column in either Image format or extracted feature format (of the model in use).
2. If the feature column is not mentioned by the user then the algorithm first checks If there is a column with data type of extracted feature (of the model in use) and if not present it chooses the Image type column
3. Changed _extract_features to extract features only when the datatype of the feature column is Image and do nothing and pass the feature column as it is when the data type of the feature of the type extracted features (of the model in use)
4. Accept the data type array (this will be the data type of the extracted feature, if present) in functions: predict_topk, _canonize_input, predict and classify.

Image similarity

1. Changed the create function to accept feature column in either Image format or extracted feature format (of the model in use).
2. If the feature column is not mentioned by the user then the algorithm first checks If there is a column with data type of extracted feature (of the model in use) and if not present it chooses the Image type column
3. Changed _extract_features to extract features only when the datatype of the feature column is Image and do nothing and pass the feature column as it is when the data type of the feature of the type extracted features (of the model in use)
4. Accept the data type array (this will be the data type of the extracted feature, if present) in functions: query

Test image classifier and test image similarity

1. Changed the get_test_data function to also include columns with extracted features for the corresponding images (from “awesome_image” column)
2. After making the changes in create functions of both image_classification and image_similarity, they accept the feature in either Image form or array form (extracted feature column type) and the test cases are changed to include models that are trained with both types of features.

   Below are the issues I am facing in detail:

After finishing all this, I tested the same with both c++ unit test cases and python unit test cases:

1. It passed all the c++ test cases
2. It passed all the python test cases except test_export_coreml (in test_image_similarity) and test_export_coreml_predict (in test_image_classifier). It throws the error : ValueError: Cannot call 'resize' on objects that are not either an Image or SArray of Images. And as far as I know this is happening because the module predict function in coremltools accepts only Image as data type and I am not sure where to make these changes

Please let me know if the approach I have taken is correct and what do you think of it and if you want I pull put out a WIP pull request for you to check what I have done.
Please share your thoughts!

[TobyRoseman]

Hi @NeerajKomuravalli - Wow, this all sounds great. Yes, please put up a WIP pull request and I'll be happy to take a detailed look. I'm not sure what's happening with those exporting to CoreML Python unit test but I'm sure we can figure it out.

[NeerajKomuravalli]

I have put a pull request, please let me know what you think.
Thanks!

[NeerajKomuravalli]

Hi,

I have made the required changes, please take a look whenever you can.

Thanks!

[NeerajKomuravalli]

Hi,

I have made all the necessary changes and have run the test cases as well to see if everything is working as it should. All the test cases are passed successfully except:

1. test_export_coreml_predict in test_image_classifier.py: Here when the code comes to coreml_model.predict it runs successfully when the feature is awesome_images but fails when we use deep_features.
   Error thrown : "The model expects input feature resnet-50_deep_features to be an image, but the input is of type 5."
2. test_export_coreml in test_image_similarity.py : Same as above.
   Error thrown : "Error Domain=com.apple.CoreML Code=0 "Input image feature 'VisionFeaturePrint_Scene_deep_features' not found" UserInfo={NSLocalizedDescription=Input image feature 'VisionFeaturePrint_Scene_deep_features' not found}"

My assumption is predict method on coremltools.models.MLModel can be called only with image as input and doesn't work with deep features.

Please let me know your thoughts and what are the next steps.

[TobyRoseman]

The relevant pull request here has been merged and was included in our latest release. Closing.