Skip to content
This repository has been archived by the owner on Dec 21, 2023. It is now read-only.

Feature extraction #3210

Merged
merged 45 commits into from
Jul 16, 2020
Merged

Feature extraction #3210

Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
daa5b4a
Adding features column to the data and compensating for the same in m…
NeerajKomuravalli May 6, 2020
63736cf
Adding features column to the data to test with features (#3126)
NeerajKomuravalli May 6, 2020
37d5ccd
Changing import statements to make the image_analysis callable (#3126)
NeerajKomuravalli May 6, 2020
67216b5
Adding get_deep_features module to extract features and adding more t…
NeerajKomuravalli May 6, 2020
d891038
Adding a way to create model by passing either feature column or imag…
NeerajKomuravalli May 6, 2020
18d5b70
Adding a way to create model by passing either feature column or imag…
NeerajKomuravalli May 6, 2020
40ec7a5
Fixing syntax error
NeerajKomuravalli May 7, 2020
d51b193
Adding more elaborate error messages to also include Missing value er…
NeerajKomuravalli May 7, 2020
ea237b5
Adding image_classifier test methods to include testing for data with…
NeerajKomuravalli May 7, 2020
f1ea4ac
Making Image feature extraction feature more robust and indipendent
NeerajKomuravalli May 12, 2020
8e45833
Changing the _extract_features functions to first recognize the featu…
NeerajKomuravalli May 12, 2020
e14a97b
Doing small edits
NeerajKomuravalli May 12, 2020
7dc12a1
Changing the _extract_features functions to first recognize the featu…
NeerajKomuravalli May 13, 2020
8cc488d
passing model name specifically in test_select_correct_feature_column…
NeerajKomuravalli May 13, 2020
62aa5e2
Adding a way to account for extracted features instead of images as i…
NeerajKomuravalli May 14, 2020
e0a301a
Adding feature based test cases for both image as a feature and extra…
NeerajKomuravalli May 14, 2020
1aa4dc1
Using actual features using get_deep_features instead of randomly gen…
NeerajKomuravalli May 15, 2020
4bfedd9
Adding test cases for model using both image and extracted_feature as…
NeerajKomuravalli May 15, 2020
e7195d6
Removing unnecessary code
NeerajKomuravalli May 15, 2020
c874c97
Adding a way to accept input as extracted feature in quary function
NeerajKomuravalli May 15, 2020
049c1e5
Created test cases for model that is being trained with deep features
NeerajKomuravalli May 23, 2020
c02ac72
Created test cases for model that is being trained with deep features
NeerajKomuravalli May 23, 2020
26a9e61
Hiding the functions is_image_deep_feature_sarray and find_only_imag…
NeerajKomuravalli May 23, 2020
80dd166
Changing the function name find_only_image_extracted_features_column …
NeerajKomuravalli May 23, 2020
b563393
Changing the generic deep feature column name in data generation to t…
NeerajKomuravalli Jun 4, 2020
c4b4ada
Simplifing the code to detect type of sfrane column by using _find_on…
NeerajKomuravalli Jun 4, 2020
b35b2e7
Removing unnecessary try except block and correcting spelling mistakes
NeerajKomuravalli Jun 9, 2020
c4798d3
Debugging to fix _find_only_column_of_type not found error
NeerajKomuravalli Jun 16, 2020
dd0c184
Fixing all the errors where image_classifier was failing in test cases
NeerajKomuravalli Jun 16, 2020
faaf5f2
Fixing all the errors where image_similarity was failing in test cases
NeerajKomuravalli Jun 16, 2020
6d308e8
Changing test_export_coreml_predict function to also account for deep…
NeerajKomuravalli Jun 16, 2020
ab26484
Changing test_export_coreml function to also account for deep features
NeerajKomuravalli Jun 16, 2020
4ff6d46
Ignoring the coremltools predict test case when deep_features are used
NeerajKomuravalli Jun 19, 2020
0efc31b
Ignoring the coremltools predict test case when deep_features are used
NeerajKomuravalli Jun 19, 2020
ca49ab2
Adding docstring for get_deep_features
NeerajKomuravalli Jun 19, 2020
b1c27ba
Merge branch 'master' into feature_extraction
NeerajKomuravalli Jun 19, 2020
3b4455d
Removing unnecessary code and makeing recommended changes
NeerajKomuravalli Jun 22, 2020
61ac0c2
Ignoring getting deep features when mac version is less than 10.14 or…
NeerajKomuravalli Jun 25, 2020
fd0cbd5
Removing coremltools import line to adhere to the turicreate testing …
NeerajKomuravalli Jul 1, 2020
199e1ce
Shifting deep feature building in from get_test_data to setUpClass (n…
NeerajKomuravalli Jul 9, 2020
3b97d6f
Shifting deep feature building in from get_test_data to setUpClass (f…
NeerajKomuravalli Jul 9, 2020
6a29bc5
Shifting deep feature building in from get_test_data to setUpClass
NeerajKomuravalli Jul 9, 2020
0c5ff0c
Removing unnecessary comments
NeerajKomuravalli Jul 9, 2020
df86dbd
Changed the logic that decides when to extract deepFeatures and chang…
NeerajKomuravalli Jul 10, 2020
fd01bcd
Using the feature fed to unit test classes based on their names to ma…
NeerajKomuravalli Jul 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 86 additions & 25 deletions src/python/turicreate/test/test_image_classifier.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
_raise_error_if_not_sframe,
_raise_error_if_not_sarray,
)
from turicreate.toolkits.image_analysis.image_analysis import MODEL_TO_FEATURE_SIZE_MAPPING, get_deep_features

from . import util as test_util

Expand Down Expand Up @@ -73,7 +74,10 @@ def get_test_data():
images.append(tc_image)

labels = ["white"] * 5 + ["black"] * 5
return tc.SFrame({"awesome_image": images, "awesome_label": labels})
data_dict = {"awesome_image": images, "awesome_label": labels}
data = tc.SFrame(data_dict)

return data
TobyRoseman marked this conversation as resolved.
Show resolved Hide resolved


data = get_test_data()
Expand All @@ -84,19 +88,24 @@ class ImageClassifierTest(unittest.TestCase):
def setUpClass(
self,
model="resnet-50",
feature="resnet-50_deep_features",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this right. If the class name doesn't end in WithDeepFeature, then I think it should be feature="awesome_image". This code is only calling get_deep_features when self.feature != "awesome_image".

I think you need to make this change to all of your test classes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the deep feature column name in test data to the one suggested by you and made all the requested changes.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you misunderstood my previous comment. I was not suggesting you need to change the name of the feature. I was basically just trying to say that it's important for the name of the class to represent what that class is actually testing.

For example, if the name of the class is ImageClassifierResnetTestWithDeepFeatures, then it should be feature="resnet-50_WithDeepFeature" or feature="resnet-50_deep_features", but not feature="awesome_image",. Similarly, if the name of the class is VisionFeaturePrintSceneTest then it should not be creating the model from deep features, so it should be feature="awesome_image", not feature="VisionFeaturePrint_Scene_WithDeepFeature".

Does that make sense? Let me know if you have any questions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, you are right I did not see it there. It was correct before but during one of my recent pushes I changed it to test something and forgot to revert it back. My bad, will change it and push it.
And about changing the name of feature columns in the test data in test_image_classifier.py and test_image_similarity.py to have a suffix _WithDeepFeature instead of _deep_features makes more sense to me. So I am keeping that change as it is.

input_image_shape=(3, 224, 224),
tol=0.02,
num_examples=100,
label_type=int,
):
self.feature = "awesome_image"
self.feature = feature
self.target = "awesome_label"
self.input_image_shape = input_image_shape
self.pre_trained_model = model
self.tolerance = tol

# Get deep features if needed
if self.feature != "awesome_image":
data[self.feature] = get_deep_features(data["awesome_image"], self.feature.split('_deep_features')[0])

self.model = tc.image_classifier.create(
data, target=self.target, model=self.pre_trained_model, seed=42
data, target=self.target, feature=self.feature, model=self.pre_trained_model, seed=42
)
self.nn_model = self.model.feature_extractor
self.lm_model = self.model.classifier
Expand Down Expand Up @@ -132,14 +141,13 @@ def assertListAlmostEquals(self, list1, list2, tol):
self.assertAlmostEqual(a, b, delta=tol)

def test_create_with_missing_value(self):
data_dict = {}
for col_name, col_type in zip(data.column_names(), data.column_types()):
data_dict[col_name] = tc.SArray([None], dtype=col_type)
data_with_none = data.append(
tc.SFrame(
{
self.feature: tc.SArray([None], dtype=tc.Image),
self.target: [data[self.target][0]],
}
)
tc.SFrame(data_dict)
)

with self.assertRaises(_ToolkitError):
tc.image_classifier.create(
data_with_none, feature=self.feature, target=self.target
Expand All @@ -161,6 +169,13 @@ def test_create_with_empty_dataset(self):
with self.assertRaises(_ToolkitError):
tc.image_classifier.create(data[:0], target=self.target)

def test_select_correct_feature_column_to_train(self):
# sending both, the correct extracted features colum and image column
if self.feature == "awesome_image":
test_data = data.select_columns([self.feature, self.target, self.pre_trained_model+"_deep_features"])
test_model = tc.image_classifier.create(test_data, target=self.target, model=self.pre_trained_model)
self.assertTrue(test_model.feature == self.pre_trained_model+"_deep_features")

def test_predict(self):
model = self.model
for output_type in ["class", "probability_vector"]:
Expand Down Expand Up @@ -235,22 +250,26 @@ def test_export_coreml_predict(self):
self.model.export_coreml(filename)

coreml_model = coremltools.models.MLModel(filename)
img = data[0:1][self.feature][0]
img_fixed = tc.image_analysis.resize(img, *reversed(self.input_image_shape))
from PIL import Image

pil_img = Image.fromarray(img_fixed.pixel_data)

if _mac_ver() >= (10, 13):
classes = self.model.classifier.classes
ret = coreml_model.predict({self.feature: pil_img})
coreml_values = [ret[self.target + "Probability"][l] for l in classes]

self.assertListAlmostEquals(
coreml_values,
list(self.model.predict(img_fixed, output_type="probability_vector")),
self.tolerance,
)
if self.feature == "awesome_image":
img = data[0:1][self.feature][0]
img_fixed = tc.image_analysis.resize(img, *reversed(self.input_image_shape))
from PIL import Image

pil_img = Image.fromarray(img_fixed.pixel_data)

if _mac_ver() >= (10, 13):
classes = self.model.classifier.classes
ret = coreml_model.predict({self.feature: pil_img})
coreml_values = [ret[self.target + "Probability"][l] for l in classes]

self.assertListAlmostEquals(
coreml_values,
list(self.model.predict(img_fixed, output_type="probability_vector")),
self.tolerance,
)
else:
# If the code came here that means the type of the feature used is deep_deatures and the predict fwature in coremltools doesn't work with deep_features yet so we will ignore this specific test case unitl the same is written.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should "fwature" be "function"? We try to maintain a 80 or 100 character length line limit. I would be nice if the text wrapped to the next line after 80 or 100 characters.

pass

def test_classify(self):
model = self.model
Expand Down Expand Up @@ -333,6 +352,18 @@ def test_evaluate_explore(self):
evaluation.explore()


class ImageClassifierResnetTestWithDeepFeatures(ImageClassifierTest):
@classmethod
def setUpClass(self):
super(ImageClassifierResnetTestWithDeepFeatures, self).setUpClass(
model="resnet-50",
input_image_shape=(3, 224, 224),
tol=0.02,
num_examples=100,
feature="awesome_image",
)


class ImageClassifierSqueezeNetTest(ImageClassifierTest):
TobyRoseman marked this conversation as resolved.
Show resolved Hide resolved
@classmethod
def setUpClass(self):
Expand All @@ -341,6 +372,18 @@ def setUpClass(self):
input_image_shape=(3, 227, 227),
tol=0.005,
num_examples=200,
feature="squeezenet_v1.1_deep_features",
)

class ImageClassifierSqueezeNetTestWithDeepFeatures(ImageClassifierTest):
@classmethod
def setUpClass(self):
super(ImageClassifierSqueezeNetTestWithDeepFeatures, self).setUpClass(
model="squeezenet_v1.1",
input_image_shape=(3, 227, 227),
tol=0.005,
num_examples=200,
feature="awesome_image",
)


Expand All @@ -357,4 +400,22 @@ def setUpClass(self):
tol=0.005,
num_examples=100,
label_type=str,
feature="VisionFeaturePrint_Scene_deep_features",
)


# TODO: if on skip OS, test negative case
@unittest.skipIf(
_mac_ver() < (10, 14), "VisionFeaturePrint_Scene only supported on macOS 10.14+"
)
class VisionFeaturePrintSceneTestWithDeepFeatures(ImageClassifierTest):
@classmethod
def setUpClass(self):
super(VisionFeaturePrintSceneTestWithDeepFeatures, self).setUpClass(
model="VisionFeaturePrint_Scene",
input_image_shape=(3, 299, 299),
tol=0.005,
num_examples=100,
label_type=str,
feature="awesome_image",
)
95 changes: 73 additions & 22 deletions src/python/turicreate/test/test_image_similarity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,11 @@
from turicreate.toolkits._internal_utils import _mac_ver
import tempfile
from . import util as test_util
import numpy as np

from turicreate.toolkits._main import ToolkitError as _ToolkitError
from turicreate.toolkits.image_analysis.image_analysis import MODEL_TO_FEATURE_SIZE_MAPPING, get_deep_features

import numpy as np


def get_test_data():
Expand Down Expand Up @@ -62,19 +65,22 @@ def get_test_data():
)
images.append(tc_image)

return tc.SFrame({"awesome_image": images})
data_dict = {"awesome_image": images}
data = tc.SFrame(data_dict)

return data
TobyRoseman marked this conversation as resolved.
Show resolved Hide resolved


data = get_test_data()


class ImageSimilarityTest(unittest.TestCase):
@classmethod
def setUpClass(self, input_image_shape=(3, 224, 224), model="resnet-50"):
def setUpClass(self, input_image_shape=(3, 224, 224), model="resnet-50", feature="awesome_image"):
"""
The setup class method for the basic test case with all default values.
"""
self.feature = "awesome_image"
self.feature = feature
self.label = None
self.input_image_shape = input_image_shape
self.pre_trained_model = model
Expand All @@ -85,6 +91,10 @@ def setUpClass(self, input_image_shape=(3, 224, 224), model="resnet-50"):
"verbose": True,
}

# Get deep features if needed
if self.feature != "awesome_image":
data[self.feature] = get_deep_features(data["awesome_image"], self.feature.split('_deep_features')[0])

# Model
self.model = tc.image_similarity.create(
data, feature=self.feature, label=None, model=self.pre_trained_model
Expand Down Expand Up @@ -251,21 +261,25 @@ def get_psnr(x, y):
)

# Get model distances for comparison
img = data[0:1][self.feature][0]
img_fixed = tc.image_analysis.resize(img, *reversed(self.input_image_shape))
tc_ret = self.model.query(img_fixed, k=data.num_rows())

if _mac_ver() >= (10, 13):
from PIL import Image as _PIL_Image

pil_img = _PIL_Image.fromarray(img_fixed.pixel_data)
coreml_ret = coreml_model.predict({"awesome_image": pil_img})

# Compare distances
coreml_distances = np.array(coreml_ret["distance"])
tc_distances = tc_ret.sort("reference_label")["distance"].to_numpy()
psnr_value = get_psnr(coreml_distances, tc_distances)
self.assertTrue(psnr_value > 50)
if self.feature == "awesome_image":
img = data[0:1][self.feature][0]
img_fixed = tc.image_analysis.resize(img, *reversed(self.input_image_shape))
tc_ret = self.model.query(img_fixed, k=data.num_rows())

if _mac_ver() >= (10, 13):
from PIL import Image as _PIL_Image

pil_img = _PIL_Image.fromarray(img_fixed.pixel_data)
coreml_ret = coreml_model.predict({"awesome_image": pil_img})

# Compare distances
coreml_distances = np.array(coreml_ret["distance"])
tc_distances = tc_ret.sort("reference_label")["distance"].to_numpy()
psnr_value = get_psnr(coreml_distances, tc_distances)
self.assertTrue(psnr_value > 50)
else:
# If the code came here that means the type of the feature used is deep_deatures and the predict fwature in coremltools doesn't work with deep_features yet so we will ignore this specific test case unitl the same is written.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment

pass

def test_save_and_load(self):
with test_util.TempDirectory() as filename:
Expand All @@ -287,11 +301,27 @@ def test_save_and_load(self):
print("Export coreml passed")


class ImageSimilarityResnetTestWithDeepFeatures(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilarityResnetTestWithDeepFeatures, self).setUpClass(
model="resnet-50", input_image_shape=(3, 224, 224), feature="resnet-50_deep_features"
)


class ImageSimilaritySqueezeNetTest(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilaritySqueezeNetTest, self).setUpClass(
model="squeezenet_v1.1", input_image_shape=(3, 227, 227)
model="squeezenet_v1.1", input_image_shape=(3, 227, 227), feature="awesome_image"
)


class ImageSimilaritySqueezeNetTestWithDeepFeatures(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilaritySqueezeNetTestWithDeepFeatures, self).setUpClass(
model="squeezenet_v1.1", input_image_shape=(3, 227, 227), feature="squeezenet_v1.1_deep_features"
)


Expand All @@ -302,10 +332,20 @@ class ImageSimilarityVisionFeaturePrintSceneTest(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilarityVisionFeaturePrintSceneTest, self).setUpClass(
model="VisionFeaturePrint_Scene", input_image_shape=(3, 299, 299)
model="VisionFeaturePrint_Scene", input_image_shape=(3, 299, 299), feature="awesome_image"
)


@unittest.skipIf(
_mac_ver() < (10, 14), "VisionFeaturePrint_Scene only supported on macOS 10.14+"
)
class ImageSimilarityVisionFeaturePrintSceneTestWithDeepFeatures(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilarityVisionFeaturePrintSceneTestWithDeepFeatures, self).setUpClass(
model="VisionFeaturePrint_Scene", input_image_shape=(3, 299, 299), feature="VisionFeaturePrint_Scene_deep_features"
)

# A test to gaurantee that old code using the incorrect name still works.
@unittest.skipIf(
_mac_ver() < (10, 14), "VisionFeaturePrint_Scene only supported on macOS 10.14+"
Expand All @@ -314,5 +354,16 @@ class ImageSimilarityVisionFeaturePrintSceneTest_bad_name(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilarityVisionFeaturePrintSceneTest_bad_name, self).setUpClass(
model="VisionFeaturePrint_Screen", input_image_shape=(3, 299, 299)
model="VisionFeaturePrint_Screen", input_image_shape=(3, 299, 299), feature="awesome_image"
)


@unittest.skipIf(
_mac_ver() < (10, 14), "VisionFeaturePrint_Scene only supported on macOS 10.14+"
)
class ImageSimilarityVisionFeaturePrintSceneTestWithDeepFeatures_bad_name(ImageSimilarityTest):
@classmethod
def setUpClass(self):
super(ImageSimilarityVisionFeaturePrintSceneTestWithDeepFeatures_bad_name, self).setUpClass(
model="VisionFeaturePrint_Screen", input_image_shape=(3, 299, 299), feature="VisionFeaturePrint_Scene_deep_features"
)
4 changes: 2 additions & 2 deletions src/python/turicreate/toolkits/image_analysis/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
from __future__ import division as _
from __future__ import absolute_import as _

__all__ = ["image_analysis"]
# __all__ = ["image_analysis"]

from . import image_analysis
from .image_analysis import *
TobyRoseman marked this conversation as resolved.
Show resolved Hide resolved
Loading