Custom data augmentation

[kmr2017]

Hi

Thanks for your wonder KerasCV library.

I am working on custom data augmentation for object detection. I will want to apply that with keras_CV existing augmentation. Can you please suggest, how can I do that?

Lets say, I want to apply random erasing on saliency part. Can you help me?

Best regards,
Teerath Kumar

[LukeWood]

Hey @kmr2017 thanks for your inquiry. Can you check out:

https://keras.io/guides/keras_cv/retina_net_overview/

to see how to use OD augmentations?

[kmr2017]

@LukeWood I check that, but I am unable to add custom data augemtnaiton to below code:

augment = keras_cv.layers.Augmenter(
layers=[
keras_cv.layers.RandomFlip(mode="horizontal", bounding_box_format="xywh"),
keras_cv.layers.RandAugment(
value_range=(0, 255),
rate=0.5,
magnitude=0.25,
augmentations_per_image=2,
geometric=False,
),
keras_cv.layers.JitteredResize(
target_size=(640, 640), scale_factor=(0.75, 1.3), bounding_box_format="xywh"
),
]
)

Can you help me to add custom data augmentation?

Thanks,
Teerath Kumar

[LukeWood]

What do you mean by custom? Lets re-phrase the question a bit, as I think I am missing the issue.

[kmr2017]

I mean I want to add custom data augmentation layer to the above code, how can I add?

[LukeWood]

have you implemented a custom augmentation?

https://keras.io/guides/keras_cv/custom_image_augmentations/

If you share the code you'd ideally like to achieve, that would clarify it a lot. Its not clear what the issue or hold up is. You can run any TensorFlow code the pipeline!

[kmr2017]

Here my code, I am writing code to find the saliencyPart (findSaliency) in class SaleincyAug and I added ' SaleincyAug' in augment. Data loading and model from link: https://github.com/keras-team/keras-io/blob/master/guides/keras_cv/retina_net_overview.py

Code

import tensorflow as tf
import cv2
import numpy as np
from keras_cv.layers.preprocessing.base_image_augmentation_layer import (
BaseImageAugmentationLayer,
)
from keras_cv.utils import preprocessing

def FindSaliency(img):

temp_img = img.copy()

saliency = cv2.saliency.StaticSaliencyFineGrained_create()
(success, saliencyMap) = saliency.computeSaliency(temp_img)
saliencyMap = (saliencyMap * 255).astype("uint8")
H = temp_img.shape[0]
W = temp_img.shape[1]
lam = np.random.beta(1,1)
cut_rat = np.sqrt(1. - lam)
cut_w = int(W * cut_rat)
cut_h = int(H * cut_rat)
maximum_indices = np.unravel_index(np.argmax(saliencyMap, axis=None), saliencyMap.shape)
x = maximum_indices[0]
y = maximum_indices[1]

bbx1 = np.clip(x - cut_w // 2, 0, W)
bby1 = np.clip(y - cut_h // 2, 0, H)
bbx2 = np.clip(x + cut_w // 2, 0, W)
bby2 = np.clip(y + cut_h // 2, 0, H)

saliency_img = temp_img[bby1:bby2, bbx1:bbx2].copy()

temp_img = cv2.resize(saliency_img, (H,W))
return temp_img

@tf.keras.utils.register_keras_serializable(package="keras_cv")

class SaleincyAug(BaseImageAugmentationLayer):
"""Performs the AutoContrast operation on an image.

find the saliency part 

Args:
    value_range: the range of values the incoming images will have.
        Represented as a two number tuple written [low, high].
        This is typically either `[0, 1]` or `[0, 255]` depending
        on how your preprocessing pipeline is setup.
"""

def __init__(
    self,
    value_range,
    **kwargs,
):
    super().__init__(**kwargs)
    self.value_range = value_range

def augment_image(self, image, transformation=None, **kwargs):
    original_image = image
    image = preprocessing.transform_value_range(
        image,
        original_range=self.value_range,
        target_range=(0, 255),
        dtype=self.compute_dtype,
    )

    #  Here is my funciton 
    ### **image = FindSaliency(image)** 


    result = tf.where(tf.math.is_nan(result), original_image, result)
    return result

def augment_bounding_boxes(self, bounding_boxes, **kwargs):
    return bounding_boxes

def augment_label(self, label, transformation=None, **kwargs):
    return label

def augment_segmentation_mask(self, segmentation_mask, transformation, **kwargs):
    return segmentation_mask

def get_config(self):
    config = super().get_config()
    config.update({"value_range": self.value_range})
    return config

augment = keras.Sequential(
layers=[
keras_cv.layers.RandomFlip(mode="horizontal", bounding_box_format="xywh"),
keras_cv.layers.RandAugment(
value_range=(0, 255),
rate=0.5,
magnitude=0.25,
augmentations_per_image=2,
geometric=False,
),
keras_cv.layers.JitteredResize(
target_size=(640, 640), scale_factor=(0.75, 1.3), bounding_box_format="xywh"
),
### SaleincyAug(value_range=(1.0,5.0))

]

)

train_ds = train_ds.map(
lambda x: augment(x, training=True), num_parallel_calls=tf.data.AUTOTUNE
)
visualize_dataset(train_ds, bounding_box_format="xywh")

Error I am getting is:

"

AttributeError Traceback (most recent call last)
in
103 )
104
--> 105 train_ds = train_ds.map(
106 lambda x: augment(x, training=True), num_parallel_calls=tf.data.AUTOTUNE
107 )

29 frames
/tmp/__autograph_generated_file0e6xhk_s.py in tf__FindSaliency(img)
8 do_return = False
9 retval_ = ag__.UndefinedReturnValue()
---> 10 temp_img = ag__.converted_call(ag__.ld(img).copy, (), None, fscope)
11 saliency = ag__.converted_call(ag__.ld(cv2).saliency.StaticSaliencyFineGrained_create, (), None, fscope)
12 (success, saliencyMap) = ag__.converted_call(ag__.ld(saliency).computeSaliency, (ag__.ld(temp_img),), None, fscope)

AttributeError: in user code:

File "<ipython-input-21-dacc51ba71f9>", line 106, in None  *
    lambda x: augment(x, training=True)
File "/usr/local/lib/python3.9/dist-packages/keras/utils/traceback_utils.py", line 70, in error_handler  **
    raise e.with_traceback(filtered_tb) from None
File "/tmp/__autograph_generated_filefuyxr4a9.py", line 75, in tf__call
    ag__.if_stmt(ag__.ld(training), if_body_2, else_body_2, get_state_2, set_state_2, ('do_return', 'retval_', 'inputs'), 2)
File "/tmp/__autograph_generated_filefuyxr4a9.py", line 63, in if_body_2
    ag__.if_stmt(ag__.ld(images).shape.rank == 3, if_body_1, else_body_1, get_state_1, set_state_1, ('do_return', 'retval_'), 2)
File "/tmp/__autograph_generated_filefuyxr4a9.py", line 62, in else_body_1
    ag__.if_stmt(ag__.ld(images).shape.rank == 4, if_body, else_body, get_state, set_state, ('do_return', 'retval_'), 2)
File "/tmp/__autograph_generated_filefuyxr4a9.py", line 54, in if_body
    retval_ = ag__.converted_call(ag__.ld(self)._format_output, (ag__.converted_call(ag__.ld(self)._batch_augment, (ag__.ld(inputs),), None, fscope), ag__.ld(metadata)), None, fscope)
File "/tmp/__autograph_generated_filen9jkg0nf.py", line 12, in tf___batch_augment
    retval_ = ag__.converted_call(ag__.ld(self)._map_fn, (ag__.ld(self)._augment, ag__.ld(inputs)), None, fscope)
File "/tmp/__autograph_generated_filepezu_2ia.py", line 60, in tf___map_fn
    ag__.if_stmt(ag__.or_(lambda : ag__.converted_call(ag__.ld(self)._any_ragged, (ag__.ld(inputs),), None, fscope), lambda : ag__.ld(self).force_output_ragged_images), if_body_1, else_body_1, get_state_1, set_state_1, ('do_return', 'retval_'), 2)
File "/tmp/__autograph_generated_filepezu_2ia.py", line 27, in if_body_1
    retval_ = ag__.converted_call(ag__.ld(tf).map_fn, (ag__.ld(func), ag__.ld(inputs)), dict(fn_output_signature=ag__.converted_call(ag__.ld(self)._compute_output_signature, (ag__.ld(inputs),), None, fscope)), fscope)
File "/tmp/__autograph_generated_files8qeha7i.py", line 34, in tf___augment
    image = ag__.converted_call(ag__.ld(self).augment_image, (ag__.ld(image),), dict(transformation=ag__.ld(transformation), bounding_boxes=ag__.ld(bounding_boxes), label=ag__.ld(label)), fscope)
File "/tmp/__autograph_generated_file_g8_g6is.py", line 12, in tf__augment_image
    image = ag__.converted_call(ag__.ld(FindSaliency), (ag__.ld(image),), None, fscope)
File "/tmp/__autograph_generated_file0e6xhk_s.py", line 10, in tf__FindSaliency
    temp_img = ag__.converted_call(ag__.ld(img).copy, (), None, fscope)

AttributeError: Exception encountered when calling layer 'saleincy_aug_2' (type SaleincyAug).

in user code:

    File "/usr/local/lib/python3.9/dist-packages/keras_cv/layers/preprocessing/base_image_augmentation_layer.py", line 373, in call  *
        return self._format_output(self._batch_augment(inputs), metadata)
    File "/usr/local/lib/python3.9/dist-packages/keras_cv/layers/preprocessing/base_image_augmentation_layer.py", line 463, in _batch_augment  *
        return self._map_fn(self._augment, inputs)
    File "/usr/local/lib/python3.9/dist-packages/keras_cv/layers/preprocessing/base_image_augmentation_layer.py", line 242, in _map_fn  *
        func, inputs, fn_output_signature=self._compute_output_signature(inputs)
    File "/usr/local/lib/python3.9/dist-packages/keras_cv/layers/preprocessing/base_image_augmentation_layer.py", line 406, in _augment  *
        image = self.augment_image(
    File "<ipython-input-27-dacc51ba71f9>", line 67, in augment_image  *
        image = FindSaliency(image)
    File "<ipython-input-27-dacc51ba71f9>", line 11, in FindSaliency  *
        temp_img = img.copy()

    AttributeError: 'Tensor' object has no attribute 'copy'


Call arguments received by layer 'saleincy_aug_2' (type SaleincyAug):
  • inputs={'images': 'tf.Tensor(shape=(None, 640, 640, 3), dtype=float32)', 'bounding_boxes': {'boxes': 'tf.RaggedTensor(values=tf.RaggedTensor(values=Tensor("sequential_3/jittered_resize_5/map/RaggedFromVariant/RaggedTensorFromVariant:2", shape=(None,), dtype=float32), row_splits=Tensor("sequential_3/jittered_resize_5/map/RaggedFromVariant/RaggedTensorFromVariant:1", shape=(None,), dtype=int64)), row_splits=Tensor("sequential_3/jittered_resize_5/map/RaggedFromVariant/RaggedTensorFromVariant:0", shape=(None,), dtype=int64))', 'classes': 'tf.RaggedTensor(values=Tensor("sequential_3/jittered_resize_5/map/RaggedFromVariant_1/RaggedTensorFromVariant:1", shape=(None,), dtype=float32), row_splits=Tensor("sequential_3/jittered_resize_5/map/RaggedFromVariant_1/RaggedTensorFromVariant:0", shape=(None,), dtype=int64))'}}
  • training=True

"

[kmr2017]

Hi @LukeWood

Are you looking into above issue? I need this one. I have to perform experiments.

Thanks,

[LukeWood]

Hi @LukeWood

Are you looking into above issue? I need this one. I have to perform experiments.

Thanks,

it looks like you are using cv2 calls in your augment_image() body. You'll need to use tf.py_function for this

[kmr2017]

Can you fix it? I tried, it is throwing same error.

[LukeWood]

Apologies @kmr2017, I do not have bandwidth to debug custom end user code - only KerasCV components. this is expected behavior. Try a using a py_function.

Please feel free to re-open if you have a colab to share with issues regarding KerasCV! This is just expected behavior as of now!